goingUnified Runtime For Heterogeneous Devices CUDA Driver NPU Driver Device Drivers External Runtimes NPUModule CUDAModule TFModule tvm::runtime::Module GetFunction(string) -> tvm::runtime::PackedFunc nc SaveToBinary/LoadFromBinary Runtime Module Interface SubclassesUnified Runtime Benefit mod.export_library("mylib.so") Unified library packaging Free API (Py/Java/Go) lib = tvm.module.load("mylib remote_b)Virtual Machine: Supporting Dynamic Workload Dynamic shape workloads More runtime objects: Arrays, Tuples, Trees, ADTs Minimum runtime for dynamic models Credit: Jared Roesch, Haichen Shen et.aluTVM: TVM

Model Analysis Graph optimization Blade Graph Optimizer TensorRT Customized OptimizeT TAO Compiler (XLA) cuUBLAS/VcuDNNVCUTL， Blade Kernel Lib S， ation 计算平台事业部 COMPUTING PLATFORM Weight Adjustment

dependent: arange, nms, etc. ○ Control flow: concatenate within a while loop Limitation of TVM/graph runtime ● Cannot compile and run dynamic models© 2019, Amazon Web Services, Inc. or its Affiliates. All TVM ● Support Any-dim in typing ● Use shape function to compute the type at runtime ● Virtual machine as a new runtime for Relay ● Dynamic codegen (WIP) ○ Kernel dispatch for a single op ○ Graph Fit for operator such as conv2d_NCHWc. Graph tuning is well defined for each subgraph. 3. Avoid runtime layout tracking system for operator requires layout transformation to optimize.© 2019, Amazon Web

FC layers - 24kHz sampling frequency requires 40us sampling net runtime - First PyTorch model used a 3,400us sampling net runtime Image from LPCNetExit, Pursued By A Bear - 3400us (baseline), 40us

ACL/CMSIS-NN kernels into TVM? ○ Implement Arm NN generic backend in TVM for more flexibility with the runtime plugins? ○ Integrate TVM codegen into Arm NN? ● CI and benchmark testing for TVM on member hardware