ingests a FP32 graph and a small dataset • Finds suitable quantization scale • Produces a quantized graph • Compiling Pre-quantized models – QNN Dialect • TVM ingests a pre-quantized graph in TFLite or rights reserved. TVM Overview Framework Graph Mxnet TF …. parsers Relay Graph Target-independent Relay passes Target-optimized graph Target-dependent Relay passes Intel x86 ARM CPU Nvidia GPU targets AutoTVM – Tuning the kernels Optimized Binary Codegen – LLVM, Cuda, C, … Framework Parsers Graph level optimizations Tensor-level optimizations Machine code generation© 2019, Amazon Web Services

from tvm import relay 2. Load a pretrained network mod, params = relay.testing.mobilenet.get_workload(batch_size=1) 3. Partition and build the network with an external codegen mod = relay.build_extern(mod build_extern(mod, “dnnl”) 4. Run the inference exe = relay.create_executor(“vm”, mod=mod, ctx=tvm.cpu(0)) data = np.random.uniform(size=(1, 3, 224, 224)).astype(“float32”) out = exe.evaluate()(data, **params) How System Overview Relay IR Graph Annotation with Your Annotator Graph Partitioning Your Codegen LLVM, CUDA, Metal, VTA Serialized Subgraph Library Relay Runtime (VM, Graph Runtime, Interpreter)

dependent: arange, nms, etc. ○ Control flow: concatenate within a while loop Limitation of TVM/graph runtime ● Cannot compile and run dynamic models© 2019, Amazon Web Services, Inc. or its Affiliates at runtime ● Virtual machine as a new runtime for Relay ● Dynamic codegen (WIP) ○ Kernel dispatch for a single op ○ Graph dispatch for a (sub-)graph In collaboration with Jared Roesch, Zhi Chen, Wei Wei Chen© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. “Any” in Relay typing Any: represent an unknown dimension at compilation time. Define a tensor type: Tensor<(Any, 3, 32

Tensor Graph Optimization Framework Tensor Graph to Xilinx Tensor Graph Frontend Deep Learning Frameworks https://github.com/xilinx© Copyright 2018 Xilinx TVM as Unified ML Front End >> 6 Relay (and (and NNVM) Graph Parser XIR Compiler Quantizer Partitioner @relay.transform.module_pass(opt_level=4) class AccelModule:© Copyright 2018 Xilinx TVM Partitioning >> 7 Subgraph 1 Parallel Subgraphs supported/not supported, pattern matching graph colorization - Choices how to partition especially for multi-branch networks (i.e. YOLOv3, SSD)© Copyright 2018 Xilinx TVM Graph Partitioning/Fusion >> 8 Subgraph

ASIC Optimization AutoTVM Device FleetExisting Deep Learning Frameworks High-level data flow graph Hardware Primitive Tensor operators such as Conv2D eg. cuDNN Offload to heavily optimized intensiveMachine Learning based Program Optimizer TVM: Learning-based Learning System High-level data flow graph and optimizations Directly generate optimized program for new operator workloads and hardware module/pass, type system, with function variants supportCompilation Flow under the New Infra IRModule (relay::Function) IRModule (te::Function, ExternFunc, …) runtime::Module High-level optimizations (Auto)

OpenAI- Add relay.nn.sparse_dense for block-sparse matrix multiplication (~50 lines of TVM IR) - Add relay.reinterpret to implement rational approximations in user space (~10 lines of Relay IR) - A few icache/ dcache - also available today in FBGEMMPyTorch and TVM - Lots of opportunity in PyTorch - Graph optimization - Existing fusion infrastructure fairly limited (CUDA-only, injective-only) - Kernel synthesis - Dynamic shapes, stride specialization - Impedance mismatch with PyTorch JIT IR and Relay IR - Watch this space :)Big thanks to the community

TVM @ Hexagon DSP 人NiOS ! 驱动万物知 Tensorflow deploy.so / deploy.json / deploy.bin | NNVM / Relay 让 Graph Optimization 站 站 Compile | libtvm_hexagon_runtime.so Alios TVM @ Hexagon DSP 。 Compute

with maths and automation, we can plot and visualize the change in values over time on a simple 2D graph. What’s more, we can also draw lines and curves that show the computer how we want it to calculate the selected keyframe to a specific value, as well as buttons to help zoom and navigate the main graph view. 2. Settings – While all of the high-traffic controls are presented directly, the right end color and its visibility within the graph view can be toggled by clicking on the eyeball icon. 4. Graph View – Last but not least is the graph view, the big graph of values and times that we use to animate

with maths and automation, we can plot and visualize the change in values over time on a simple 2D graph. What's more, we can also draw lines and curves that show the computer how we want it to calculate the selected keyframe to a specific value, as well as buttons to help zoom and navigate the main graph view. 2. Settings -- While all of the high-traffic controls are presented directly, the right end color and its visibility within the graph view can be toggled by clicking on the eyeball icon. 4. Graph View -- Last but not least is the graph view, the big graph of values and times that we use to

one unit per package each with different symbols and pin configurations. Consider for instance a relay with two switches which can be designed as a component with three different units: a coil, switch 1. Example of a Component Having Multiple Units with Different Symbols: This is an example of a relay defined with three units per package, switch 1, switch 2, and the coil: Option: pins are not linked 2. Graphical Symbolic Elements Shown below are properties for a graphic body element. From the relay example above, the three units have different symbolic representations. Therefore, each unit was created