reserved. Presenter: Haichen Shen, Yao Wang Amazon SageMaker Neo, Deep Engine Science Dynamic Model in TVM AWS AI© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Models with dynamism loop Limitation of TVM/graph runtime ● Cannot compile and run dynamic models© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Support dynamic model in TVM ● Support Any-dim in Invoke Invokes a function at in index. InvokeClosure Invokes a Relay closure. InvokePacked Invokes a TVM compiled kernel. AllocStorage Allocates a storage block. AllocTensor Allocates a tensor value of

Affiliates. All rights reserved. Animesh Jain Amazon SageMaker Neo Compilation of Quantized Models in TVM AWS AI© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Quantization Overview Services, Inc. or its Affiliates. All rights reserved. Quantization in TVM • Quantization within TVM - Automatic Quantization • TVM stack ingests a FP32 graph and a small dataset • Finds suitable quantization QNN Dialect • TVM ingests a pre-quantized graph in TFLite or MxNet • Use high-level wrapper ops of QNN dialect© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. TVM Overview Framework

PRESENTATION AGENDA 人 人 e 人 e@ TVM Q@ AliOs Overview TVM @ AliOs ARM CPU TVM @ AliOos Hexagon DSP TVM @ Alios Intel GPU Misc /NiiOS ! 驱动万物智能 PART ONE TVM Q@ AliOs Overview AiOS 1驱动万物智能 AliOs ROEWE负风 auto industry /NiiOS ! 驱动万物智能 TVM Timeline @ Alios 吕 2018.4 咏 2018.12 | 2019.6 ee 2019.10 Alios TVM Team Set up TFLite Quantized RX5 MAX OpenVINO @ Intel GPU AliDS AR-Nav Product @ SUV Release and adopt TVM (Apollo Lake Gold) Ready accelerated NLU model

绝赞招聘中 TVM CAFFE 前端 2019·11·16绝赞招聘中 TVM 在平头哥 • 工具链产品 平头哥芯片平台发布的配套软件中， TVM 是工具链产品的重要组成部分： 负责将预训练好的 caffe 或者 tensorflow 的模型，转换到 LLVM IR，最后生成可以在无剑 SoC 平台上 执行的二进制。绝赞招聘中 为何添加 caffe 前端？ 客户需求 评估 评估阶段：客户用于评估芯片的网络，caffe 模型占很大比重。 竞品已支持 caffe 前端 当前各大芯片厂商的部署工具大多数都支持，支持 caffe 前端有利于提高竞争力。 开源社区 存量的开源 caffe 网络模型众多，TVM 直接支持 caffe 让大家更方便尝试 caffe 资源。绝赞招聘中 当前进度 无 caffe 依赖 from_caffe 直接导入 caffe 模型文件，不需要预先安装 caffe 。 net

TVM: Where are we going Tianqi ChenCurrent Deep Learning Landscape Frameworks and Inference engines DL Compilers Kenrel Libraries Hardware CuDNN NNPack MKL-DNN Hand optimized Open source, automated automated end-to- end optimization framework for deep learning.TVM Stack High-Level Differentiable IR Tensor Expression and Optimization Search Space LLVM, CUDA, Metal VTA Edge FPGA Cloud FPGA optimization potential benefit: 1.5x speedup Engineering intensiveMachine Learning based Program Optimizer TVM: Learning-based Learning System High-level data flow graph and optimizations Directly generate optimized

© Copyright 2018 Xilinx Elliott Delaye FPGA CNN Accelerator and TVM© Copyright 2018 Xilinx TVM Target devices and models >> 2 HW Platforms ZCU102 ZCU104 Ultra96 PYNQ Face detection Pose estimation Xilinx TVM as Unified ML Front End >> 6 Relay (and NNVM) Graph Parser XIR Compiler Quantizer Partitioner @relay.transform.module_pass(opt_level=4) class AccelModule:© Copyright 2018 Xilinx TVM Partitioning SSD)© Copyright 2018 Xilinx TVM Graph Partitioning/Fusion >> 8 Subgraph 1 Parallel Subgraphs Post-Processing Pre-Processing CPU FPGA CPU CPU FPGA© Copyright 2018 Xilinx TVM Code Generation >> 9 Subgraph

�������������� ����� TVM for deloyment www.yealink.com dolphintear� ������������������� �����������������������3 � ���������1��1��,�/����,�1��,�������/��,�����/������,� .������1���1��,4 ����������� not deploy our network(with depthwise conv2d, ) 2. TVM can not only deploy our network, but also get a good performance gain by autotuning 3. TVM can support many kinds of hardware platform: Intel/arm For application on 32bits, no support of 32bit tensorflow , a workround from FrozenGene a. python/tvm/contrib/ndk.py options = options if options else [ “-shared”, “-fPIC”, “-m32”] b. python tensorflow_blur

Presenter: Zhi Chen, Cody Yu Amazon SageMaker Neo, Deep Engine Science Bring Your Own Codegen to TVM AWS AI© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Considering You. NMS is supported by TVM!© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Let TVM Be the Compiler of Your Chip Your chip can run any models Your compiler (TVM) supports multiple reserved. Example showcase: Intel MKL-DNN (DNNL) library 1. Import packages import numpy as np from tvm import relay 2. Load a pretrained network mod, params = relay.testing.mobilenet.get_workload(batch_size=1)

TVM at Facebook Lots of contributors at FB and elsewhere- Performance matters a lot - Heterogenous computing environment - High variety of workloads - Ever-increasing set of primitives (over 500 500 aten kernels) - Interpreter methods not delivering generalized performance 2 Why TVM? XTVM for Speech Synthesis - WaveRNN-style model architecture - Autoregressive sampling net running at faster from LPCNetExit, Pursued By A Bear - 3400us (baseline), 40us (target) - 85x speedup - Uh ohEnter, TVM and model co-design - PyTorch operator overhead makes interpreter infeasible - Reduce FLOPs with

Outline 计算平台事业部 。TensorCore AutoCodeGen in TVM “。FP16 Mixed-Precision Training on PAI 。INT8 Inference on PAI-Blade 计算平台事业部 COMPUTING PLATFORM nvcuda::wmma::mem_col_majon Background 1 。TVM TensorCore Intrinsics 。Authored by @Hzfengsy 。 Intrinsics: tvm_load_matrix_sync tvm_mma_sync … “New Memory Scopes: wmma.matrix_a/b, accumulator 26X 1.51X 1.30X 1.21X Performance on T4 计算下从事业部 国 Cublas INT8， 9 国 TVM INT8 国 TVM INT4 罩 TVMINT1 675 旨 号 昌 45 全 2.25 ”cublas baseline (512, 64, 512 ) (512, 32, 512