reserved. Presenter: Haichen Shen, Yao Wang Amazon SageMaker Neo, Deep Engine Science Dynamic Model in TVM AWS AI© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Models with dynamism loop Limitation of TVM/graph runtime ● Cannot compile and run dynamic models© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Support dynamic model in TVM ● Support Any-dim in Invoke Invokes a function at in index. InvokeClosure Invokes a Relay closure. InvokePacked Invokes a TVM compiled kernel. AllocStorage Allocates a storage block. AllocTensor Allocates a tensor value of

TVM: Where are we going Tianqi ChenCurrent Deep Learning Landscape Frameworks and Inference engines DL Compilers Kenrel Libraries Hardware CuDNN NNPack MKL-DNN Hand optimized Open source, automated automated end-to- end optimization framework for deep learning.TVM Stack High-Level Differentiable IR Tensor Expression and Optimization Search Space LLVM, CUDA, Metal VTA Edge FPGA Cloud FPGA optimization potential benefit: 1.5x speedup Engineering intensiveMachine Learning based Program Optimizer TVM: Learning-based Learning System High-level data flow graph and optimizations Directly generate optimized

�������������� ����� TVM for deloyment www.yealink.com dolphintear� ������������������� �����������������������3 � ���������1��1��,�/����,�1��,�������/��,�����/������,� .������1���1��,4 ����������� not deploy our network(with depthwise conv2d, ) 2. TVM can not only deploy our network, but also get a good performance gain by autotuning 3. TVM can support many kinds of hardware platform: Intel/arm For application on 32bits, no support of 32bit tensorflow , a workround from FrozenGene a. python/tvm/contrib/ndk.py options = options if options else [ “-shared”, “-fPIC”, “-m32”] b. python tensorflow_blur

TVM at Facebook Lots of contributors at FB and elsewhere- Performance matters a lot - Heterogenous computing environment - High variety of workloads - Ever-increasing set of primitives (over 500 500 aten kernels) - Interpreter methods not delivering generalized performance 2 Why TVM? XTVM for Speech Synthesis - WaveRNN-style model architecture - Autoregressive sampling net running at faster from LPCNetExit, Pursued By A Bear - 3400us (baseline), 40us (target) - 85x speedup - Uh ohEnter, TVM and model co-design - PyTorch operator overhead makes interpreter infeasible - Reduce FLOPs with

Outline 计算平台事业部 。TensorCore AutoCodeGen in TVM “。FP16 Mixed-Precision Training on PAI 。INT8 Inference on PAI-Blade 计算平台事业部 COMPUTING PLATFORM nvcuda::wmma::mem_col_majon Background 1 。TVM TensorCore Intrinsics 。Authored by @Hzfengsy 。 Intrinsics: tvm_load_matrix_sync tvm_mma_sync … “New Memory Scopes: wmma.matrix_a/b, accumulator 26X 1.51X 1.30X 1.21X Performance on T4 计算下从事业部 国 Cublas INT8， 9 国 TVM INT8 国 TVM INT4 罩 TVMINT1 675 旨 号 昌 45 全 2.25 ”cublas baseline (512, 64, 512 ) (512, 32, 512

TVM Meetup - Linaro Jammy Zhou November 16th, 2019Bringing together the Arm ecosystemLinaro AI Initiative Provide the best-in-class Deep Learning performance by leveraging Neural Network acceleration Arm Compute Library has been integrated by: ○ MATLAB Coder ○ ONNX RuntimeArm platform support in TVM upstream IPs Target Hardware/Model Options Codegen CPU arm_cpu pixel2 (snapdragon 835), mate10/mate10pro Hexagon DSP (via llvm), Ascend NPU, and more Green: Linaro 96BoardsLinaro for TVM ● Linaro AI/ML group can be a good fit for TVM collaborations on Arm based platforms to support more devices with various