time could still turn out to be expensive. There is also a very real concern around the carbon footprint of datacenters that are used for training and deploying these large models. Large organizations about. First, we have quality metrics like accuracy, precision, recall, F1, AUC, etc. Then we have footprint metrics like model size, latency, RAM, etc. Empirically, we have seen that larger deep learning train and deploy hence worse footprint. On the other hand, smaller and shallower models might have suboptimal quality. Figure 1-6: Trade-offs between quality metrics and footprint metrics. In case we have

changes to add a couple of windows and a balcony. Similarly, to gain orders of magnitude in terms of footprint or quality, we should consider employing suitable efficient architectures. The progress of deep Efficient Architectures aim to improve model deployability by proposing novel ways to reduce model footprint and improve inference efficiency while preserving the problem solving capabilities of their giant 2. Even after compression, the vocabulary itself is large: Large vocabularies have a tangible footprint by themselves, which excludes the actual embeddings. They are persisted with the model to help with

compression techniques. Compression techniques aim to reduce the model footprint (size, latency, memory etc.). We can reduce the model footprint by reducing the number of trainable parameters. However, this approach categories: footprint metrics such as model size, prediction latency, RAM consumption and the quality metrics, such as accuracy, F1, precision and recall as shown in table 2-1. Footprint Metrics Quality ● Accuracy ● Precision ● Recall ● F1 ● AUC Table 2-1: A few examples of footprint and quality metrics. The footprint and the quality metrics are typically at odds with each other. As stated earlier

that enable us to achieve our quality goals. High quality models have an additional benefit in footprint constrained environments like mobile and edge devices where they provide the flexibility to trade learning techniques. It is followed by a short discussion on exchanging model quality and model footprint. An in-depth discussion of data augmentation and distillation follows right after. Following the and/or label efficient training setup, can we exchange some of this to achieve a model with a better footprint? The next subsection elaborates it further. Using learning techniques to build smaller and faster

without increasing the footprint of the model (size, latency, etc). And as we have described earlier, some of these improved quality metrics can be traded off for a smaller footprint as desired. Continuing techniques to help you improve your model’s quality metrics without taking a hit on any of the footprint metrics. These techniques might get superseded by other better methods over time, but again our chapter 3, we found that distillation was a very handy technique to improve our model’s quality v/s footprint tradeoff. The motivation behind Subclass Distillation (Mueller et al.24) comes from the observation

precision. We will leave the biases untouched since they do not contribute significantly to the layer’s footprint. num_bits = 8 weights_dequantized, weights_reconstruction_error_quant = simulate_quantization( their specific model training setup. Sparsity by itself helps with compressing the model size (footprint metric) since many connections can be removed without a noticeable impact on quality metrics. However compression technique, yet implementing it is quite straightforward. We can achieve quality and footprint gains on top of quantization because clustering is a much more generic approach of allocating precision

turning the knobs (the hyperparameters) until we are satisfied with the sound (model quality and footprint) that each string produces. Unlike the guitar which has a few knobs, the hyperparameter search space

定层的数据作为输入，跨多个后续层 对数据进行处理，然后将数据发送到下一个GPU。与单个GPU所能处理的数据相比，我们可以用更大的网络 处理数据。此外，每个GPU占用的显存（memory footprint）可以得到很好的控制，虽然它只是整个网络显 存的一小部分。 然而，GPU的接口之间需要的密集同步可能是很难办的，特别是层之间计算的工作负载不能正确匹配的时候， 还有层之间的接口需要大量的