Batch Norm 主讲人：龙良曲 Intuitive explanation Intuitive explanation Feature scaling ▪ Image Normalization ▪ Batch Normalization Batch Norm https://medium.com/syncedreview/facebook-ai-proposes-group-normalization-

置、均值、范数等信息。由于张量维度通常较大，直接观察数据很难获得有用信息，因此 通过获取这些张量的统计信息可以较轻松地推测张量数值的分布。下面将介绍一些常用的 张量统计函数。 5.2.1 向量范数 向量范数(Vector Norm)是表征向量“长度”的一种度量方法，它可以推广到张量上。 在神经网络中，常用来表示张量的权值大小、梯度大小等。常用的向量范数有： 预览版202112 第 5 章 PyTorch 进阶 计算，统称为向量范数。 在 PyTorch 中，可以通过 torch.norm(x, p, dim=None)求解张量的 L1、L2、∞等范数， 其中参数 p 指定为 1、2 时计算 L1、L2 范数，指定为 float(‘inf’)时计算∞ −范数，例如： In [13]: x = torch.ones([2,2]) torch.norm(x, p=1) # 计算 L1 范数 Out[13]: Out[13]: tensor(4.) In [14]: torch.norm(x, p=2) # 计算 L2 范数 Out[14]: tensor(2.) In [15]: import numpy as np torch.norm(x, p=np.inf) # 计算∞范数 Out[15]: tensor(1.) 在神经网络调试的过程中，通常需要在合适的地方查看张量的数值，直接打印张量并

Point{Float64}: function norm(p::Point{Real}) sqrt(p.x^2 + p.y^2) end A correct way to define a method that accepts all arguments of type Point{T} where T is a subtype of Real is: function norm(p::Point{<:Real}) norm(p::Point{<:Real}) sqrt(p.x^2 + p.y^2) end (Equivalently, one could define function norm(p::Point{T} where T<:Real) or function norm(p::Point{T}) where T<:Real; see UnionAll Types.) More examples will be discussed generic func�on on it. function power_method(M, v) for i in 1:100 v = M*v v /= norm(v) end return v, norm(M*v) / norm(v) # or (M*v) ./ v end power_method repeatedly creates a new vector and normalizes

Point{Float64}: function norm(p::Point{Real}) sqrt(p.x^2 + p.y^2) end A correct way to define a method that accepts all arguments of type Point{T} where T is a subtype of Real is: function norm(p::Point{<:Real}) norm(p::Point{<:Real}) sqrt(p.x^2 + p.y^2) end (Equivalently, one could define function norm(p::Point{T} where T<:Real) or function norm(p::Point{T}) where T<:Real; see UnionAll Types.) More examples will be discussed generic func�on on it. function power_method(M, v) for i in 1:100 v = M*v v /= norm(v) end return v, norm(M*v) / norm(v) # or (M*v) ./ v end power_method repeatedly creates a new vector and normalizes

Point{Float64}: function norm(p::Point{Real}) sqrt(p.x^2 + p.y^2) end A correct way to define a method that accepts all arguments of type Point{T} where T is a subtype of Real is: function norm(p::Point{<:Real}) norm(p::Point{<:Real}) sqrt(p.x^2 + p.y^2) end (Equivalently, one could define function norm(p::Point{T} where T<:Real) or function norm(p::Point{T}) where T<:Real; see UnionAll Types.) More examples will be discussed function power_method(M, v) for i in 1:100 v = M*v v /= norm(v) end 32.12. NOTEWORTHY EXTERNAL PACKAGES 303 return v, norm(M*v) / norm(v) # or (M*v) ./ v end power_method repeatedly creates a new

Point{Float64}: function norm(p::Point{Real}) sqrt(p.x^2 + p.y^2) end A correct way to define a method that accepts all arguments of type Point{T} where T is a subtype of Real is: function norm(p::Point{<:Real}) norm(p::Point{<:Real}) sqrt(p.x^2 + p.y^2) end (Equivalently, one could define function norm(p::Point{T} where T<:Real) or function norm(p::Point{T}) where T<:Real; see UnionAll Types.) More examples will be discussed function power_method(M, v) for i in 1:100 v = M*v v /= norm(v) end 33.12. NOTEWORTHY EXTERNAL PACKAGES 305 return v, norm(M*v) / norm(v) # or (M*v) ./ v end power_method repeatedly creates a new

Point{Float64}: function norm(p::Point{Real}) sqrt(p.x^2 + p.y^2) end A correct way to define a method that accepts all arguments of type Point{T} where T is a subtype of Real is: function norm(p::Point{<:Real}) p.y^2) end CHAPTER 11. TYPES 112 (Equivalently, one could define function norm(p::Point{T} where T<:Real) or function norm(p::Point{T}) where T<:Real; see UnionAll Types.) More examples will be discussed generic function on it. function power_method(M, v) for i in 1:100 v = M*v v /= norm(v) end return v, norm(M*v) / norm(v) # or (M*v) ./ v end power_method repeatedly creates a new vector and normalizes

Point{Float64}: function norm(p::Point{Real}) sqrt(p.x^2 + p.y^2) end A correct way to define a method that accepts all arguments of type Point{T} where T is a subtype of Real is: function norm(p::Point{<:Real}) p.y^2) end CHAPTER 11. TYPES 112 (Equivalently, one could define function norm(p::Point{T} where T<:Real) or function norm(p::Point{T}) where T<:Real; see UnionAll Types.) More examples will be discussed generic function on it. function power_method(M, v) for i in 1:100 v = M*v v /= norm(v) end return v, norm(M*v) / norm(v) # or (M*v) ./ v end power_method repeatedly creates a new vector and normalizes

Point{Float64}: function norm(p::Point{Real}) sqrt(p.x^2 + p.y^2) end A correct way to define a method that accepts all arguments of type Point{T} where T is a subtype of Real is: function norm(p::Point{<:Real}) p.y^2) end CHAPTER 11. TYPES 112 (Equivalently, one could define function norm(p::Point{T} where T<:Real) or function norm(p::Point{T}) where T<:Real; see UnionAll Types.) More examples will be discussed generic function on it. function power_method(M, v) for i in 1:100 v = M*v v /= norm(v) end return v, norm(M*v) / norm(v) # or (M*v) ./ v end power_method repeatedly creates a new vector and normalizes

Point{Float64}: function norm(p::Point{Real}) sqrt(p.x^2 + p.y^2) end A correct way to define a method that accepts all arguments of type Point{T} where T is a subtype of Real is: function norm(p::Point{<:Real}) p.y^2) end 118 CHAPTER 11. TYPES (Equivalently, one could define function norm(p::Point{T} where T<:Real) or function norm(p::Point{T}) where T<:Real; see UnionAll Types.) More examples will be discussed generic function on it. function power_method(M, v) for i in 1:100 v = M*v v /= norm(v) end return v, norm(M*v) / norm(v) # or (M*v) ./ v end power_method repeatedly creates a new vector and normalizes