计算学习理论 (1)
作者：光火
邮箱：victor_b_zhang@163.com
计算学习理论旨在通过分析学习任务的困难本质，为学习算法提供理论保证，并根据分析结果指导算法设计。该部分内容在机器学习中难度较高，笔者计划通过一系列文章 / 笔记，对其中的重点知识进行分析与讲解，方便有需求的读者检索、查阅
基础知识
目前机器学习的主流仍为统计机器学习，而统计机器学习的基本假设就是：训练样本、测试样本均是从某一未知分布中随机采样得到的，且不同样本间没有关联，不会相互影响（独立同分布假设）
期望误差：E ( h ) = E ( x , y ) ∼ D ( x , y ) ℓ ( h ( x ) , y ) \large{\mathcal{E} (h) = E_{(x,y)\sim D(x,y)}\ell (h(x), y)} E ( h ) = E ( x , y ) ∼ D ( x , y ) ℓ ( h ( x ) , y )
经验误差：E ^ D n ( h ) = 1 n ∑ i = 1 n ℓ ( h ( x i ) , y i ) \large{\hat {\mathcal{E}}_{\mathcal{D_n}} (h) = \frac{1}{n}\sum_{i=1}^n\ell(h(x_i),y_i)} E ^ D n ( h ) = n 1 ∑ i = 1 n ℓ ( h ( x i ) , y i )
下标 ( x , y ) ∼ D ( x , y ) (x,y) \sim D(x,y) ( x , y ) ∼ D ( x , y ) ( x , y ) (x,y) ( x , y ) D D D
直观理解：期望误差就相当于对分布 D D D D D D h ( x ) h(x) h ( x )
由于期望误差无法计算，因此在实际编程中，我们一般采用经验误差，对采样到的有限条样本计算损失函数值，并求平均（受算力等因素影响，更为常见的做法是在一个 batch 上计算损失）
经验误差是期望误差的无偏估计。测试误差则是我们人为对期望误差做的一个近似，我们希望它能够较好地反映出期望误差
贝叶斯错误率：倘若我们知道样本是从哪个分布中采样得到的，就可以依据该分布，计算出样本对应的条件概率，并取其中的最大值作为样本的标签，即 h B a y e s ( x ) = arg max P [ y ∣ x ] h_{Bayes}(x) = \arg \max \mathbb{P} [y|x] h B a yes ( x ) = arg max P [ y ∣ x ]
如果你认为自己设计的模型足够优秀，就可以尝试在某一已知分布上采样，并将得到的样本送入你的模型，比较最终计算结果与贝叶斯错误率之间的差异
很多人会觉得，我都知道目标分布了，这不相当于开卷考试吗？给定输入，我直接代入目标分布得到输出，正确率应该 100 % 100\% 100% y y y y y y
具体推导：假设在某一回归问题中，对于给定输入 x x x h D n ( x ) h_{\mathcal{D_n}}(x) h D n ( x ) y y y
h D n h_{\mathcal{D_n}} h D n D n \mathcal{D_n} D n D n \mathcal{D_n} D n D D D D n \mathcal{D_n} D n h D n ( x ) h_{\mathcal{D_n}}(x) h D n ( x )
进一步，定义 R e g r e s s i o n f u n c t i o n : Regression \quad function: R e g ress i o n f u n c t i o n : f ∗ ( x ) = E y [ y ∣ x ] f^{*}(x) = \mathbb{E}_y[y|x] f ∗ ( x ) = E y [ y ∣ x ] D D D x x x y y y ∗ * ∗
对于回归问题，我们可以采用 L 2 L2 L 2
E D n , y [ ( h D n ( x ) − y ) 2 ∣ x ] = E D n , y [ ( h D n ( x ) − f ∗ ( x ) + f ∗ ( x ) − y ) 2 ∣ x ] ( 1 ) = E D n , y [ ( h D n ( x ) − f ∗ ( x ) ) 2 + ( f ∗ ( x ) − y ) 2 + 2 ( h D n ( x ) − f ∗ ( x ) ) ( f ∗ ( x ) − y ) ∣ x ] ( 2 ) = E D n [ ( h D n ( x ) − f ∗ ( x ) ) 2 ∣ x ] + E y [ ( f ∗ ( x ) − y ) 2 ∣ x ] + 2 E D n , y [ ( h D n ( x ) − f ∗ ( x ) ) ( f ∗ ( x ) − y ) ∣ x ] ( 3 ) \mathbb{E}_{D_n, y} \big [ \big(h_{D_n}(x) - y \big)^2 | x\big ] =
\mathbb{E}_{D_n, y} \big [ \big(h_{D_n}(x) - f^*(x) + f^*(x) - y \big)^2 | x\big ] \quad (1)
\\
= \mathbb{E}_{D_n, y}\big [\big(h_{D_n}(x) - f^*(x)\big)^2 + \big(f^*(x) -y\big)^2 + 2\big(h_{D_n}(x) - f^*(x)\big)\big(f^*(x) -y\big) |x \big ] \quad (2)
\\
= \mathbb{E}_{D_n} \big [\big(h_{D_n}(x) - f^*(x)\big)^2 |x\big] +
\mathbb{E}_{y}\big[(f^*(x) -y\big)^2|x \big ] +
2\mathbb{E}_{D_n, y}\big[\big(h_{D_n}(x) - f^*(x)\big)\big(f^*(x) -y\big)|x\big ] \quad(3) E D n , y [ ( h D n ( x ) − y ) 2 ∣ x ] = E D n , y [ ( h D n ( x ) − f ∗ ( x ) + f ∗ ( x ) − y ) 2 ∣ x ] ( 1 ) = E D n , y [ ( h D n ( x ) − f ∗ ( x ) ) 2 + ( f ∗ ( x ) − y ) 2 + 2 ( h D n ( x ) − f ∗ ( x ) ) ( f ∗ ( x ) − y ) ∣ x ] ( 2 ) = E D n [ ( h D n ( x ) − f ∗ ( x ) ) 2 ∣ x ] + E y [ ( f ∗ ( x ) − y ) 2 ∣ x ] + 2 E D n , y [ ( h D n ( x ) − f ∗ ( x ) ) ( f ∗ ( x ) − y ) ∣ x ] ( 3 ) 2 E D n , y [ ( h D n ( x ) − f ∗ ( x ) ) ( f ∗ ( x ) − y ) ∣ x ] = 2 E D n [ h D n ( x ) − f ∗ ( x ) ∣ x ] E y [ f ∗ ( x ) − y ∣ x ] ∵ R e g r e s s i o n f u n c t i o n : f ∗ ( x ) = E y [ y ∣ x ] ∴ E y [ f ∗ ( x ) − y ∣ x ] = E y [ f ∗ ( x ) ∣ x ] − E y [ y ∣ x ] = 0 ( 4 ) ( 3 ) = E D n [ ( h D n ( x ) − f ∗ ( x ) ) 2 ∣ x ] + E y [ ( f ∗ ( x ) − y ) 2 ∣ x ] 2\mathbb{E}_{D_n, y}\big[\big(h_{D_n}(x) - f^*(x)\big)\big(f^*(x) - y\big)|x\big ] =
2\mathbb{E}_{D_n}\big[h_{D_n}(x) - f^*(x)|x\big ]\mathbb{E}_{y}\big[f^*(x) - y|x\big ]
\\
\because Regression \quad function: f^{*}(x) = \mathbb{E}_y[y|x]
\\
\therefore \mathbb{E}_{y}\big[f^*(x) - y|x\big ] = \mathbb{E}_{y}\big[f^*(x)|x\big ] - \mathbb{E}_{y}\big[y|x\big ] = 0\quad (4)
\\
(3) = \mathbb{E}_{D_n} \big [\big(h_{D_n}(x) - f^*(x)\big)^2 |x\big] +
\mathbb{E}_{y}\big[(f^*(x) -y\big)^2|x \big ] 2 E D n , y [ ( h D n ( x ) − f ∗ ( x ) ) ( f ∗ ( x ) − y ) ∣ x ] = 2 E D n [ h D n ( x ) − f ∗ ( x ) ∣ x ] E y [ f ∗ ( x ) − y ∣ x ] ∵ R e g ress i o n f u n c t i o n : f ∗ ( x ) = E y [ y ∣ x ] ∴ E y [ f ∗ ( x ) − y ∣ x ] = E y [ f ∗ ( x ) ∣ x ] − E y [ y ∣ x ] = 0 ( 4 ) ( 3 ) = E D n [ ( h D n ( x ) − f ∗ ( x ) ) 2 ∣ x ] + E y [ ( f ∗ ( x ) − y ) 2 ∣ x ] 其中，E y [ ( f ∗ ( x ) − y ) 2 ∣ x ] \mathbb{E}_{ y}\big[(f^*(x) -y\big)^2|x \big ] E y [ ( f ∗ ( x ) − y ) 2 ∣ x ] y y y f ∗ ( x ) f^*(x) f ∗ ( x ) V a r ( y ∣ x ) Var(y|x) Va r ( y ∣ x )
继续考察第一项 E D n [ ( h D n ( x ) − f ∗ ( x ) ) 2 ∣ x ] \mathbb{E}_{D_n} \big [\big(h_{D_n}(x) - f^*(x)\big)^2 |x\big] E D n [ ( h D n ( x ) − f ∗ ( x ) ) 2 ∣ x ]
E D n [ ( h D n ( x ) − f ∗ ( x ) ) 2 ∣ x ] = E D n [ ( h D n ( x ) − E D n [ h D n ( x ) ] + E D n [ h D n ( x ) ] − f ∗ ( x ) ) 2 ∣ x ] = E D n [ ( h D n ( x ) − E D n [ h D n ( x ) ] ) 2 ∣ x ] + E D n [ ( E D n [ h D n ( x ) ] − f ∗ ( x ) ) 2 ∣ x ] + 2 E D n [ h D n ( x ) − E D n [ h D n ( x ) ] ∣ x ] ⋅ E D n [ E D n [ h D n ( x ) ] − f ∗ ( x ) ∣ x ] ( 5 ) \mathbb{E}_{D_n} \big [\big(h_{D_n}(x) - f^*(x)\big)^2 |x\big]
\\
= \mathbb{E}_{D_n} \big [\big(h_{D_n}(x) - {E}_{D_n} \big [ h_{D_n}(x)\big ] + {E}_{D_n} \big [h_{D_n}(x) \big ]- f^*(x) \big)^2|x\big]
\\
= \mathbb{E}_{D_n} \big [\big(h_{D_n}(x) - {E}_{D_n} \big [ h_{D_n}(x)\big ]\big)^2|x\big ] +
\mathbb{E}_{D_n} \big [\big( {E}_{D_n} \big [h_{D_n}(x) \big ]- f^*(x) \big)^2|x\big ]
\\
+2\mathbb{E}_{D_n} \big [h_{D_n}(x) - {E}_{D_n} \big [ h_{D_n}(x)\big ]|x\big ]
\cdot
\mathbb{E}_{D_n} \big [ {E}_{D_n} \big [h_{D_n}(x) \big ]- f^*(x) |x\big ]
\quad (5) E D n [ ( h D n ( x ) − f ∗ ( x ) ) 2 ∣ x ] = E D n [ ( h D n ( x ) − E D n [ h D n ( x ) ] + E D n [ h D n ( x ) ] − f ∗ ( x ) ) 2 ∣ x ] = E D n [ ( h D n ( x ) − E D n [ h D n ( x ) ] ) 2 ∣ x ] + E D n [ ( E D n [ h D n ( x ) ] − f ∗ ( x ) ) 2 ∣ x ] + 2 E D n [ h D n ( x ) − E D n [ h D n ( x ) ] ∣ x ] ⋅ E D n [ E D n [ h D n ( x ) ] − f ∗ ( x ) ∣ x ] ( 5 ) E D n [ h D n ( x ) − E D n [ h D n ( x ) ] ∣ x ] = E D n [ h D n ( x ) ∣ x ] − E D n [ h D n ( x ) ∣ x ] = 0 ( 5 ) = E D n [ ( h D n ( x ) − E D n [ h D n ( x ) ] ) 2 ∣ x ] + E D n [ ( E D n [ h D n ( x ) ] − f ∗ ( x ) ) 2 ∣ x ] \mathbb{E}_{D_n} \big [h_{D_n}(x) - {E}_{D_n} \big [ h_{D_n}(x)\big ]|x\big ] =
\mathbb{E}_{D_n} \big [h_{D_n}(x)|x \big]-\mathbb{E}_{D_n} \big [h_{D_n}(x)|x \big] = 0
\\
(5) = \mathbb{E}_{D_n} \big [\big(h_{D_n}(x) - {E}_{D_n} \big [ h_{D_n}(x)\big ]\big)^2|x\big ] +
\mathbb{E}_{D_n} \big [\big( {E}_{D_n} \big [h_{D_n}(x) \big ]- f^*(x) \big)^2|x\big ] E D n [ h D n ( x ) − E D n [ h D n ( x ) ] ∣ x ] = E D n [ h D n ( x ) ∣ x ] − E D n [ h D n ( x ) ∣ x ] = 0 ( 5 ) = E D n [ ( h D n ( x ) − E D n [ h D n ( x ) ] ) 2 ∣ x ] + E D n [ ( E D n [ h D n ( x ) ] − f ∗ ( x ) ) 2 ∣ x ] 其中，E D n [ ( h D n ( x ) − E D n [ h D n ( x ) ] ) 2 ∣ x ] \mathbb{E}_{D_n} \big [\big(h_{D_n}(x) - {E}_{D_n} \big [ h_{D_n}(x)\big ]\big)^2|x\big ] E D n [ ( h D n ( x ) − E D n [ h D n ( x ) ] ) 2 ∣ x ] V a r D [ h D ( x ) ∣ x ] Var_{D}[h_{D}(x)|x] Va r D [ h D ( x ) ∣ x ] D n D_n D n D D D D D D
E D n [ ( E D n [ h D n ( x ) ] − f ∗ ( x ) ) 2 ∣ x ] \mathbb{E}_{D_n} \big [\big( {E}_{D_n} \big [h_{D_n}(x) \big ]- f^*(x) \big)^2|x\big ] E D n [ ( E D n [ h D n ( x ) ] − f ∗ ( x ) ) 2 ∣ x ] h D n h_{D_n} h D n f ∗ f^* f ∗
综上所述，回归问题的期望误差由三部分组成
E L 2 ( x ) = V a r ( y ∣ x ) + B i a s [ h D ( x ) ∣ x ] 2 + V a r D [ h D ( x ) ∣ x ] \large \mathcal{E}_{L2}(x) = Var(y|x) + Bias\big[h_D(x)|x \big]^2 + Var_{D}\big[h_D(x)|x\big] E L 2 ( x ) = Va r ( y ∣ x ) + B ia s [ h D ( x ) ∣ x ] 2 + Va r D [ h D ( x ) ∣ x ] 反映在实际操作中：简单模型偏差大、方差小；复杂模型偏差小、方差大
上述推导是在回归问题下进行的，更为通用的概念是近似误差和估计误差
g i v a a t a r g e t f u n c t i o n f , f o r a n y h ∈ H E ( h ) − E ∗ ( f ) = [ E ( h ) − E ( h ∗ ) ] + [ E ( h ∗ ) − E ∗ ( f ) ] giva\quad a \quad target\quad function\quad f,\quad for \quad any\quad h\in \mathcal{H}
\\
\large \mathcal{E}(h) - \mathcal{E}^*(f) = \big[\mathcal{E}(h) - \mathcal{E}(h^*)\big] + \big[\mathcal{E}(h^*) - \mathcal{E}^*(f)\big] g i v a a t a r g e t f u n c t i o n f , f or an y h ∈ H E ( h ) − E ∗ ( f ) = [ E ( h ) − E ( h ∗ ) ] + [ E ( h ∗ ) − E ∗ ( f ) ] 第一项为估计误差、第二项为近似误差
选用简单的函数族：近似误差大、估计误差小
选用复杂的函数族：近似误差小、估计误差大
常用不等式
J e n s e n Jensen J e n se n ≥ \geq ≥
i f f i s c o n v e x , t h e n f ( E [ X ] ) ≤ E [ f ( X ) ] if \quad f\quad is\quad convex,
\\
then\quad f(\mathbb{E}[X]) \leq \mathbb{E}[f(X)] i f f i s co n v e x , t h e n f ( E [ X ]) ≤ E [ f ( X )]
马尔可夫不等式（通常用于推导更高阶的不等式）
P ( z ≥ ϵ ) ≤ E [ Z ] ϵ P(z \geq \epsilon) \leq \frac{\mathbb{E}[Z]}{\epsilon} P ( z ≥ ϵ ) ≤ ϵ E [ Z ]
切比雪夫不等式（可由马尔可夫不等式推导得到）
切比雪夫不等式是一个典型的集中不等式
P ( ∣ Z − E [ Z ] ∣ ≥ ϵ ) ≤ V a r ( Z ) ϵ 2 P(\big|Z-\mathbb E[Z]\big| \geq \epsilon) \leq \frac{Var(Z)}{\epsilon^2} P ( ∣ ∣ Z − E [ Z ] ∣ ∣ ≥ ϵ ) ≤ ϵ 2 Va r ( Z )
H o e f f d i n g Hoeffding Hoe ffd in g
若 x 1 , x 2 , . . . , x m x_1, x_2,...,x_m x 1 , x 2 , ... , x m m m m 0 ≤ x i ≤ 1 0 \leq x_i \leq 1 0 ≤ x i ≤ 1 ϵ > 0 \epsilon > 0 ϵ > 0
P ( 1 m ∑ i = 1 m x i − 1 m ∑ i = 1 m E ( x i ) ≥ ϵ ) ≤ exp ( − 2 m ϵ 2 ) P ( ∣ 1 m ∑ i = 1 m x i − 1 m ∑ i = 1 m E ( x i ) ∣ ≥ ϵ ) ≤ 2 exp ( − 2 m ϵ 2 ) \large P\big(\frac{1}{m}\sum_{i=1}^m x_i - \frac{1}{m}\sum_{i=1}^m \mathbb E(x_i) \geq \epsilon\big) \leq \exp(-2m\epsilon^2)
\\
\large P\big(\bigg|\frac{1}{m}\sum_{i=1}^m x_i - \frac{1}{m}\sum_{i=1}^m \mathbb E(x_i) \bigg| \geq \epsilon \big) \leq 2\exp(-2m\epsilon^2) P ( m 1 i = 1 ∑ m x i − m 1 i = 1 ∑ m E ( x i ) ≥ ϵ ) ≤ exp ( − 2 m ϵ 2 ) P ( ∣ ∣ m 1 i = 1 ∑ m x i − m 1 i = 1 ∑ m E ( x i ) ∣ ∣ ≥ ϵ ) ≤ 2 exp ( − 2 m ϵ 2 ) 另外一种常用形式：
f o r a i ≤ X i ≤ b i P ( ∣ ∑ i = 1 n ( X i − E X i ) ∣ ≥ ϵ ) ≤ 2 exp [ − 2 ϵ 2 ∑ i = 1 n ( b i − a i ) 2 ] for\quad a_i \leq X_i \leq b_i
\\
P\bigg(\bigg|\sum_{i=1}^n \big(X_i - \mathbb EX_i \big) \bigg| \geq \epsilon \bigg) \leq 2 \exp \bigg[ - \frac{2\epsilon^2}{\sum_{i=1}^n (b_i - a_i)^2} \bigg] f or a i ≤ X i ≤ b i P ( ∣ ∣ i = 1 ∑ n ( X i − E X i ) ∣ ∣ ≥ ϵ ) ≤ 2 exp [ − ∑ i = 1 n ( b i − a i ) 2 2 ϵ 2 ]
M c D i a r m i d McDiarmid M cD ia r mi d
若 x 1 , x 2 , . . . , x m x_1, x_2,...,x_m x 1 , x 2 , ... , x m m m m 1 ≤ i ≤ m 1 \leq i \leq m 1 ≤ i ≤ m f f f
sup x 1 , . . . , x m , x i ′ ∣ f ( x 1 , . . . , x m ) − f ( x 1 , . . . , x i − 1 , x i ′ , x i + 1 , . . . , x m ) ∣ ≤ c i \large \sup_{x_1, ..., x_m, x'_i} |f(x_1, ...,x_m) - f(x_1,...,x_{i-1},x'_i,x_{i+1},...,x_m)| \leq c_i x 1 , ... , x m , x i ′ sup ∣ f ( x 1 , ... , x m ) − f ( x 1 , ... , x i − 1 , x i ′ , x i + 1 , ... , x m ) ∣ ≤ c i 则对任意 ϵ > 0 \epsilon > 0 ϵ > 0
P ( f ( x 1 , . . . , x m ) − E ( f ( x 1 , . . . , x m ) ) ≥ ϵ ) ≤ exp ( − 2 ϵ 2 ∑ i = 1 c i 2 ) P ( ∣ f ( x 1 , . . . , x m ) − E ( f ( x 1 , . . . , x m ) ) ∣ ≥ ϵ ) ≤ 2 exp ( − 2 ϵ 2 ∑ i = 1 c i 2 ) P\big(f(x_1,...,x_m) - \mathbb E(f(x_1,...,x_m))\geq \epsilon \big) \leq \exp(\frac{-2\epsilon^2}{\sum_{i=1}c_i^2})
\\
P\big(\big| f(x_1,...,x_m) - \mathbb E(f(x_1,...,x_m)) \big |\geq \epsilon \big) \leq 2\exp(\frac{-2\epsilon^2}{\sum_{i=1}c_i^2}) P ( f ( x 1 , ... , x m ) − E ( f ( x 1 , ... , x m )) ≥ ϵ ) ≤ exp ( ∑ i = 1 c i 2 − 2 ϵ 2 ) P ( ∣ ∣ f ( x 1 , ... , x m ) − E ( f ( x 1 , ... , x m )) ∣ ∣ ≥ ϵ ) ≤ 2 exp ( ∑ i = 1 c i 2 − 2 ϵ 2 )
有限假设空间泛化误差界
回忆 H o e f f d i n g Hoeffding Hoe ffd in g
P ( ∣ ∑ i = 1 n ( X i − E X i ) ∣ ≥ ϵ ) ≤ 2 exp [ − 2 ϵ 2 ∑ i = 1 n ( b i − a i ) 2 ] P\bigg(\bigg|\sum_{i=1}^n \big(X_i - \mathbb EX_i \big) \bigg| \geq \epsilon \bigg) \leq 2 \exp \bigg[ - \frac{2\epsilon^2}{\sum_{i=1}^n (b_i - a_i)^2} \bigg] P ( ∣ ∣ i = 1 ∑ n ( X i − E X i ) ∣ ∣ ≥ ϵ ) ≤ 2 exp [ − ∑ i = 1 n ( b i − a i ) 2 2 ϵ 2 ] 这里我们要求随机变量 X i X_i X i a i a_i a i b i b_i b i
在机器学习中，损失函数 ℓ ( h ( x i ) , y i ) \ell (h(x_i), y_i) ℓ ( h ( x i ) , y i ) X i = ℓ ( h ( x i ) , y i ) X_i = \ell (h(x_i), y_i) X i = ℓ ( h ( x i ) , y i ) x i x_i x i y i y_i y i X X X
∑ i = 1 n ( X i − E X i ) = n { [ 1 n ∑ i = 1 n ℓ ( h ( x i ) , y i ) ] − E ( x , y ) ∼ D ℓ ( h ( x ) , y ) } ( ∗ ) = n ( ε ^ D n ( h ) − ε ( h ) ) \sum_{i=1}^n \big(X_i - \mathbb EX_i \big)
\\
= n\bigg\{\bigg[\frac{1}{n}\sum_{i=1}^n \ell(h(x_i), y_i) \bigg]- \mathbb E_{(x,y)\sim D}\ell(h(x), y) \bigg\} \quad (*)
\\
= n\big(\hat{\varepsilon}_{D_n}(h) - \varepsilon (h) \big) i = 1 ∑ n ( X i − E X i ) = n { [ n 1 i = 1 ∑ n ℓ ( h ( x i ) , y i ) ] − E ( x , y ) ∼ D ℓ ( h ( x ) , y ) } ( ∗ ) = n ( ε ^ D n ( h ) − ε ( h ) ) ( ∗ ) (*) ( ∗ ) n n n 1 n \frac{1}{n} n 1 H o e f f d i n g Hoeffding Hoe ffd in g 01 01 01 1 [ h ( x ) ≠ y ] 1[h(x) \not = y] 1 [ h ( x ) = y ] a i = 0 , b i = 1 a_i = 0, b_i = 1 a i = 0 , b i = 1
P ( n ∣ ε ^ D n ( h ) − ε ( h ) ∣ ≥ ϵ ) ≤ 2 e − 2 ϵ 2 n \large P\big( n \big| \hat{\varepsilon}_{D_n}(h) - \varepsilon (h)\big| \geq \epsilon \big) \leq 2 e^{-\frac{2\epsilon^2}{n}} P ( n ∣ ∣ ε ^ D n ( h ) − ε ( h ) ∣ ∣ ≥ ϵ ) ≤ 2 e − n 2 ϵ 2 上式的 n n n ϵ \epsilon ϵ n ϵ n\epsilon n ϵ
P ( n ∣ ε ^ D n ( h ) − ε ( h ) ∣ ≥ n ϵ ) ≤ 2 e − 2 ϵ 2 n 2 n = P ( ∣ ε ^ D n ( h ) − ε ( h ) ∣ ≥ ϵ ) ≤ 2 e − 2 n ϵ 2 \large P\big( n \big| \hat{\varepsilon}_{D_n}(h) - \varepsilon (h)\big| \geq n\epsilon \big) \leq 2 e^{-\frac{2\epsilon^2n^2}{n}}
\\
= \large P\big(\big| \hat{\varepsilon}_{D_n}(h) - \varepsilon (h)\big| \geq \epsilon \big) \leq 2 e^{-2n\epsilon^2} P ( n ∣ ∣ ε ^ D n ( h ) − ε ( h ) ∣ ∣ ≥ n ϵ ) ≤ 2 e − n 2 ϵ 2 n 2 = P ( ∣ ∣ ε ^ D n ( h ) − ε ( h ) ∣ ∣ ≥ ϵ ) ≤ 2 e − 2 n ϵ 2 我们希望 ∣ E ^ D n ( h ) − E ( h ) ∣ ≥ ϵ \big| \hat{\mathcal{E}}_{D_n}(h) - \mathcal{E} (h)\big| \geq \epsilon ∣ ∣ E ^ D n ( h ) − E ( h ) ∣ ∣ ≥ ϵ
L e t δ = P ( ∣ E ^ D n ( h ) − E ( h ) ∣ ≥ ϵ ) T h e n δ ≤ 2 e − 2 n ϵ 2 ⇒ ϵ ≤ log 2 δ 2 n Let\quad \delta = P\big(\big| \hat{\mathcal{E}}_{D_n}(h) - \mathcal{E} (h)\big| \geq \epsilon \big)
\\
Then\quad \delta \leq 2 e^{-2n\epsilon^2} \Rightarrow \epsilon \leq \frac{\log \frac{2}{\delta}}{2n} L e t δ = P ( ∣ ∣ E ^ D n ( h ) − E ( h ) ∣ ∣ ≥ ϵ ) T h e n δ ≤ 2 e − 2 n ϵ 2 ⇒ ϵ ≤ 2 n log δ 2 因此，E ( h ) ≤ E ^ D n ( h ) + log 2 δ 2 n \mathcal{E} (h) \leq \hat{\mathcal{E}}_{D_n}(h) + \frac{\log \frac{2}{\delta}}{2n} E ( h ) ≤ E ^ D n ( h ) + 2 n l o g δ 2 1 − δ 1 - \delta 1 − δ ∣ E ^ D n ( h ) − E ( h ) ∣ ≤ ϵ \big| \hat{\mathcal{E}}_{D_n}(h) - \mathcal{E} (h)\big| \leq \epsilon ∣ ∣ E ^ D n ( h ) − E ( h ) ∣ ∣ ≤ ϵ 1 − δ 1 - \delta 1 − δ
上述推导中，我们假设 h h h h h h h h h
D n → h D n = arg min h ∈ H E ^ D n ( h ) \mathcal{D_n} \rightarrow h_{\mathcal{D_n}} = \arg \min_{h \in \mathcal{H} } \hat{\mathcal{E}}_{\mathcal{D_n}}(h) D n → h D n = arg h ∈ H min E ^ D n ( h ) 即在函数族 H \mathcal{H} H D n \mathcal{D_n} D n h h h
这时，我们就应采用保守学习的思想，考虑最坏情况，获得一个相对松弛的上界 (uniform bound )
P ( ∃ h ∈ H , ∣ E ^ D n ( h ) − E ( h ) ∣ ≥ ϵ ) = P ( sup h ∈ H ∣ E ^ D n ( h ) − E ( h ) ∣ ≥ ϵ ∣ ) ( 1 ) = P ( [ ∣ E ^ D n ( h 1 ) − E ( h 1 ) ∣ ≥ ϵ ] ∨ . . . ∨ [ ∣ E ^ D n ( h ∣ H ∣ ) − E ( h ∣ H ∣ ) ∣ ≥ ϵ ] ) ( 2 ) ≤ ∑ h ∈ H P ( ∣ E ^ D n ( h ) − E ( h ) ∣ ≥ ϵ ) ≤ 2 ∣ H ∣ exp ( − 2 n ϵ 2 ) ( 3 ) P\big(\exist h \in \mathcal{H}, \big| \hat{\mathcal{E}}_{D_n}(h) - \mathcal{E}(h) \big | \geq \epsilon\big) = P\big(\sup_{h\in \mathcal{H}} \big| \hat{\mathcal{E}}_{D_n}(h) - \mathcal{E}(h) \big | \geq \epsilon \big|\big)\quad (1)
\\
= P\big(\big[\big| \hat{\mathcal{E}}_{D_n}(h_1) - \mathcal{E}(h_1) \big| \geq \epsilon \big] \vee ... \vee \big[\big| \hat{\mathcal{E}}_{D_n}(h_{\mathcal{|H|}}) - \mathcal{E}(h_{\mathcal{|H|}})\big| \geq \epsilon \big]\big) \quad (2)
\\
\leq \sum_{h \in \mathcal{H}} P\big(\big| \hat{\mathcal{E}}_{D_n}(h) - \mathcal{E}(h) \big | \geq \epsilon \big) \leq 2\mathcal{|H|} \exp(-2n\epsilon^2) \quad (3) P ( ∃ h ∈ H , ∣ ∣ E ^ D n ( h ) − E ( h ) ∣ ∣ ≥ ϵ ) = P ( h ∈ H sup ∣ ∣ E ^ D n ( h ) − E ( h ) ∣ ∣ ≥ ϵ ∣ ∣ ) ( 1 ) = P ( [ ∣ ∣ E ^ D n ( h 1 ) − E ( h 1 ) ∣ ∣ ≥ ϵ ] ∨ ... ∨ [ ∣ ∣ E ^ D n ( h ∣ H ∣ ) − E ( h ∣ H ∣ ) ∣ ∣ ≥ ϵ ] ) ( 2 ) ≤ h ∈ H ∑ P ( ∣ ∣ E ^ D n ( h ) − E ( h ) ∣ ∣ ≥ ϵ ) ≤ 2 ∣ H ∣ exp ( − 2 n ϵ 2 ) ( 3 ) ( 1 ) (1) ( 1 ) sup \sup sup
( 2 ) (2) ( 2 ) ∃ \exist ∃ ( 3 ) (3) ( 3 ) union bound 性质，并使用 H o e f f d i n g Hoeffding Hoe ffd in g
L e t δ = ∑ h ∈ H P ( ∣ E ^ D n ( h ) − E ( h ) ∣ ≥ ϵ ) T h e n δ ≤ 2 ∣ H ∣ exp ( − 2 n ϵ 2 ) ϵ ≤ log ∣ H ∣ + log ( 2 δ ) 2 n Let\quad \delta = \sum_{h \in \mathcal{H}} P\big(\big| \hat{\mathcal{E}}_{D_n}(h) - \mathcal{E}(h) \big | \geq \epsilon \big)
\\
Then\quad \delta \leq 2\mathcal{|H|} \exp(-2n\epsilon^2)
\\
\epsilon \leq \sqrt{\frac{\log \mathcal{|H|} + \log(\frac{2}{\delta})}{2n}} L e t δ = h ∈ H ∑ P ( ∣ ∣ E ^ D n ( h ) − E ( h ) ∣ ∣ ≥ ϵ ) T h e n δ ≤ 2 ∣ H ∣ exp ( − 2 n ϵ 2 ) ϵ ≤ 2 n log ∣ H ∣ + log ( δ 2 )
由此，得到结论
Let H \mathcal{H} H be a finite hypothesis space, H < ∞ , \mathcal{H} < \infin, H < ∞ , then for any δ > 0 , \delta > 0, δ > 0 , with probability at least 1 − δ 1 - \delta 1 − δ
∀ h ∈ H ， E ( h ) ≤ E ^ D n ( h ) + log ∣ H ∣ + log 2 δ 2 n \forall h \in \mathcal{H}， \mathcal{E}(h) \leq \hat{\mathcal{E}}_{D_n}(h) +
\sqrt{\frac{\log|\mathcal{H}| + \log \frac{2}{\delta}}{2n}} ∀ h ∈ H ， E ( h ) ≤ E ^ D n ( h ) + 2 n log ∣ H ∣ + log δ 2 利用该式可以解释很多现象，例如为什么当样本数 n n n U 形（当假设空间 H \mathcal{H} H E ^ D n ( h ) \hat{\mathcal{E}}_{D_n}(h) E ^ D n ( h ) log ∣ H ∣ + log 2 δ 2 n \sqrt{\frac{\log|\mathcal{H}| + \log \frac{2}{\delta}}{2n}} 2 n l o g ∣ H ∣ + l o g δ 2
最后解释为何我们总是执着于计算期望误差与经验误差之间的差距 —— 因为两者的差距就代表了模型泛化能力的强弱。我们希望在自己有限的数据集上训练得到的模型，能够顺利得到应用，即便是面对它未曾接触过的输入，也可以给出正确合理的答案，这就是泛化
