《机器学习理论导引》笔记目录
0 破事水 感言
当我在完成本科毕业论文后更新完第三章的时候，我无论如何都不会想到 : 关于第四章的学习，会拖延到半年以后。这就使得事情变得有些滑稽了起来 : 前三章的具体内容在我的脑海里已经烟消云散，但是我仍要努力地去 “狗尾续貂”。
其实在过年那会儿，在反思过去的一年的时候，我就努力在想 : 系统性地去了解理论机器学习，到底有什么好处? 虽然自己做的也确实需要用到一些理论机器学习，但是并不需要这么多、这么全面。而且更难受的是 : 好不容易努力地将这些高深到不属于自己脑子的知识塞进脑子不久，就会因为不太使用而逐渐淡忘 。那到底是为什么呢?
最后我想通了 : 很多知识我们其实并不能完全理解，但我们仍要努力学习，这不是一种 “夸父逐日” 的壮烈与惨烈，而是要带着一种更高知识层次的视野来看待问题 : 虽然可能不熟练，但是我知道从哪些角度来思考问题 ，真的如果需要具体解决这些问题的时候我也知道在哪里去寻找我想要的资料 来辅助我解决需要解决的问题。这也许就是我们孜孜不倦 “活到老，学到老” 的更深层的好处吧。
最后，还是感谢组里的师兄师姐留下的宝贵的 PPT 资料，有时候真的感觉很多东西有种薪火相传的感觉 。我也会不断改进自己的排版使得大家都能看得更清晰明了。冲呀我的小空调毯!
0.1 来点这一章的引言 (这一章的知识在解决什么问题呢?)
判断一个学习算法的性能优劣，我们往往通过衡量其泛化误差的大小。第 2 章介绍的 PAC 学习理论告诉我们，学习算法的泛化误差与假设空间 H \mathcal{H} H m m m D \mathcal{D} D
同时非常符合直觉地，我们研究一个量会考察其上界与下届那样，我们也从泛化误差上界和下界分别展开讨论。
4.1 泛化误差上界
4.1.1 有限假设空间
定义回顾
概念 c c c : 从样本空间 X \mathcal{X} X Y \mathcal{Y} Y 假设空间 H \mathcal{H} H : 学习算法考虑的所有可能概念的集合根据假设空间内概念的数目，我们把假设空间分为有限假设空间 和无限假设空间
根据目标概念 c c c H \mathcal{H} H H \mathcal{H} H 可分 H \mathcal{H} H 和不可分 H \mathcal{H} H
可分有限假设空间
定义 2.2.PAC 可学
令 m m m D \mathcal{D} D 0 < ϵ , δ < 1 0<\epsilon,\delta<1 0 < ϵ , δ < 1 D \mathcal{D} D L \mathcal{L} L p o l y ( ⋅ , ⋅ , ⋅ , ⋅ ) poly(\cdot,\cdot,\cdot,\cdot) p o l y ( ⋅ , ⋅ , ⋅ , ⋅ ) m ⩾ p o l y ( 1 ϵ , 1 δ , size ( x ) , size ( c ) ) m\geqslant poly(\frac{1}{\epsilon},\frac{1}{\delta},\text{size}(x),\text{size}(c)) m ⩾ p o l y ( ϵ 1 , δ 1 , size ( x ) , size ( c )) L \mathcal{L} L H \mathcal{H} H C \mathcal{C} C C \mathcal{C} C H \mathcal{H} H C \mathcal{C} C
根据定义 2.2 ，只要我们能找到一个下界，使得在训练集规模大于这个下界的时候，学习算法 L \mathcal{L} L 1 − δ 1-\delta 1 − δ ϵ \epsilon ϵ H \mathcal{H} H C \mathcal{C} C
定理 4.1
令 H \mathcal{H} H D D D D \mathcal{D} D m m m L \mathfrak{L} L D D D h ∈ H h\in \mathcal{H} h ∈ H 0 < ϵ , δ < 1 0<\epsilon,\delta<1 0 < ϵ , δ < 1 m ⩾ 1 ϵ ( ln ∣ H ∣ + ln 1 δ ) m\geqslant\frac{1}{\epsilon}\left(\ln\lvert\mathcal{H}\rvert+\ln\frac{1}{\delta}\right) m ⩾ ϵ 1 ( ln ∣ H ∣ + ln δ 1 )
P ( E ( h ) ⩽ ϵ ) ⩾ 1 − δ P(E(h) \leqslant \epsilon) \geqslant 1-\delta P ( E ( h ) ⩽ ϵ ) ⩾ 1 − δ 即 E ( h ) ⩽ ϵ E(h)\leqslant\epsilon E ( h ) ⩽ ϵ 1 − δ 1-\delta 1 − δ
证明
如果 h h h ϵ \epsilon ϵ
P ( ∃ h ∈ H : E ( h ) > ϵ ∧ E ^ ( h ) = 0 ) P(\exists h \in \mathcal{H}: E(h)>\epsilon \wedge \widehat{E}(h)=0) P ( ∃ h ∈ H : E ( h ) > ϵ ∧ E ( h ) = 0 ) 我们只需证明这一事件发生的概率至多为 δ \delta δ 1 − δ 1 − \delta 1 − δ
根据联合界不等式，P ( X ∪ Y ) ⩽ P ( X ) + P ( Y ) P(X\cup Y)\leqslant P(X)+P(Y) P ( X ∪ Y ) ⩽ P ( X ) + P ( Y )
P ( ∃ h ∈ H : E ( h ) > ϵ ∧ E ^ ( h ) = 0 ) ⩽ ∑ h ∈ H P ( E ( h ) > ϵ ∧ E ^ ( h ) = 0 ) < ∣ H ∣ ( 1 − ϵ ) m \begin{aligned}
P(\exists h \in \mathcal{H}: E(h)>\epsilon \wedge \widehat{E}(h)=0) & \leqslant \sum_{h \in \mathcal{H}} P(E(h)>\epsilon \wedge \widehat{E}(h)=0) \\
& <|\mathcal{H}|(1-\epsilon)^m
\end{aligned} P ( ∃ h ∈ H : E ( h ) > ϵ ∧ E ( h ) = 0 ) ⩽ h ∈ H ∑ P ( E ( h ) > ϵ ∧ E ( h ) = 0 ) < ∣ H ∣ ( 1 − ϵ ) m E ( h ) > ϵ E(h)>\epsilon E ( h ) > ϵ h h h D \mathcal{D} D ϵ \epsilon ϵ D D D D \mathcal{D} D
现在只需要保证小于号最右端不大于 δ \delta δ ( 1 − ϵ ) m ⩽ e − ϵ m (1−\epsilon)^m\leqslant e^{−\epsilon m} ( 1 − ϵ ) m ⩽ e − ϵ m m ⩾ 1 ϵ ( ln ∣ H ∣ + ln 1 δ ) m\geqslant\frac{1}{\epsilon}\left(\ln\lvert\mathcal{H}\rvert+\ln\frac{1}{\delta}\right) m ⩾ ϵ 1 ( ln ∣ H ∣ + ln δ 1 )
∣ H ∣ ( 1 − ϵ ) m ⩽ ∣ H ∣ e − ϵ m ⩽ δ |\mathcal{H}|(1-\epsilon)^m \leqslant|\mathcal{H}| e^{-\epsilon m} \leqslant \delta ∣ H ∣ ( 1 − ϵ ) m ⩽ ∣ H ∣ e − ϵ m ⩽ δ 从而可知 P ( E ( h ) > ϵ ) ⩽ δ P(E(h)>\epsilon)\leqslant\delta P ( E ( h ) > ϵ ) ⩽ δ P ( E ( h ) ⩽ ϵ ) ⩾ 1 − δ P(E(h)\leqslant\epsilon)\geqslant 1 −\delta P ( E ( h ) ⩽ ϵ ) ⩾ 1 − δ
引理 2.1
若训练集 D D D m m m D \mathcal{D} D h ∈ H h\in\mathcal{H} h ∈ H
P ( ∣ E ( h ) − E ^ ( h ) ∣ ⩾ ϵ ) ⩽ 2 exp ( − 2 m ϵ 2 ) P(|E(h)-\hat{E}(h)| \geqslant \epsilon) \leqslant 2 \exp \left(-2 m \epsilon^2\right) P ( ∣ E ( h ) − E ^ ( h ) ∣ ⩾ ϵ ) ⩽ 2 exp ( − 2 m ϵ 2 ) 定理 4.2
令 H \mathcal{H} H D D D D \mathcal{D} D m m m h ∈ H h\in\mathcal{H} h ∈ H 0 < δ < 1 0<\delta<1 0 < δ < 1
P ( ∣ E ( h ) − E ^ ( h ) ∣ ⩽ ln ∣ H ∣ + ln ( 2 / δ ) 2 m ) ⩾ 1 − δ P\left(|E(h)-\hat{E}(h)| \leqslant \sqrt{\frac{\ln |\mathcal{H}|+\ln (2 / \delta)}{2 m}}\right) \geqslant 1-\delta P ( ∣ E ( h ) − E ^ ( h ) ∣ ⩽ 2 m ln ∣ H ∣ + ln ( 2/ δ ) ) ⩾ 1 − δ 证明
将 H \mathcal{H} H h 1 , h 2 , … , h ∣ H ∣ h1,h2,\ldots,h_{\lvert\mathcal{H}\rvert} h 1 , h 2 , … , h ∣ H ∣
P ( ∃ h ∈ H : ∣ E ^ ( h ) − E ( h ) ∣ > ϵ ) = P ( ( ∣ E ^ ( h 1 ) − E ( h 1 ) ∣ > ϵ ) ∨ ⋯ ∨ ( ∣ E ^ ( h ∣ H ∣ ) − E ( h ∣ H ∣ ∣ ⟩ ϵ ) ) ⩽ ∑ h ∈ H P ( ∣ E ^ ( h ) − E ( h ) ∣ > ϵ ) \begin{aligned}
& P(\exists h \in \mathcal{H}:|\widehat{E}(h)-E(h)|>\epsilon) \\
= & P\left(\left(\left|\hat{E}\left(h_1\right)-E\left(h_1\right)\right|>\epsilon\right) \vee \cdots \vee\left(\mid \widehat{E}\left(h_{|\mathcal{H}|}\right)-E\left(h_{|\mathcal{H}|}|\rangle \epsilon\right)\right)\right. \\
\leqslant & \sum_{h \in \mathcal{H}} P(|\widehat{E}(h)-E(h)|>\epsilon)
\end{aligned} = ⩽ P ( ∃ h ∈ H : ∣ E ( h ) − E ( h ) ∣ > ϵ ) P ( ( ∣ ∣ E ^ ( h 1 ) − E ( h 1 ) ∣ ∣ > ϵ ) ∨ ⋯ ∨ ( ∣ E ( h ∣ H ∣ ) − E ( h ∣ H ∣ ∣ ⟩ ϵ ) ) h ∈ H ∑ P ( ∣ E ( h ) − E ( h ) ∣ > ϵ ) 基于引理 2.1，令 2 exp ( − 2 m ϵ 2 ) = δ ∣ H ∣ 2\exp(-2m\epsilon^2)=\frac{\delta}{\lvert\mathcal{H}\rvert} 2 exp ( − 2 m ϵ 2 ) = ∣ H ∣ δ
P ( ∣ E ^ ( h ) − E ( h ) ∣ > ϵ ) ⩽ ∑ h ∈ H δ ∣ H ∣ = δ P(|\widehat{E}(h)-E(h)|>\epsilon)
\leqslant \sum_{h \in \mathcal{H}} \frac{\delta}{\lvert\mathcal{H}\rvert}= \delta P ( ∣ E ( h ) − E ( h ) ∣ > ϵ ) ⩽ h ∈ H ∑ ∣ H ∣ δ = δ 由 2 exp ( − 2 m ϵ 2 ) = δ ∣ H ∣ 2\exp(-2m\epsilon^2)=\frac{\delta}{\lvert\mathcal{H}\rvert} 2 exp ( − 2 m ϵ 2 ) = ∣ H ∣ δ ϵ = ln ∣ H ∣ + ln ( 2 / δ ) 2 m \epsilon=\sqrt{\frac{\ln |\mathcal{H}|+\ln (2 / \delta)}{2 m}} ϵ = 2 m l n ∣ H ∣ + l n ( 2/ δ )
4.1.2 无限假设空间
增长函数
对于 m ∈ N m\in\mathbb{N} m ∈ N H \mathcal{H} H Π H ( m ) \Pi_\mathcal{H}(m) Π H ( m )
Π H ( m ) = max { x 1 , … , x m } ⊂ X ∣ { ( h ( x 1 ) , … , h ( x m ) ) ∣ h ∈ H } ∣ \Pi_{\mathcal{H}}(m)=\max _{\left\{\mathbf{x}_1, \ldots, \mathbf{x}_m\right\} \subset \mathcal{X}}\left|\left\{\left(h\left(\boldsymbol{x}_1\right), \ldots, h\left(\boldsymbol{x}_m\right)\right) \mid h \in \mathcal{H}\right\}\right| Π H ( m ) = { x 1 , … , x m } ⊂ X max ∣ { ( h ( x 1 ) , … , h ( x m ) ) ∣ h ∈ H } ∣ 增长函数 Π H ( m ) \Pi_\mathcal{H}(m) Π H ( m ) H \mathcal{H} H m m m
定义 3.1. VC维
VC维 : 假设空间 H \mathcal{H} H H \mathcal{H} H
V C ( H ) = max { m : Π H ( m ) = 2 m } V C(\mathcal{H})=\max \left\{m: \Pi_{\mathcal{H}}(m)=2^m\right\} V C ( H ) = max { m : Π H ( m ) = 2 m } Chebyshev 不等式(回顾)
∀ ϵ > 0 \forall\epsilon>0 ∀ ϵ > 0
P ( ∣ X − E [ X ] ∣ ⩾ ϵ ) ⩽ V [ X ] ϵ 2 P(\lvert X-\mathbb{E}[X]\rvert\geqslant\epsilon)\leqslant\frac{\mathbb{V}[X]}{\epsilon^2} P (∣ X − E [ X ]∣ ⩾ ϵ ) ⩽ ϵ 2 V [ X ] 引理 4.1
对于假设空间 H , h ∈ H , m ∈ N \mathcal{H},h\in\mathcal{H},m\in\mathbb{N} H , h ∈ H , m ∈ N 0 < ϵ < 1 0<\epsilon<1 0 < ϵ < 1 m ⩾ 2 ϵ 2 m\geqslant\frac{2}{\epsilon^2} m ⩾ ϵ 2 2
P ( ∣ E ( h ) − E ^ ( h ) ∣ > ϵ ) ⩽ 4 Π H ( 2 m ) exp ( − m ϵ 2 8 ) P(|E(h)-\widehat{E}(h)|>\epsilon) \leqslant 4 \Pi_{\mathcal{H}}(2 m) \exp \left(-\frac{m \epsilon^2}{8}\right) P ( ∣ E ( h ) − E ( h ) ∣ > ϵ ) ⩽ 4 Π H ( 2 m ) exp ( − 8 m ϵ 2 ) 证明
考虑两个大小均为 m m m D \mathcal{D} D D D D D ′ D' D ′
P ( sup h ∈ H ∣ E ^ D ( h ) − E ^ D ′ ( h ) ∣ ⩾ 1 2 ϵ ) ⩾ 1 2 P ( sup h ∈ H ∣ E ( h ) − E ^ D ( h ) ∣ > ϵ ) P\left(\sup _{h \in \mathcal{H}}\left|\hat{E}_D(h)-\widehat{E}_{D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right) \geqslant \frac{1}{2} P\left(\sup _{h \in \mathcal{H}}\left|E(h)-\widehat{E}_D(h)\right|>\epsilon\right) P ( h ∈ H sup ∣ ∣ E ^ D ( h ) − E D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) ⩾ 2 1 P ( h ∈ H sup ∣ ∣ E ( h ) − E D ( h ) ∣ ∣ > ϵ ) 用 Q Q Q
Q = { D ∼ D m ∣ sup h ∈ H ∣ E ( h ) − E ^ D ( h ) ∣ > ϵ } Q=\left\{D\sim\mathcal{D}^m|\sup_{h\in\mathcal{H}}\left\rvert E(h)-\widehat{E}_D(h)\right\rvert>\epsilon\right\} Q = { D ∼ D m ∣ h ∈ H sup ∣ ∣ E ( h ) − E D ( h ) ∣ ∣ > ϵ } 计算可得
P ( sup h ∈ H ∣ E ^ D ( h ) − E ^ D ′ ( h ) ∣ ⩾ 1 2 ϵ ) = E D , D ′ ∼ D m [ I ( sup h ∈ H ∣ E ^ D ( h ) − E ^ D ′ ( h ) ∣ ⩾ 1 2 ϵ ) ] = E D ∼ D m [ E D ′ ∼ D m [ I ( sup h ∈ H ∣ E ^ D ( h ) − E ^ D ′ ( h ) ∣ ⩾ 1 2 ϵ ) ] ] ⩾ E D ∈ Q [ E D ′ ∼ D m [ I ( sup h ∈ H ∣ E ^ D ( h ) − E ^ D ′ ( h ) ∣ ⩾ 1 2 ϵ ) ] ] \begin{aligned}
& P\left(\sup _{h \in \mathcal{H}}\left|\widehat{E}_D(h)-\widehat{E}_{D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right) \\
= & \mathbb{E}_{D, D^{\prime} \sim \mathcal{D}^m}\left[\mathbb{I}\left(\sup _{h \in \mathcal{H}}\left|\widehat{E}_D(h)-\widehat{E}_{D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right)\right] \\
= & \mathbb{E}_{D \sim \mathcal{D}^m}\left[\mathbb{E}_{D^{\prime} \sim \mathcal{D}^m}\left[\mathbb{I}\left(\sup _{h \in \mathcal{H}}\left|\widehat{E}_D(h)-\widehat{E}_{D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right)\right]\right] \\
\geqslant & \mathbb{E}_{D \in Q}\left[\mathbb{E}_{D^{\prime} \sim \mathcal{D}^m}\left[\mathbb{I}\left(\sup _{h \in \mathcal{H}}\left|\widehat{E}_D(h)-\widehat{E}_{D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right)\right]\right]
\end{aligned} = = ⩾ P ( h ∈ H sup ∣ ∣ E D ( h ) − E D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) E D , D ′ ∼ D m [ I ( h ∈ H sup ∣ ∣ E D ( h ) − E D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) ] E D ∼ D m [ E D ′ ∼ D m [ I ( h ∈ H sup ∣ ∣ E D ( h ) − E D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) ] ] E D ∈ Q [ E D ′ ∼ D m [ I ( h ∈ H sup ∣ ∣ E D ( h ) − E D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) ] ] 根据 Q Q Q D ∈ Q D\in Q D ∈ Q h 0 ∈ H h_0\in\mathcal{H} h 0 ∈ H ∣ E ( h 0 ) − E ^ D ( h 0 ) ∣ > ϵ \lvert E(h_0)-\widehat{E}_D(h_0)\rvert>\epsilon ∣ E ( h 0 ) − E D ( h 0 )∣ > ϵ h 0 h_0 h 0
E D ′ ∼ D m [ I ( sup h ∈ H ∣ E ^ D ( h ) − E ^ D ′ ( h ) ∣ ⩾ 1 2 ϵ ) ] ⩾ E D ′ ∼ D m [ I ( ∣ E ^ D ( h 0 ) − E ^ D ′ ( h 0 ) ∣ ⩾ 1 2 ϵ ) ] = E D ′ ∼ D m [ I ( ∣ E ^ D ( h 0 ) − E ( h 0 ) − ( E ^ D ′ ( h 0 ) − E ( h 0 ) ) ∣ ⩾ 1 2 ϵ ) ] ⩾ E D ′ ∼ D m [ I ( ∣ E ^ D ( h 0 ) − E ( h 0 ) ∣ − ∣ E ^ D ′ ( h 0 ) − E ( h 0 ) ∣ ⩾ 1 2 ϵ ) ] \begin{aligned}
& \mathbb{E}_{D^{\prime} \sim \mathcal{D}^m}\left[\mathbb{I}\left(\sup _{h \in \mathcal{H}}\left|\widehat{E}_D(h)-\widehat{E}_{D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right)\right] \\
\geqslant \mathbb{E}_{D^{\prime} \sim \mathcal{D}^m} & {\left[\mathbb{I}\left(\left|\widehat{E}_D\left(h_0\right)-\widehat{E}_{D^{\prime}}\left(h_0\right)\right| \geqslant \frac{1}{2} \epsilon\right)\right] } \\
=\mathbb{E}_{D^{\prime} \sim \mathcal{D}^m} & {\left[\mathbb{I}\left(\left|\widehat{E}_D\left(h_0\right)-E\left(h_0\right)-\left(\widehat{E}_{D^{\prime}}\left(h_0\right)-E\left(h_0\right)\right)\right| \geqslant \frac{1}{2} \epsilon\right)\right] } \\
\geqslant \mathbb{E}_{D^{\prime} \sim \mathcal{D}^m} & {\left[\mathbb{I}\left(\left|\widehat{E}_D\left(h_0\right)-E\left(h_0\right)\right|-\left|\widehat{E}_{D^{\prime}}\left(h_0\right)-E\left(h_0\right)\right| \geqslant \frac{1}{2} \epsilon\right)\right] }
\end{aligned} ⩾ E D ′ ∼ D m = E D ′ ∼ D m ⩾ E D ′ ∼ D m E D ′ ∼ D m [ I ( h ∈ H sup ∣ ∣ E D ( h ) − E D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) ] [ I ( ∣ ∣ E D ( h 0 ) − E D ′ ( h 0 ) ∣ ∣ ⩾ 2 1 ϵ ) ] [ I ( ∣ ∣ E D ( h 0 ) − E ( h 0 ) − ( E D ′ ( h 0 ) − E ( h 0 ) ) ∣ ∣ ⩾ 2 1 ϵ ) ] [ I ( ∣ ∣ E D ( h 0 ) − E ( h 0 ) ∣ ∣ − ∣ ∣ E D ′ ( h 0 ) − E ( h 0 ) ∣ ∣ ⩾ 2 1 ϵ ) ] ∣ E ^ D ( h 0 ) − E ( h 0 ) ∣ > ϵ \left|\widehat{E}_D\left(h_0\right)-E\left(h_0\right)\right|>\epsilon ∣ ∣ E D ( h 0 ) − E ( h 0 ) ∣ ∣ > ϵ ∣ E ^ D ′ ( h 0 ) − E ( h 0 ) ∣ ≤ ϵ 2 \left|\widehat{E}_{D^{\prime}}\left(h_0\right)-E\left(h_0\right)\right|\le\frac{\epsilon}{2} ∣ ∣ E D ′ ( h 0 ) − E ( h 0 ) ∣ ∣ ≤ 2 ϵ ∣ E ^ D ( h 0 ) − E ( h 0 ) ∣ − ∣ E ^ D ′ ( h 0 ) − E ( h 0 ) ∣ ⩾ 1 2 ϵ \left|\widehat{E}_D\left(h_0\right)-E\left(h_0\right)\right|-\left|\widehat{E}_{D^{\prime}}\left(h_0\right)-E\left(h_0\right)\right| \geqslant \frac{1}{2} \epsilon ∣ ∣ E D ( h 0 ) − E ( h 0 ) ∣ ∣ − ∣ ∣ E D ′ ( h 0 ) − E ( h 0 ) ∣ ∣ ⩾ 2 1 ϵ
E D ′ ∼ D m [ I ( sup h ∈ H ∣ E ^ D ( h ) − E ^ D ′ ( h ) ∣ ⩾ 1 2 ϵ ) ] ⩾ E D ′ ∼ D m [ I ( ∣ E ^ D ( h 0 ) − E ( h 0 ) ∣ − ∣ ( E ^ D ′ ( h 0 ) − E ( h 0 ) ) ∣ ⩾ 1 2 ϵ ) ] ⩾ E D ′ ∼ D m [ I ( ∣ E ^ D ′ ( h 0 ) − E ( h 0 ) ∣ ⩽ 1 2 ϵ ) ] = P ( ∣ E ^ D ′ ( h 0 ) − E ( h 0 ) ∣ ⩽ 1 2 ϵ ) = 1 − P ( ∣ E ^ D ′ ( h 0 ) − E ( h 0 ) ∣ > 1 2 ϵ ) \begin{aligned}
& \mathbb{E}_{D^{\prime} \sim \mathcal{D}^m}\left[\mathbb{I}\left(\sup _{h \in \mathcal{H}}\left|\widehat{E}_D(h)-\widehat{E}_{D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right)\right] \\
\geqslant & \mathbb{E}_{D^{\prime} \sim \mathcal{D}^m}\left[\mathbb{I}\left(\left|\widehat{E}_D\left(h_0\right)-E\left(h_0\right)\right|-\left|\left(\widehat{E}_{D^{\prime}}\left(h_0\right)-E\left(h_0\right)\right)\right| \geqslant \frac{1}{2} \epsilon\right)\right] \\
\geqslant & \mathbb{E}_{D^{\prime} \sim \mathcal{D}^m}\left[\mathbb{I}\left(\left|\widehat{E}_{D^{\prime}}\left(h_0\right)-E\left(h_0\right)\right| \leqslant \frac{1}{2} \epsilon\right)\right] \\
= & P\left(\left|\widehat{E}_{D^{\prime}}\left(h_0\right)-E\left(h_0\right)\right| \leqslant \frac{1}{2} \epsilon\right) \\
= & 1-P\left(\left|\widehat{E}_{D^{\prime}}\left(h_0\right)-E\left(h_0\right)\right|>\frac{1}{2} \epsilon\right)
\end{aligned} ⩾ ⩾ = = E D ′ ∼ D m [ I ( h ∈ H sup ∣ ∣ E D ( h ) − E D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) ] E D ′ ∼ D m [ I ( ∣ ∣ E D ( h 0 ) − E ( h 0 ) ∣ ∣ − ∣ ∣ ( E D ′ ( h 0 ) − E ( h 0 ) ) ∣ ∣ ⩾ 2 1 ϵ ) ] E D ′ ∼ D m [ I ( ∣ ∣ E D ′ ( h 0 ) − E ( h 0 ) ∣ ∣ ⩽ 2 1 ϵ ) ] P ( ∣ ∣ E D ′ ( h 0 ) − E ( h 0 ) ∣ ∣ ⩽ 2 1 ϵ ) 1 − P ( ∣ ∣ E D ′ ( h 0 ) − E ( h 0 ) ∣ ∣ > 2 1 ϵ ) E ( h 0 ) E(h_0) E ( h 0 ) E ^ D ′ ( h 0 ) \widehat{E}_{D^{\prime}}\left(h_0\right) E D ′ ( h 0 ) D D D
P ( ∣ E ^ D ′ ( h 0 ) − E ( h 0 ) ∣ > 1 2 ϵ ) ⩽ 4 ( 1 − E ( h 0 ) E ( h 0 ) ) ϵ 2 m ⩽ 1 ϵ 2 m P\left(\left|\widehat{E}_{D^{\prime}}\left(h_0\right)-E\left(h_0\right)\right|>\frac{1}{2} \epsilon\right)\\ \leqslant \frac{4(1-E(h_0)E(h_0))}{\epsilon^2m}\leqslant \frac{1}{\epsilon^2m} P ( ∣ ∣ E D ′ ( h 0 ) − E ( h 0 ) ∣ ∣ > 2 1 ϵ ) ⩽ ϵ 2 m 4 ( 1 − E ( h 0 ) E ( h 0 )) ⩽ ϵ 2 m 1 当 m ⩾ 2 ϵ 2 m\geqslant\frac{2}{\epsilon^2} m ⩾ ϵ 2 2 P ( ∣ E ^ D ′ ( h 0 ) − E ( h 0 ) ∣ > 1 2 ϵ ) ⩽ 1 2 P\left(\left|\widehat{E}_{D^{\prime}}\left(h_0\right)-E\left(h_0\right)\right|>\frac{1}{2} \epsilon\right)\leqslant\frac{1}{2} P ( ∣ ∣ E D ′ ( h 0 ) − E ( h 0 ) ∣ ∣ > 2 1 ϵ ) ⩽ 2 1
P ( sup h ∈ H ∣ E ^ D ( h ) − E ^ D ′ ( h ) ∣ ⩾ 1 2 ϵ ) ⩾ E D ∈ Q [ E D ′ ∼ D m [ I ( sup h ∈ H ∣ E ^ D ( h ) − E ^ D ′ ( h ) ∣ ⩾ 1 2 ϵ ) ] ] ⩾ E D ∈ Q [ 1 − P ( ∣ E ^ D ′ ( h 0 ) − E ( h 0 ) ∣ > 1 2 ϵ ) ] ⩾ E D ∈ Q [ 1 2 ] = 1 2 P ( sup h ∈ H ∣ E ( h ) − E ^ D ( h ) ∣ > ϵ ) \begin{aligned}
& P\left(\sup _{h \in \mathcal{H}}\left|\widehat{E}_D(h)-\widehat{E}_{D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right) \\
\geqslant & \mathbb{E}_{D \in Q}\left[\mathbb{E}_{D^{\prime} \sim D^m}\left[\mathbb{I}\left(\sup _{h \in \mathcal{H}}\left|\widehat{E}_D(h)-\widehat{E}_{D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right)\right]\right] \\
\geqslant & \mathbb{E}_{D \in Q}\left[1-P\left(\left|\widehat{E}_{D^{\prime}}\left(h_0\right)-E\left(h_0\right)\right|>\frac{1}{2} \epsilon\right)\right] \\
\geqslant & \mathbb{E}_{D \in Q}\left[\frac{1}{2}\right] = \frac{1}{2} P\left(\sup _{h \in \mathcal{H}}\left|E(h)-\widehat{E}_D(h)\right|>\epsilon\right)
\end{aligned} ⩾ ⩾ ⩾ P ( h ∈ H sup ∣ ∣ E D ( h ) − E D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) E D ∈ Q [ E D ′ ∼ D m [ I ( h ∈ H sup ∣ ∣ E D ( h ) − E D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) ] ] E D ∈ Q [ 1 − P ( ∣ ∣ E D ′ ( h 0 ) − E ( h 0 ) ∣ ∣ > 2 1 ϵ ) ] E D ∈ Q [ 2 1 ] = 2 1 P ( h ∈ H sup ∣ ∣ E ( h ) − E D ( h ) ∣ ∣ > ϵ ) 由于 D D D D ′ D' D ′ D \mathcal{D} D m m m D D D D ′ D′ D ′ 2 m 2m 2 m
令 T i T_i T i 2 m 2m 2 m ( 2 m ) ! (2m)! ( 2 m )! T i T_i T i T i D T_iD T i D 2 m 2m 2 m T i T_i T i m m m T i D ′ T_i D' T i D ′ 2 m 2m 2 m T i T_i T i m m m D , D ′ D,D' D , D ′ T i D T_iD T i D T i D ′ T_i D' T i D ′
P ( sup h ∈ H ∣ E ^ D ( h ) − E ^ D ′ ( h ) ∣ ⩾ 1 2 ϵ ) = E D , D ′ [ I ( sup h ∈ H ∣ E ^ D ( h ) − E ^ D ′ ( h ) ∣ ⩾ 1 2 ϵ ) ] = E D , D ′ [ 1 ( 2 m ) ! ∑ i = 1 ( 2 m ) ! I ( sup h ∈ H ∣ E ^ T i D ( h ) − E ^ T i D ′ ( h ) ∣ ⩾ 1 2 ϵ ) ] = E D , D ′ [ 1 ( 2 m ) ! ∑ i = 1 ( 2 m ) ! sup h ∈ H I ( ∣ E ^ T i D ( h ) − E ^ T i D ′ ( h ) ∣ ⩾ 1 2 ϵ ) ] ⩽ E D , D ′ [ ∑ h ∈ H ∣ D + D ′ 1 ( 2 m ) ! ∑ i = 1 ( 2 m ) ! I ( ∣ E ^ T i D ( h ) − E ^ T i D ′ ( h ) ∣ ⩾ 1 2 ϵ ) ] \begin{aligned}
& P\left(\sup _{h \in \mathcal{H}}\left|\widehat{E}_D(h)-\widehat{E}_{D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right) \\
& =\mathbb{E}_{D, D^{\prime}}\left[\mathbb{I}\left(\sup _{h \in \mathcal{H}}\left|\widehat{E}_D(h)-\widehat{E}_{D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right)\right] \\
& =\mathbb{E}_{D, D^{\prime}}\left[\frac{1}{(2 m) !} \sum_{i=1}^{(2 m) !} \mathbb{I}\left(\sup _{h \in \mathcal{H}}\left|\widehat{E}_{T_i D}(h)-\widehat{E}_{T_i D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right)\right] \\
& =\mathbb{E}_{D, D^{\prime}}\left[\frac{1}{(2 m) !} \sum_{i=1}^{(2 m) !} \sup _{h \in \mathcal{H}} \mathbb{I}\left(\left|\widehat{E}_{T_i D}(h)-\widehat{E}_{T_i D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right)\right] \\
& \leqslant \mathbb{E}_{D, D^{\prime}}\left[\sum_{h \in \mathcal{H}_{\mid D+D^{\prime}}} \frac{1}{(2 m) !} \sum_{i=1}^{(2 m) !} \mathbb{I}\left(\left|\widehat{E}_{T_i D}(h)-\widehat{E}_{T_i D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right)\right] \\
&
\end{aligned} P ( h ∈ H sup ∣ ∣ E D ( h ) − E D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) = E D , D ′ [ I ( h ∈ H sup ∣ ∣ E D ( h ) − E D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) ] = E D , D ′ ⎣ ⎡ ( 2 m )! 1 i = 1 ∑ ( 2 m )! I ( h ∈ H sup ∣ ∣ E T i D ( h ) − E T i D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) ⎦ ⎤ = E D , D ′ ⎣ ⎡ ( 2 m )! 1 i = 1 ∑ ( 2 m )! h ∈ H sup I ( ∣ ∣ E T i D ( h ) − E T i D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) ⎦ ⎤ ⩽ E D , D ′ ⎣ ⎡ h ∈ H ∣ D + D ′ ∑ ( 2 m )! 1 i = 1 ∑ ( 2 m )! I ( ∣ ∣ E T i D ( h ) − E T i D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) ⎦ ⎤ 其中 H ∣ D + D ′ \mathcal{H}_{\mid D+D^{\prime}} H ∣ D + D ′ H \mathcal{H} H D + D ′ D+D^{\prime} D + D ′
1 ( 2 m ) ! ∑ i = 1 ( 2 m ) ! I ( ∣ E ^ T i D ( h ) − E ^ T i D ′ ( h ) ∣ ⩾ 1 2 ϵ ) = ∑ k ∈ [ I ] ( l k ) ( 2 m − l m − k ) ( 2 m m ) ≤ 2 exp ( − ϵ 2 m 8 ) s.t. ∣ 2 k / m − l / m ∣ ⩾ ϵ / 2 \begin{aligned}
& \frac{1}{(2 m) !} \sum_{i=1}^{(2 m) !} \mathbb{I}\left(\left|\widehat{E}_{T_i D}(h)-\widehat{E}_{T_i D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right)=\sum_{k \in[I]} \frac{\left(\begin{array}{c}
l \\
k
\end{array}\right)\left(\begin{array}{c}
2 m-l \\
m-k
\end{array}\right)}{\left(\begin{array}{c}
2 m \\
m
\end{array}\right)} \\
& \leq 2 \exp \left(-\frac{\epsilon^2 m}{8}\right) \\
&\\
&\text{s.t.} |2 \mathrm{k} / \mathrm{m}-\mathrm{l} / \mathrm{m}| \geqslant \epsilon / 2&
\end{aligned} ( 2 m )! 1 i = 1 ∑ ( 2 m )! I ( ∣ ∣ E T i D ( h ) − E T i D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) = k ∈ [ I ] ∑ ( 2 m m ) ( l k ) ( 2 m − l m − k ) ≤ 2 exp ( − 8 ϵ 2 m ) s.t. ∣2 k / m − l / m ∣ ⩾ ϵ /2 k k k T i D T_iD T i D h h h m − k m−k m − k ( l k ) \left(\begin{array}{c}
l \\
k
\end{array}\right) ( l k ) l l l k k k ( 2 m − l m − k ) \left(\begin{array}{c}
2 m-l \\
m-k
\end{array}\right) ( 2 m − l m − k ) 2 m − l 2m−l 2 m − l m − k m−k m − k
P ( ∣ E ( h ) − E ^ D ( h ) ∣ > ϵ ) ⩽ P ( sup h ∈ H ∣ E ( h ) − E ^ ( h ) ∣ > ϵ ) ⩽ 2 P ( sup h ∈ H ∣ E ^ D ( h ) − E ^ D ′ ( h ) ∣ ⩾ 1 2 ϵ ) ⩽ 2 E D , D ′ [ ∑ h ∈ H ∣ D + D ′ 1 ( 2 m ) ! ∑ i = 1 ( 2 m ) ! I ( ∣ E ^ T i D ( h ) − E ^ T i D ′ ( h ) ∣ ⩾ 1 2 ϵ ) ] ⩽ 4 ∣ H ∣ D + D ′ ∣ exp ( − ϵ 2 m 8 ) ⩽ 4 Π H ( 2 m ) exp ( − m ϵ 2 8 ) \begin{aligned}
& P\left(\left|E(h)-\widehat{E}_D(h)\right|>\epsilon\right) \\
\leqslant & P\left(\sup _{h \in \mathcal{H}}|E(h)-\widehat{E}(h)|>\epsilon\right) \\
\leqslant & 2 P\left(\sup _{h \in \mathcal{H}}\left|\hat{E}_D(h)-\widehat{E}_{D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right) \\
\leqslant & 2 \mathbb{E}_{D, D^{\prime}}\left[\sum_{h \in \mathcal{H}_{\mid D+D^{\prime}}} \frac{1}{(2 m) !} \sum_{i=1}^{(2 m) !} \mathbb{I}\left(\left|\widehat{E}_{T_i D}(h)-\widehat{E}_{T_i D^{\prime}}(h)\right| \geqslant \frac{1}{2} \epsilon\right)\right] \\
\leqslant & 4\left|\mathcal{H}_{\mid D+D^{\prime}}\right| \exp \left(-\frac{\epsilon^2 m}{8}\right) \\
\leqslant & 4 \Pi_{\mathcal{H}}(2 m) \exp \left(-\frac{m \epsilon^2}{8}\right)
\end{aligned} ⩽ ⩽ ⩽ ⩽ ⩽ P ( ∣ ∣ E ( h ) − E D ( h ) ∣ ∣ > ϵ ) P ( h ∈ H sup ∣ E ( h ) − E ( h ) ∣ > ϵ ) 2 P ( h ∈ H sup ∣ ∣ E ^ D ( h ) − E D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) 2 E D , D ′ ⎣ ⎡ h ∈ H ∣ D + D ′ ∑ ( 2 m )! 1 i = 1 ∑ ( 2 m )! I ( ∣ ∣ E T i D ( h ) − E T i D ′ ( h ) ∣ ∣ ⩾ 2 1 ϵ ) ⎦ ⎤ 4 ∣ ∣ H ∣ D + D ′ ∣ ∣ exp ( − 8 ϵ 2 m ) 4 Π H ( 2 m ) exp ( − 8 m ϵ 2 ) 引理 4.1 得证。
定理 3.1
若假设空间 H \mathcal{H} H d d d m ⩾ d m\geqslant d m ⩾ d
Π H ( m ) ⩽ ( e ⋅ m d ) d \Pi_\mathcal{H}(m) \leqslant \left(\frac{e\cdot m}{d}\right)^d Π H ( m ) ⩽ ( d e ⋅ m ) d 定理 4.3
若假设空间 H \mathcal{H} H d , h ∈ H d,h\in\mathcal{H} d , h ∈ H m > d m>d m > d 0 < δ < 1 0<\delta<1 0 < δ < 1
P ( ∣ E ( h ) − E ^ ( h ) ∣ ⩽ 8 d ln 2 e m d + 8 ln 4 δ m ) ⩾ 1 − δ P\left(|E(h)-\widehat{E}(h)| \leqslant \sqrt{\frac{8 d \ln \frac{2 e m}{d}+8 \ln \frac{4}{\delta}}{m}}\right) \geqslant 1-\delta P ⎝ ⎛ ∣ E ( h ) − E ( h ) ∣ ⩽ m 8 d ln d 2 e m + 8 ln δ 4 ⎠ ⎞ ⩾ 1 − δ 证明
定理 4.3 等价于 P ( ∣ E ( h ) − E ^ ( h ) ∣ ⩾ 8 d ln 2 e m d + 8 ln 4 δ m ) ⩽ δ P\left(|E(h)-\widehat{E}(h)| \geqslant \sqrt{\frac{8 d \ln \frac{2 e m}{d}+8 \ln \frac{4}{\delta}}{m}}\right) \leqslant \delta P ( ∣ E ( h ) − E ( h ) ∣ ⩾ m 8 d l n d 2 e m + 8 l n δ 4 ) ⩽ δ
ϵ ϵ 8 d ln 2 e m d + 8 ln 4 δ m P ( ∣ E ( h ) − E ^ ( h ) ∣ ⩾ ϵ ) ⩽ 4 Π H ( 2 m ) exp ( − m ϵ 2 8 ) ⩽ 4 ( 2 e m d ) d exp ( − m ϵ 2 8 ) = δ \begin{aligned}
\epsilon & \epsilon \sqrt{\frac{8 d \ln \frac{2 e m}{d}+8 \ln \frac{4}{\delta}}{m}} \\
& P(|E(h)-\widehat{E}(h)| \geqslant \epsilon) \\
\leqslant & 4 \Pi_{\mathcal{H}}(2 m) \exp \left(-\frac{m \epsilon^2}{8}\right) \\
\leqslant & 4\left(\frac{2 e m}{d}\right)^d \exp \left(-\frac{m \epsilon^2}{8}\right) \\
= & \delta
\end{aligned} ϵ ⩽ ⩽ = ϵ m 8 d ln d 2 e m + 8 ln δ 4 P ( ∣ E ( h ) − E ( h ) ∣ ⩾ ϵ ) 4 Π H ( 2 m ) exp ( − 8 m ϵ 2 ) 4 ( d 2 e m ) d exp ( − 8 m ϵ 2 ) δ 基于 Rademacher复杂度的泛化误差界
考记实值函数空间 F : Z ↦ R \mathcal{F}:\mathcal{Z} \mapsto\mathbb{R} F : Z ↦ R Z = { z 1 , … , z m } Z = \left\{z_1,\ldots,z_m\right\} Z = { z 1 , … , z m } z i ∈ Z z_i\in \mathcal{Z} z i ∈ Z
定义 3.3
函数空间 F \mathcal{F} F Z Z Z
R ^ Z ( F ) = E σ [ sup f ∈ F 1 m ∑ i = 1 m σ i f ( z i ) ] \hat{\mathfrak{R}}_Z(\mathcal{F})=\mathbb{E}_{\sigma}\left[\sup_{f\in \mathcal{F}}\frac{1}{m}\sum_{i=1}^m \sigma_i f(z_i)\right] R ^ Z ( F ) = E σ [ f ∈ F sup m 1 i = 1 ∑ m σ i f ( z i ) ] 这里 Z Z Z F \mathcal{F} F Z Z Z
定义 3.4
函数空间 F \mathcal{F} F Z Z Z D \mathcal{D} D
ℜ m ( F ) = E Z ⊂ Z : ∣ Z ∣ = m [ R ^ Z ( F ) ] \Re_m(\mathcal{F})=\mathbb{E}_{Z \subset \mathcal{Z}:|Z|=m}\left[\hat{\mathfrak{R}}_Z(\mathcal{F})\right] ℜ m ( F ) = E Z ⊂ Z : ∣ Z ∣ = m [ R ^ Z ( F ) ] McDiarmid 不等式(回顾)
对 m m m X i ∈ X , i ∈ [ m ] X_i \in \mathcal{X},i \in [m] X i ∈ X , i ∈ [ m ] f : X m → R f : \mathcal{X}^
m \rightarrow \mathbb{R} f : X m → R X i X_i X i ∀ x 1 , … , x m , x i ′ ∈ X \forall x_1,\ldots,x_m, x'_i \in \mathcal{X} ∀ x 1 , … , x m , x i ′ ∈ X
∣ f ( x 1 , . . . , x i , . . . , x m ) − f ( x 1 , . . . , x i ′ , . . . , x m ) ∣ ≤ c i |f(x_1,...,x_i,...,x_m)-f(x_1,...,x_i',...,x_m)|\le c_i ∣ f ( x 1 , ... , x i , ... , x m ) − f ( x 1 , ... , x i ′ , ... , x m ) ∣ ≤ c i 则 ∀ ϵ > 0 \forall\epsilon>0 ∀ ϵ > 0
P ( f ( x 1 , . . . , x i , . . . , x m ) − E [ f ( x 1 , . . . , x i , . . . , x m ) ] ≥ ϵ ) ≤ e − 2 ϵ 2 ∑ i = 1 m c i 2 P ( f ( x 1 , . . . , x i , . . . , x m ) − E [ f ( x 1 , . . . , x i , . . . , x m ) ] ≤ − ϵ ) ≤ e − 2 ϵ 2 ∑ i = 1 m c i 2 P(f(x_1,...,x_i,...,x_m)-\mathbb{E}[f(x_1,...,x_i,...,x_m)]\ge\epsilon)\le e^{-\frac{2\epsilon^2}{\sum_{i=1}^m c_i^2}}\\
P(f(x_1,...,x_i,...,x_m)-\mathbb{E}[f(x_1,...,x_i,...,x_m)]\le-\epsilon)\le e^{-\frac{2\epsilon^2}{\sum_{i=1}^m c_i^2}} P ( f ( x 1 , ... , x i , ... , x m ) − E [ f ( x 1 , ... , x i , ... , x m )] ≥ ϵ ) ≤ e − ∑ i = 1 m c i 2 2 ϵ 2 P ( f ( x 1 , ... , x i , ... , x m ) − E [ f ( x 1 , ... , x i , ... , x m )] ≤ − ϵ ) ≤ e − ∑ i = 1 m c i 2 2 ϵ 2 Jensen 不等式
对任意凸函数f ( ⋅ ) f(\cdot) f ( ⋅ )
f ( E [ X ] ) ≤ E [ f ( X ) ] f(\mathbb{E}[X])\le \mathbb{E}[f(X)] f ( E [ X ]) ≤ E [ f ( X )] 定理 4.4
对于实值函数空间 F : Z ↦ [ 0 , 1 ] \mathcal{F} : \mathcal{Z} \mapsto [0, 1] F : Z ↦ [ 0 , 1 ] D D D m m m Z = { z 1 , z 2 , … , z m } , z i ∈ Z , f ∈ F Z=\{z_1,z_2,\ldots,z_m\},z_i\in\mathcal{Z},f\in\mathcal{F} Z = { z 1 , z 2 , … , z m } , z i ∈ Z , f ∈ F 0 < δ < 1 0<\delta<1 0 < δ < 1 1 − δ 1 −\delta 1 − δ
E [ f ( z ) ] ⩽ 1 m ∑ i = 1 m f ( z i ) + 2 ℜ m ( F ) + ln ( 1 / δ ) 2 m E [ f ( z ) ] ⩽ 1 m ∑ i = 1 m f ( z i ) + 2 R ^ Z ( F ) + 3 ln ( 2 / δ ) 2 m \begin{aligned}
& \mathbb{E}[f(z)] \leqslant \frac{1}{m} \sum_{i=1}^m f\left(z_i\right)+2 \Re_m(\mathcal{F})+\sqrt{\frac{\ln (1 / \delta)}{2 m}} \\
& \mathbb{E}[f(z)] \leqslant \frac{1}{m} \sum_{i=1}^m f\left(z_i\right)+2 \widehat{R}_Z(\mathcal{F})+3 \sqrt{\frac{\ln (2 / \delta)}{2 m}}
\end{aligned} E [ f ( z )] ⩽ m 1 i = 1 ∑ m f ( z i ) + 2 ℜ m ( F ) + 2 m ln ( 1/ δ ) E [ f ( z )] ⩽ m 1 i = 1 ∑ m f ( z i ) + 2 R Z ( F ) + 3 2 m ln ( 2/ δ ) 证明
E ^ Z ( f ) = 1 m ∑ i = 1 m f ( z i ) Φ ( Z ) = sup f ∈ F ( E [ f ] − E ^ Z ( f ) ) \color{blue}\widehat{E}_Z(f)=\frac{1}{m}\sum_{i=1}^m f(z_i)\\
\Phi(Z)=\sup_{f\in\mathcal{F}}\left(\mathbb{E}[f]-\widehat{E}_Z(f)\right)\\ E Z ( f ) = m 1 i = 1 ∑ m f ( z i ) Φ ( Z ) = f ∈ F s u p ( E [ f ] − E Z ( f ) ) Z ′ Z' Z ′ Z Z Z z m ∈ Z z_m\in Z z m ∈ Z z m ′ ∈ Z ′ z'_m\in Z' z m ′ ∈ Z ′
Φ ( Z ′ ) − Φ ( Z ) = sup f ∈ F ( E [ f ] − E ^ Z ′ ( f ) ) − sup f ∈ F ( E [ f ] − E ^ Z ( f ) ) ⩽ sup f ∈ F ( E ^ Z ( f ) − E ^ Z ′ ( f ) ) = sup f ∈ F f ( z m ) − f ( z m ′ ) m ⩽ 1 m \begin{aligned}
& \Phi\left(Z^{\prime}\right)-\Phi(Z) \\
= & \sup _{f \in \mathcal{F}}\left(\mathbb{E}[f]-\widehat{E}_{Z^{\prime}}(f)\right)-\sup _{f \in \mathcal{F}}\left(\mathbb{E}[f]-\widehat{E}_Z(f)\right) \\
\leqslant & \sup _{f \in \mathcal{F}}\left(\widehat{E}_Z(f)-\widehat{E}_{Z^{\prime}}(f)\right) \\
= & \sup _{f \in \mathcal{F}} \frac{f\left(z_m\right)-f\left(z_m^{\prime}\right)}{m}
\leqslant \frac{1}{m}
\end{aligned} = ⩽ = Φ ( Z ′ ) − Φ ( Z ) f ∈ F sup ( E [ f ] − E Z ′ ( f ) ) − f ∈ F sup ( E [ f ] − E Z ( f ) ) f ∈ F sup ( E Z ( f ) − E Z ′ ( f ) ) f ∈ F sup m f ( z m ) − f ( z m ′ ) ⩽ m 1 同理可得
Φ ( Z ) − Φ ( Z ′ ) ⩽ 1 m ∣ Φ ( Z ) − Φ ( Z ′ ) ∣ ⩽ 1 m \Phi(Z)-\Phi(Z')\leqslant\frac{1}{m}\\
\left\lvert\Phi(Z)-\Phi(Z')\right\rvert\leqslant\frac{1}{m} Φ ( Z ) − Φ ( Z ′ ) ⩽ m 1 ∣ Φ ( Z ) − Φ ( Z ′ ) ∣ ⩽ m 1 由 McDiarmid 不等式可得，对于 0 < δ < 1 0<\delta<1 0 < δ < 1
Φ ( Z ) ⩽ E Z [ Φ ( Z ) ] + ln ( 1 / δ ) 2 m \color{blue}\Phi(Z)\leqslant\mathbb{E}_Z[\Phi(Z)]+\sqrt{\frac{\ln(1/\delta)}{2m}}\\ Φ ( Z ) ⩽ E Z [ Φ ( Z )] + 2 m l n ( 1/ δ ) 以至少 1 − δ 1-\delta 1 − δ E Z [ Φ ( Z ) ] \mathbb{E}_Z[\Phi(Z)] E Z [ Φ ( Z )]
E Z [ Φ ( Z ) ] = E Z [ sup f ∈ F ( E [ f ] − E ^ Z ( f ) ) ] = E Z [ sup f ∈ F E Z ′ [ E ^ Z ′ ( f ) − E ^ Z ( f ) ] ] ⩽ E Z , Z ′ [ sup f ∈ F ( E ^ Z ′ ( f ) − E ^ Z ( f ) ) ] = E Z , Z ′ [ sup f ∈ F 1 m ∑ i = 1 m ( f ( z i ) − f ( z i ) ) ] = E σ , Z , Z ′ [ sup f ∈ F 1 m ∑ i = 1 m σ i ( f ( z i ′ ) − f ( z i ) ) ] ⩽ E σ , Z ′ [ sup f ∈ F 1 m ∑ i = 1 m σ i f ( z i ) ] + E σ , Z [ sup f ∈ F 1 m ∑ i = 1 m − σ i f ( z i ) ] = 2 E σ , Z [ sup f ∈ F 1 m ∑ i = 1 m σ i f ( z i ) ] = ( ⩽ ) 2 R m ( F ) \begin{aligned}
&{\color{blue} \mathbb{E}_Z[\Phi(Z)]} \\
= & \mathbb{E}_Z\left[\sup_{f\in\mathcal{F}}\left(\mathbb{E}[f]-\widehat{E}_Z(f)\right)\right] \\
= & \mathbb{E}_Z\left[\sup_{f\in\mathcal{F}}\mathbb{E}_{Z'}\left[\widehat{E}_{Z'}(f)-\widehat{E}_Z(f)\right]\right] \\
\leqslant & \mathbb{E}_{Z,Z'}\left[\sup_{f\in\mathcal{F}}\left(\widehat{E}_{Z'}(f)-\widehat{E}_Z(f)\right)\right] \\
\\
= & \mathbb{E}_{Z, Z^{\prime}}\left[\sup _{f \in \mathcal{F}} \frac{1}{m} \sum_{i=1}^m\left(f\left(z_i\right)-f\left(z_i\right)\right)\right] \\
= & \mathbb{E}_{\boldsymbol{\sigma}, Z, Z^{\prime}}\left[\sup _{f \in \mathcal{F}} \frac{1}{m} \sum_{i=1}^m \sigma_i\left(f\left(z_i^{\prime}\right)-f\left(z_i\right)\right)\right] \\
\leqslant &\mathbb{E}_{\boldsymbol{\sigma}, Z^{\prime}}\left[\sup _{f \in \mathcal{F}} \frac{1}{m} \sum_{i=1}^m \sigma_i f\left(z_i\right)\right]+\mathbb{E}_{\boldsymbol{\sigma}, Z}\left[\sup _{f \in \mathcal{F}} \frac{1}{m} \sum_{i=1}^m-\sigma_i f\left(\mathbf{z}_i\right)\right] \\
= & 2 \mathbb{E}_{\boldsymbol{\sigma}, \boldsymbol{Z}}\left[\sup _{f \in \mathcal{F}} \frac{1}{m} \sum_{i=1}^m \sigma_i f\left(\mathbf{z}_i\right)\right] \\
= & {\color{blue} (\leqslant)2 \mathfrak{R}_m(\mathcal{F})}
\end{aligned} = = ⩽ = = ⩽ = = E Z [ Φ ( Z )] E Z [ f ∈ F sup ( E [ f ] − E Z ( f ) ) ] E Z [ f ∈ F sup E Z ′ [ E Z ′ ( f ) − E Z ( f ) ] ] E Z , Z ′ [ f ∈ F sup ( E Z ′ ( f ) − E Z ( f ) ) ] E Z , Z ′ [ f ∈ F sup m 1 i = 1 ∑ m ( f ( z i ) − f ( z i ) ) ] E σ , Z , Z ′ [ f ∈ F sup m 1 i = 1 ∑ m σ i ( f ( z i ′ ) − f ( z i ) ) ] E σ , Z ′ [ f ∈ F sup m 1 i = 1 ∑ m σ i f ( z i ) ] + E σ , Z [ f ∈ F sup m 1 i = 1 ∑ m − σ i f ( z i ) ] 2 E σ , Z [ f ∈ F sup m 1 i = 1 ∑ m σ i f ( z i ) ] ( ⩽ ) 2 R m ( F ) 将几个标蓝的式子代入，即有
E Z [ Φ ( Z ) ] − 1 m ∑ i = 1 m f ( z i ) ⩽ Φ ( Z ) ⩽ E Z [ Φ ( Z ) ] + ln ( 1 / δ ) 2 m ⩽ 2 R m ( F ) + ln ( 1 / δ ) 2 m \begin{aligned}
& \mathbb{E}_Z[\Phi(Z)] - \frac{1}{m}\sum_{i=1}^m f(z_i)\\
\leqslant & \Phi(Z) \\
\leqslant & \mathbb{E}_Z[\Phi(Z)] + \sqrt{\frac{\ln(1/\delta)}{2m}} \\
\leqslant & 2 \mathfrak{R}_m(\mathcal{F}) + \sqrt{\frac{\ln(1/\delta)}{2m}} \\
\end{aligned} ⩽ ⩽ ⩽ E Z [ Φ ( Z )] − m 1 i = 1 ∑ m f ( z i ) Φ ( Z ) E Z [ Φ ( Z )] + 2 m ln ( 1/ δ ) 2 R m ( F ) + 2 m ln ( 1/ δ ) 即定理 4.4 的第一条不等式成立。
根据定义可知替换训练集中的一个样本后经验 Rademacher 复杂度最多改变 1 m \frac{1}{m} m 1 ∣ ℜ ^ Z ( F ) − ℜ ^ Z ′ ( F ) ∣ ⩽ 1 m \left\lvert\widehat{\Re}_Z(\mathcal{F})-\widehat{\Re}_{Z^{\prime}}(\mathcal{F}) \right\rvert \leqslant \frac{1}{m} ∣ ∣ ℜ Z ( F ) − ℜ Z ′ ( F ) ∣ ∣ ⩽ m 1
同理，由 McDiarmid 不等式可得
ℜ ^ m ( F ) ⩽ R ^ Z ( F ) + ln ( 2 / δ ) 2 m (1) \color{blue}\widehat{\Re}_m(\mathcal{F})\leqslant\widehat{\mathfrak{R}}_Z(\mathcal{F})+\sqrt{\frac{\ln(2/\delta)}{2m}}\tag{1} ℜ m ( F ) ⩽ R Z ( F ) + 2 m l n ( 2/ δ ) ( 1 ) 以至少 1 − δ / 2 1-\delta/2 1 − δ /2
Φ ( Z ) ⩽ E Z [ Φ ( Z ) ] + ln ( 2 / δ ) 2 m (2) \color{blue}\Phi(Z)\leqslant\mathbb{E}_Z[\Phi(Z)]+\sqrt{\frac{\ln(2/\delta)}{2m}}\tag{2} Φ ( Z ) ⩽ E Z [ Φ ( Z )] + 2 m l n ( 2/ δ ) ( 2 ) 以至少 1 − δ / 2 1-\delta/2 1 − δ /2
令 X X X Y Y Y
P ( X ⋃ Y ) ⩽ P ( X ) + P ( Y ) P(X\bigcup Y)\leqslant P(X)+P(Y) P ( X ⋃ Y ) ⩽ P ( X ) + P ( Y ) 那么不等式 (1) 或者不等式 (2) 不成立的概率小于 δ \delta δ 1 − δ 1−\delta 1 − δ
Φ ( Z ) ⩽ E Z [ Φ ( Z ) ] + ln ( 2 / δ ) 2 m ⩽ 2 R m ( F ) + ln ( 2 / δ ) 2 m ⩽ 2 ℜ ^ Z ( F ) + 3 ln ( 2 / δ ) 2 m \begin{aligned}
\Phi(Z) & \leqslant \mathbb{E}_Z[\Phi(Z)] + \sqrt{\frac{\ln(2/\delta)}{2m}} \\
& \leqslant 2 \mathfrak{R}_m(\mathcal{F}) + \sqrt{\frac{\ln(2/\delta)}{2m}} \\
& \leqslant 2 \widehat{\Re}_Z(\mathcal{F})+3\sqrt{\frac{\ln(2/\delta)}{2m}} \\
\end{aligned} Φ ( Z ) ⩽ E Z [ Φ ( Z )] + 2 m ln ( 2/ δ ) ⩽ 2 R m ( F ) + 2 m ln ( 2/ δ ) ⩽ 2 ℜ Z ( F ) + 3 2 m ln ( 2/ δ ) 以至少 1 − δ 1 −\delta 1 − δ
定理 4.4 适用于回归问题，对于分类问题有 :
定理 4.5
对于假设空间 H : X ↦ { − 1 , + 1 } \mathcal{H} : \mathcal{X}\mapsto\{−1, +1\} H : X ↦ { − 1 , + 1 } D \mathcal{D} D m m m D = { x 1 , … , x m } , x i ∈ X , h ∈ H D=\{x_1,\ldots,x_m\},x_i\in\mathcal{X},h\in\mathcal{H} D = { x 1 , … , x m } , x i ∈ X , h ∈ H 0 < δ < 1 0 < \delta< 1 0 < δ < 1 1 − δ 1 − \delta 1 − δ
E ( h ) ⩽ E ^ ( h ) + ℜ m ( H ) + ln ( 1 / δ ) 2 m E ( h ) ⩽ E ^ ( h ) + ℜ ^ D ( H ) + 3 ln ( 2 / δ ) 2 m \begin{aligned}
& E(h) \leqslant \widehat{E}(h)+\Re_m(\mathcal{H})+\sqrt{\frac{\ln (1 / \delta)}{2 m}} \\
& E(h) \leqslant \widehat{E}(h)+\widehat{\Re}_D(\mathcal{H})+3 \sqrt{\frac{\ln (2 / \delta)}{2 m}}
\end{aligned} E ( h ) ⩽ E ( h ) + ℜ m ( H ) + 2 m ln ( 1/ δ ) E ( h ) ⩽ E ( h ) + ℜ D ( H ) + 3 2 m ln ( 2/ δ ) 对于二分类问题的假设空间 H \mathcal{H} H Z = X × { − 1 , + 1 } \mathcal{Z}=\mathcal{X}\times\{−1, +1\} Z = X × { − 1 , + 1 } H \mathcal{H} H h h h f h ( z ) = f h ( x , y ) = I ( h ( x ) ≠ y ) f_h(\boldsymbol{z})=f_h(\boldsymbol{x},y)=\mathbb{I}(h(\boldsymbol{x})\ne y) f h ( z ) = f h ( x , y ) = I ( h ( x ) = y ) { − 1 , + 1 } \{-1,+1\} { − 1 , + 1 } H \mathcal{H} H [ 0 , 1 ] [0,1] [ 0 , 1 ] F H = { f h : h ∈ H } \mathcal{F}_{\mathcal{H}}= \{f_h : h\in\mathcal{H}\} F H = { f h : h ∈ H }
ℜ ^ Z ( F H ) = E σ [ sup f h ∈ F H 1 m ∑ i = 1 m σ i f h ( x i , y i ) ] = E σ [ sup h ∈ H 1 m ∑ i = 1 m σ i I ( h ( x i ) ≠ y i ) ] = E σ [ sup h ∈ H 1 m ∑ i = 1 m σ i 1 − y i h ( x i ) 2 ] = 1 2 E σ [ 1 m ∑ i = 1 m σ i + sup h ∈ H 1 m ∑ i = 1 m ( − y i σ i h ( x i ) ) ] = 1 2 E σ [ sup h ∈ H 1 m ∑ i = 1 m ( − y i σ i h ( x i ) ) ] = 1 2 E σ [ sup h ∈ H 1 m ∑ i = 1 m ( σ i h ( x i ) ) ] = 1 2 ℜ m ( H ) = 1 2 R ^ D ( H ) \begin{aligned}
\widehat{\Re}_Z\left(\mathcal{F}_{\mathcal{H}}\right) & =\mathbb{E}_{\boldsymbol{\sigma}}\left[\sup _{f_h \in \mathcal{F}_{\mathcal{H}}} \frac{1}{m} \sum_{i=1}^m \sigma_i f_h\left(\boldsymbol{x}_i, y_i\right)\right] \\
& =\mathbb{E}_{\boldsymbol{\sigma}}\left[\sup _{h \in \mathcal{H}} \frac{1}{m} \sum_{i=1}^m \sigma_i \mathbb{I}\left(h\left(\boldsymbol{x}_i\right) \neq y_i\right)\right] \\
& =\mathbb{E}_{\boldsymbol{\sigma}}\left[\sup _{h \in \mathcal{H}} \frac{1}{m} \sum_{i=1}^m \sigma_i \frac{1-y_i h\left(\boldsymbol{x}_i\right)}{2}\right] \\
& =\frac{1}{2} \mathbb{E}_{\boldsymbol{\sigma}}\left[\frac{1}{m} \sum_{i=1}^m \sigma_i+\sup _{h \in \mathcal{H}} \frac{1}{m} \sum_{i=1}^m\left(-y_i \sigma_i h\left(\boldsymbol{x}_i\right)\right)\right] \\
& =\frac{1}{2} \mathbb{E}_{\boldsymbol{\sigma}}\left[\sup _{h \in \mathcal{H}} \frac{1}{m} \sum_{i=1}^m\left(-y_i \sigma_i h\left(\boldsymbol{x}_i\right)\right)\right]\\
& =\frac{1}{2} \mathbb{E}_\sigma\left[\sup _{h \in \mathcal{H}} \frac{1}{m} \sum_{i=1}^m\left(\sigma_i h\left(\boldsymbol{x}_i\right)\right)\right] \\
& =\frac{1}{2} \mathfrak{\Re}_m(\mathcal{H}) \\
& =\frac{1}{2} \widehat{\mathfrak{R}}_D(\mathcal{H})
\end{aligned} ℜ Z ( F H ) = E σ [ f h ∈ F H sup m 1 i = 1 ∑ m σ i f h ( x i , y i ) ] = E σ [ h ∈ H sup m 1 i = 1 ∑ m σ i I ( h ( x i ) = y i ) ] = E σ [ h ∈ H sup m 1 i = 1 ∑ m σ i 2 1 − y i h ( x i ) ] = 2 1 E σ [ m 1 i = 1 ∑ m σ i + h ∈ H sup m 1 i = 1 ∑ m ( − y i σ i h ( x i ) ) ] = 2 1 E σ [ h ∈ H sup m 1 i = 1 ∑ m ( − y i σ i h ( x i ) ) ] = 2 1 E σ [ h ∈ H sup m 1 i = 1 ∑ m ( σ i h ( x i ) ) ] = 2 1 ℜ m ( H ) = 2 1 R D ( H ) 同时对上式两边取期望可得
ℜ Z ( F H ) = 1 2 R D ( H ) \Re_Z(\mathcal{F}_{\mathcal{H}})=\frac{1}{2}\mathfrak{R}_D(\mathcal{H}) ℜ Z ( F H ) = 2 1 R D ( H ) 将此式代入定理 4.4，定理 4.5 得证。
