《机器学习理论导引》笔记目录
6.3 划分机制
一些机器学习方法可看作是将样本空间𝒳划分成多个互不相容的区域，然后在各区域中对正例和反例分别计数，以多数的类别作为区域中样本的标记，这被称为划分机制。常见的基于划分机制的机器学习方法包括最近邻、决策树、随机森林等。
具体而言，给定训练集 D m = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , … , ( x m , y m ) } \mathcal{D}_m=\{(x_1,y_1),(x_2,y_2),\ldots,(x_m,y_m)\} D m = {( x 1 , y 1 ) , ( x 2 , y 2 ) , … , ( x m , y m )} X \mathcal{X} X Ω 1 , Ω 2 , … , Ω n \Omega_1,\Omega_2,\ldots,\Omega_n Ω 1 , Ω 2 , … , Ω n
h m ( x ) = { + 1 如果 ∑ x i ∈ Ω ( x ) I ( y i = + 1 ) ≥ ∑ x i ∈ Ω ( x ) I ( y i = − 1 ) − 1 如果 ∑ x i ∈ Ω ( x ) I ( y i = + 1 ) < ∑ x i ∈ Ω ( x ) I ( y i = − 1 ) h_m(x)= \begin{cases}+1 & \text { 如果 } \sum_{x_i \in \Omega(x)} \mathbb{I}\left(y_i=+1\right) \geq \sum_{x_i \in \Omega(x)} \mathbb{I}\left(y_i=-1\right) \\ -1 & \text { 如果 } \sum_{x_i \in \Omega(x)} \mathbb{I}\left(y_i=+1\right)<\sum_{x_i \in \Omega(x)} \mathbb{I}\left(y_i=-1\right)\end{cases} h m ( x ) = { + 1 − 1 如果 ∑ x i ∈ Ω ( x ) I ( y i = + 1 ) ≥ ∑ x i ∈ Ω ( x ) I ( y i = − 1 ) 如果 ∑ x i ∈ Ω ( x ) I ( y i = + 1 ) < ∑ x i ∈ Ω ( x ) I ( y i = − 1 ) 其中 Ω ( x ) \Omega(x) Ω ( x ) x x x
定义 6.3 划分机制一致性
随着训练数据规模 m → ∞ m\rightarrow\infty m → ∞ h m ( x ) h_m(x) h m ( x ) R ( h m ) → R ∗ R(h_m)\rightarrow R^* R ( h m ) → R ∗
直观上看，划分机制具有一致性需要满足两个条件:
划分后的区域足够小，从而保证能够捕捉数据的局部信息
划分后的区域应包含足够多的训练样本，从而确保“少数服从多数”方法的有效性
给定区域 Ω \Omega Ω Diam ( Ω ) \text{Diam}(\Omega) Diam ( Ω ) Ω \Omega Ω Diam ( Ω ) = sup x , x ′ ∈ Ω ∥ x − x ′ ∥ \text{Diam}(\Omega)=\sup_{x,x'\in\Omega}\lVert x-x'\rVert Diam ( Ω ) = sup x , x ′ ∈ Ω ∥ x − x ′ ∥ x ∈ X x\in\mathcal{X} x ∈ X N ( x ) = ∑ i = 1 m I ( x i ∈ Ω ( x ) ) N(x)=\sum_{i=1}^m\mathbb{I}(x_i\in\Omega(x)) N ( x ) = ∑ i = 1 m I ( x i ∈ Ω ( x )) Ω ( x ) \Omega(x) Ω ( x )
引理 6.4
设 X 1 , X 2 , … , X m X_1,X_2,\ldots,X_m X 1 , X 2 , … , X m m m m X i ∼ Bernoulli ( p ) X_i\sim\text{Bernoulli}(p) X i ∼ Bernoulli ( p )
E [ ∣ 1 m ∑ i = 1 m X i − E [ X i ] ∣ ] ⩽ p ( 1 − p ) m \mathbb{E}\left[\left\lvert\frac{1}{m}\sum_{i=1}^mX_i-\mathbb{E}[X_i]\right\rvert\right]\leqslant\sqrt{\frac{p(1-p)}{m}} E [ ∣ ∣ m 1 i = 1 ∑ m X i − E [ X i ] ∣ ∣ ] ⩽ m p ( 1 − p ) 证明
根据 Jensen 不等式有
E [ 1 m ∑ i = 1 m X i − E [ X i ] ] ⩽ E [ 1 m ∑ i = 1 m X i − E [ X i ] ] 2 = 1 m 2 ∑ i = 1 m E [ X i − E [ X i ] ] 2 = v ( X 1 ) m = p ( 1 − p ) m \mathbb{E}\left[\frac{1}{m} \sum_{i=1}^m X_i-\mathbb{E}\left[X_i\right]\right]\leqslant\sqrt{\mathbb{E}\left[\frac{1}{m} \sum_{i=1}^m X_i-\mathbb{E}\left[X_i\right]\right]^2}\\
=\sqrt{\frac{1}{m^2} \sum_{i=1}^m \mathbb{E}\left[X_i-\mathbb{E}\left[X_i\right]\right]^2}=\sqrt{\frac{v\left(X_1\right)}{m}}=\sqrt{\frac{p(1-p)}{m}} E [ m 1 i = 1 ∑ m X i − E [ X i ] ] ⩽ E [ m 1 i = 1 ∑ m X i − E [ X i ] ] 2 = m 2 1 i = 1 ∑ m E [ X i − E [ X i ] ] 2 = m v ( X 1 ) = m p ( 1 − p ) 引理得证。
定理 6.4
假设条件概率 η ( x ) \eta(x) η ( x ) X \mathcal{X} X m → ∞ m\rightarrow\infty m → ∞ Diam ( Ω ( x ) ) → 0 \text{Diam}(\Omega(x))\rightarrow0 Diam ( Ω ( x )) → 0 N ( x ) → ∞ N(x)\rightarrow\infty N ( x ) → ∞
其中依概率成立 (比极限更弱) 是指 : 对 ∀ ϵ > 0 , lim m → ∞ P ( ( Diam ( Ω ) − 0 ) ⩾ ) = 0 \forall\epsilon>0,\lim_{m\rightarrow\infty}P((\text{Diam}(\Omega)-0)\geqslant)=0 ∀ ϵ > 0 , lim m → ∞ P (( Diam ( Ω ) − 0 ) ⩾ ) = 0 ∀ N > 0 , lim m → ∞ P ( N ( x ) > N ) = 1 \forall N>0,\lim_{m\rightarrow\infty}P(N(x)>N)=1 ∀ N > 0 , lim m → ∞ P ( N ( x ) > N ) = 1
证明
对任意样本 x ∈ X x\in\mathcal{X} x ∈ X η ^ ( x ) = ∑ x i ∈ Ω ( x ) I ( y i = + 1 ) N ( x ) \hat{\eta}(x)=\sum_{x_i\in\Omega(x)}\frac{\mathbb
{I}(y_i=+1)}{N(x)} η ^ ( x ) = ∑ x i ∈ Ω ( x ) N ( x ) I ( y i = + 1 ) h m ( x ) = 2 I ( η ^ ( x ) ⩾ 1 2 ) − 1 h_m(x)=2\mathbb{I}\left(\hat{\eta}(x)\geqslant\frac{1}{2}\right)-1 h m ( x ) = 2 I ( η ^ ( x ) ⩾ 2 1 ) − 1 R ( h m ) − R ∗ ⩽ 2 E [ ∣ η ^ ( x ) − η ( x ) ∣ ] R(h_m)-R^*\leqslant2\mathbb{E}[\lvert\hat{\eta}(x)-\eta(x)\rvert] R ( h m ) − R ∗ ⩽ 2 E [∣ η ^ ( x ) − η ( x )∣] Ω ( x ) \Omega(x) Ω ( x ) η ˉ ( x ) = E [ η ( x ′ ) ∣ x ′ ∈ Ω ( x ) ] \bar{\eta}(x)=\mathbb{E}[\eta(x')|x'\in\Omega(x)] η ˉ ( x ) = E [ η ( x ′ ) ∣ x ′ ∈ Ω ( x )]
利用三角不等式有 E [ ∣ η ^ ( x ) − η ( x ) ∣ ] ⩽ E [ ∣ η ^ ( x ) − η ˉ ( x ) ∣ ] + E [ ∣ η ˉ ( x ) − η ( x ) ∣ ] \mathbb{E}[\lvert\hat{\eta}(x)-\eta(x)\rvert]\leqslant\mathbb{E}[\lvert\hat{\eta}(x)-\bar{\eta}(x)\rvert]+\mathbb{E}[\lvert\bar{\eta}(x)-\eta(x)\rvert] E [∣ η ^ ( x ) − η ( x )∣] ⩽ E [∣ η ^ ( x ) − η ˉ ( x )∣] + E [∣ η ˉ ( x ) − η ( x )∣] η ( x ) \eta(x) η ( x ) m → ∞ m\rightarrow\infty m → ∞ Diam ( Ω ( x ) ) → 0 \text{Diam}(\Omega(x))\rightarrow0 Diam ( Ω ( x )) → 0 E [ ∣ η ˉ ( x ) − η ( x ) ∣ ] → 0 \mathbb{E}[\lvert\bar{\eta}(x)-\eta(x)\rvert]\rightarrow0 E [∣ η ˉ ( x ) − η ( x )∣] → 0
下面只需证明 E [ ∣ η ^ ( x ) ≤ − η ˉ ( x ) ∣ ] → 0 \mathbb{E}[\lvert\hat{\eta}(x)\leq-\bar{\eta}(x)\rvert]\rightarrow0 E [∣ η ^ ( x ) ≤ − η ˉ ( x )∣] → 0
给定 x , x 1 , … , x m x,x_1,\ldots,x_m x , x 1 , … , x m
E [ ∣ η ^ ( x ) − η ˉ ( x ) ∥ x , x 1 , … , x m ] = E [ ∣ η ^ ( x ) − η ˉ ( x ) ∥ N ( x ) = 0 , x , x 1 , … , x m ] P ( N ( x ) = 0 ) + E [ ∣ η ^ ( x ) − η ˉ ( x ) ∥ N ( x ) > 0 , x , x 1 , … , x m ] P ( N ( x ) > 0 ) ⩽ P ( N ( x ) = 0 ∣ x , x 1 , … , x m ) + E [ ∣ ∑ x i ∈ Ω ( x ) I ( y i = + 1 ) − η ˉ ( x ) N ( x ) ∥ N ( x ) > 0 , x , x 1 , … , x m ] \begin{aligned}
& \mathbb{E}\left[\mid \hat{\eta}(x)-\bar{\eta}(x) \| x, x_1, \ldots, x_m\right] \\
=&\mathbb{E}\left[\mid \hat{\eta}(x)-\bar{\eta}(x) \| N(x)=0, x, x_1, \ldots, x_m\right] P(N(x)=0)+ \\
& \mathbb{E}\left[\mid \hat{\eta}(x)-\bar{\eta}(x) \| N(x)>0, x, x_1, \ldots, x_m\right] P(N(x)>0) \\
\leqslant & P\left(N(x)=0 \mid x, x_1, \ldots, x_m\right)\\
+&\mathbb{E}\left[\mid \sum_{x_i \in \Omega(x)} \frac{\mathbb{I}\left(y_i=+1\right)-\bar{\eta}(x)}{N(x)} \| N(x)>0, x, x_1, \ldots, x_m\right]\\
\end{aligned} = ⩽ + E [ ∣ η ^ ( x ) − η ˉ ( x ) ∥ x , x 1 , … , x m ] E [ ∣ η ^ ( x ) − η ˉ ( x ) ∥ N ( x ) = 0 , x , x 1 , … , x m ] P ( N ( x ) = 0 ) + E [ ∣ η ^ ( x ) − η ˉ ( x ) ∥ N ( x ) > 0 , x , x 1 , … , x m ] P ( N ( x ) > 0 ) P ( N ( x ) = 0 ∣ x , x 1 , … , x m ) E ⎣ ⎡ ∣ x i ∈ Ω ( x ) ∑ N ( x ) I ( y i = + 1 ) − η ˉ ( x ) ∥ N ( x ) > 0 , x , x 1 , … , x m ⎦ ⎤ N ( x ) η ^ ( x ) N(x)\hat{\eta}(x) N ( x ) η ^ ( x ) Ω ( x ) \Omega(x) Ω ( x ) B ( N ( x ) , η ^ ( x ) ) \mathcal{B}(N(x),\hat{\eta}(x)) B ( N ( x ) , η ^ ( x ))
E [ ∣ ∑ x i ∈ Ω ( x ) I ( y i = + 1 ) − η ˉ ( x ) N ( x ) ∥ N ( x ) > 0 , x , x 1 , … , x m ] ⩽ E [ η ˉ ( x ) ( 1 − η ˉ ( x ) ) N ( x ) I ( N ( x ) > 0 ) ∣ x , x 1 , … , x m ] ⩽ 1 2 P ( 1 ⩽ N ( x ) ⩽ k ∣ x , x 1 , … , x m ) + 1 2 k P ( N ( x ) > k ∣ x , x 1 , … , x m ) ⩽ 1 2 P ( 1 ⩽ N ( x ) ⩽ k ∣ x , x 1 , … , x m ) + 1 2 k \begin{aligned}
& \mathbb{E}\left[\mid \sum_{x_i \in \Omega(x)} \frac{\mathbb{I}\left(y_i=+1\right)-\bar{\eta}(x)}{N(x)} \| N(x)>0, x, x_1, \ldots, x_m\right] \\
& \leqslant \mathbb{\mathbb{E}}\left[\sqrt{\frac{\bar{\eta}(x)(1-\bar{\eta}(x))}{N(x)}} \mathbb{I}(N(x)>0) \mid x, x_1, \ldots, x_m\right] \\
& \leqslant \frac{1}{2} P\left(1 \leqslant N(x) \leqslant k \mid x, x_1, \ldots, x_m\right)+\frac{1}{2 \sqrt{k}} P\left(N(x)>k \mid x, x_1, \ldots, x_m\right) \\
& \leqslant \frac{1}{2} P\left(1 \leqslant N(x) \leqslant k \mid x, x_1, \ldots, x_m\right)+\frac{1}{2 \sqrt{k}}
\end{aligned} E ⎣ ⎡ ∣ x i ∈ Ω ( x ) ∑ N ( x ) I ( y i = + 1 ) − η ˉ ( x ) ∥ N ( x ) > 0 , x , x 1 , … , x m ⎦ ⎤ ⩽ E [ N ( x ) η ˉ ( x ) ( 1 − η ˉ ( x )) I ( N ( x ) > 0 ) ∣ x , x 1 , … , x m ] ⩽ 2 1 P ( 1 ⩽ N ( x ) ⩽ k ∣ x , x 1 , … , x m ) + 2 k 1 P ( N ( x ) > k ∣ x , x 1 , … , x m ) ⩽ 2 1 P ( 1 ⩽ N ( x ) ⩽ k ∣ x , x 1 , … , x m ) + 2 k 1 结合以上两个不等式，有
E [ ∣ η ^ ( x ) − η ˉ ( x ) ∣ ∣ x , x 1 , … , x m ] ⩽ P ( N ( x ) = 0 ∣ x , x 1 , … , x m ) + 1 2 P ( 1 ⩽ N ( x ) ⩽ k ∣ x , x 1 , … , x m ) + 1 2 k \begin{aligned}
& \mathbb{E}\left[|\hat{\eta}(x)-\bar{\eta}(x)| \mid x, x_1, \ldots, x_m\right] \\
& \leqslant P\left(N(x)=0 \mid x, x_1, \ldots, x_m\right)+\frac{1}{2} P\left(1 \leqslant N(x) \leqslant k \mid x, x_1, \ldots, x_m\right)+\frac{1}{2 \sqrt{k}}
\end{aligned} E [ ∣ η ^ ( x ) − η ˉ ( x ) ∣ ∣ x , x 1 , … , x m ] ⩽ P ( N ( x ) = 0 ∣ x , x 1 , … , x m ) + 2 1 P ( 1 ⩽ N ( x ) ⩽ k ∣ x , x 1 , … , x m ) + 2 k 1 对 x , x 1 , … , x m x,x_1,\ldots,x_m x , x 1 , … , x m
E [ ∣ η ^ ( x ) − η ˉ ( x ) ∣ ] ⩽ P ( N ( x ) = 0 ) + 1 2 P ( 1 ⩽ N ( x ) ⩽ s k ) + 1 2 k \mathbb{E}[|\hat{\eta}(x)-\bar{\eta}(x)|] \leqslant P(N(x)=0)+\frac{1}{2} P(1 \leqslant N(x) \leqslant sk)+\frac{1}{2 \sqrt{k}} E [ ∣ η ^ ( x ) − η ˉ ( x ) ∣ ] ⩽ P ( N ( x ) = 0 ) + 2 1 P ( 1 ⩽ N ( x ) ⩽ s k ) + 2 k 1 取 k = N ( x ) k=\sqrt{N(x)} k = N ( x ) N ( x ) → ∞ N(x)\rightarrow\infty N ( x ) → ∞ E [ ∣ η ^ ( x ) ≤ − η ˉ ( x ) ∣ ] → 0 \mathbb{E}[\lvert\hat{\eta}(x)\leq-\bar{\eta}(x)\rvert]\rightarrow0 E [∣ η ^ ( x ) ≤ − η ˉ ( x )∣] → 0
随机森林 (Random Forest)
随机森林 (Random Forest) 是一种重要的集成学习方法 (ensemble learning)，通过对数据集进行有放回采样 (bootstrap sampling) 产生多个训练集，然后基于每个训练集产生随机决策树，最后通过投票法对随机决策树进行集成，这些随机决策树是在决策树生成过程中，对划分节点、划分属性及划分点引入随机选择而产生的。
对随机决策树，可以引入一个新的随机变量 Z ∈ Z Z\in\mathcal{Z} Z ∈ Z h m ( x , Z ) h_m(x,Z) h m ( x , Z ) m m m n n n h m ( x , Z 1 ) , h m ( x , Z 2 ) , … , h m ( x , Z n ) h_m(x,Z_1),h_m(x,Z_2),\ldots,h_m(x,Z_n) h m ( x , Z 1 ) , h m ( x , Z 2 ) , … , h m ( x , Z n ) h ˉ m ( x ; Z 1 , … , Z n ) \bar{h}_m(x;Z_1,\ldots,Z_n) h ˉ m ( x ; Z 1 , … , Z n )
h ˉ m ( x ; Z 1 , … , Z n ) = { + 1 如果 ∑ i = 1 n h m ( x , Z i ) ≥ 0 − 1 如果 ∑ i = 1 n h m ( x , Z i ) < 0 \bar{h}_m\left(x ; Z_1, \ldots, Z_n\right)= \begin{cases}+1 & \text { 如果 } \sum_{i=1}^n h_m\left(x, Z_i\right) \geq 0 \\ -1 & \text { 如果 } \sum_{i=1}^n h_m\left(x, Z_i\right)<0\end{cases} h ˉ m ( x ; Z 1 , … , Z n ) = { + 1 − 1 如果 ∑ i = 1 n h m ( x , Z i ) ≥ 0 如果 ∑ i = 1 n h m ( x , Z i ) < 0 引理 6.5
对随机决策树 h m ( x , Z ) h_m(x,Z) h m ( x , Z ) h ˉ m ( x ; Z 1 , … , Z n ) \bar{h}_m(x;Z_1,\ldots,Z_n) h ˉ m ( x ; Z 1 , … , Z n )
E Z 1 , … , Z n [ R ( h ˉ m ( x ; Z 1 , … , Z n ) ) ] − R ∗ ≤ 2 ( E Z [ R ( h m ( x , Z ) ) ] − R ∗ ) E_{Z_1, \ldots, Z_n}\left[R\left(\bar{h}_m\left(x ; Z_1, \ldots, Z_n\right)\right)\right]-\mathrm{R}^* \leq 2\left(E_Z\left[R\left(h_m(x, Z)\right)\right]-R^*\right) E Z 1 , … , Z n [ R ( h ˉ m ( x ; Z 1 , … , Z n ) ) ] − R ∗ ≤ 2 ( E Z [ R ( h m ( x , Z ) ) ] − R ∗ ) 此引理表明，若随机决策树 h m ( x , Z ) h_m(x,Z) h m ( x , Z ) h ˉ m ( x ; Z 1 , … , Z n ) \bar{h}_m(x;Z_1,\ldots,Z_n) h ˉ m ( x ; Z 1 , … , Z n )
证明
根据引理 6.1 可知
E Z [ R ( h m ( x , Z ) ) ] − R ∗ = E x ∼ D x [ ∣ 1 − 2 η ( x ) ∣ I ( h m ( x , Z ) ≠ h ∗ ( x ) ) ] = E x ∼ D x [ ( 1 − 2 η ( x ) ) I ( η ( x ) < 1 2 ) P Z ( h m ( x , Z ) = 1 ) + ( 2 η ( x ) − 1 ) I ( η ( x ) > 1 2 ) P Z ( h m ( x , Z ) = − 1 ) ] \begin{aligned}
\mathbb{E}_Z\left[R\left(h_m(x, Z)\right)\right]-R^* &= \mathbb{E}_{x \sim \mathcal{D}_x}\left[|1-2 \eta(x)| \mathbb{I}\left(h_m(x, Z) \neq h^*(x)\right)\right] \\
&=\mathbb{E}_{x \sim \mathcal{D}_x}\left[(1-2 \eta(x)) \mathbb{I}\left(\eta(x)<\frac{1}{2}\right) P_Z\left(h_m(x, Z)=1\right)\right. \\
&\left.+(2 \eta(x)-1) \mathbb{I}\left(\eta(x)>\frac{1}{2}\right) P_Z\left(h_m(x, Z)=-1\right)\right]
\end{aligned} E Z [ R ( h m ( x , Z ) ) ] − R ∗ = E x ∼ D x [ ∣1 − 2 η ( x ) ∣ I ( h m ( x , Z ) = h ∗ ( x ) ) ] = E x ∼ D x [ ( 1 − 2 η ( x )) I ( η ( x ) < 2 1 ) P Z ( h m ( x , Z ) = 1 ) + ( 2 η ( x ) − 1 ) I ( η ( x ) > 2 1 ) P Z ( h m ( x , Z ) = − 1 ) ] 进一步得到
E Z 1 , … , Z n [ R ( h ˉ m ( x ; Z 1 , … , Z n ) ) ] − R ∗ = E x ∼ D x [ ( 1 − 2 η ( x ) ) I ( η ( x ) < 1 2 ) P Z 1 , … Z n ( h ˉ m ( x ; Z 1 , … , Z n ) = 1 ) + ( 2 η ( x ) − 1 ) I ( η ( x ) > 1 2 ) P Z 1 , … , Z n ( h ˉ m ( x ; Z 1 , … , Z n ) = − 1 ) ] \begin{aligned}
&\mathbb{E}_{Z_1, \ldots, Z_n}\left[R\left(\bar{h}_m\left(x ; Z_1, \ldots, Z_n\right)\right)\right]-R^*\\
=\mathbb{E}_{x \sim D_x} & {\left[(1-2 \eta(x)) \mathbb{I}\left(\eta(x)<\frac{1}{2}\right) P_{Z_1, \ldots Z_n}\left(\bar{h}_m\left(x ; Z_1, \ldots, Z_n\right)=1\right)\right.} \\
& \left.+(2 \eta(x)-1) \mathbb{I}\left(\eta(x)>\frac{1}{2}\right) P_{Z_1, \ldots, Z_n}\left(\bar{h}_m\left(x ; Z_1, \ldots, Z_n\right)=-1\right)\right]
\end{aligned} = E x ∼ D x E Z 1 , … , Z n [ R ( h ˉ m ( x ; Z 1 , … , Z n ) ) ] − R ∗ [ ( 1 − 2 η ( x )) I ( η ( x ) < 2 1 ) P Z 1 , … Z n ( h ˉ m ( x ; Z 1 , … , Z n ) = 1 ) + ( 2 η ( x ) − 1 ) I ( η ( x ) > 2 1 ) P Z 1 , … , Z n ( h ˉ m ( x ; Z 1 , … , Z n ) = − 1 ) ] 对任意样本 x ∈ X x\in\mathcal{X} x ∈ X η ( x ) < 1 / 2 \eta(x)<1/2 η ( x ) < 1/2 P Z 1 , … , Z n ( h ˉ m ( x ; Z 1 , … , Z n ) = 1 ) ⩽ 2 P Z ( h m ( x , Z ) = 1 ) P_{Z_1, \ldots, Z_n}\left(\bar{h}_m\left(x ; Z_1, \ldots, Z_n\right)=1\right) \leqslant 2 P_Z\left(h_m(x, Z)=1\right) P Z 1 , … , Z n ( h ˉ m ( x ; Z 1 , … , Z n ) = 1 ) ⩽ 2 P Z ( h m ( x , Z ) = 1 )
P Z 1 , … , Z n ( h ˉ m ( x ; Z 1 , … , Z n ) = 1 ) = P Z 1 , … , Z n ( ∑ i = 1 n I ( h m ( x , Z i ) = 1 ) ⩾ n 2 ) ( 由 Markov 不等式 ) ⩽ 2 n ∑ i = 1 n E [ I ( h m ( x , Z i ) = 1 ) ] = 2 P ( h m ( x , Z i ) = 1 ) \begin{aligned}
&P_{Z_1, \ldots, Z_n}\left(\bar{h}_m\left(x ; Z_1, \ldots, Z_n\right)=1\right)=P_{Z_1, \ldots, Z_n}\left(\sum_{i=1}^n\mathbb{I}(h_m(x,Z_i)=1)\geqslant\frac{n}{2}\right)\\
&(\text{由 Markov 不等式})\leqslant\frac{2}{n}\sum_{i=1}^n\mathbb{E}[\mathbb{I}(h_m(x,Z_i)=1)]=2P(h_m(x,Z_i)=1)\\
\end{aligned} P Z 1 , … , Z n ( h ˉ m ( x ; Z 1 , … , Z n ) = 1 ) = P Z 1 , … , Z n ( i = 1 ∑ n I ( h m ( x , Z i ) = 1 ) ⩾ 2 n ) ( 由 Markov 不等式 ) ⩽ n 2 i = 1 ∑ n E [ I ( h m ( x , Z i ) = 1 )] = 2 P ( h m ( x , Z i ) = 1 ) 同理可证 η ( x ) ⩾ 1 / 2 \eta(x)\geqslant1/2 η ( x ) ⩾ 1/2
定理 6.5
当训练集规模 m → ∞ m\rightarrow\infty m → ∞ k = k ( m ) → ∞ k=k(m)\rightarrow\infty k = k ( m ) → ∞ k m → 0 \frac{k}{m}\rightarrow0 m k → 0
证明
首先研究随机决策树的一致性，随机决策树本质上是基于划分机制的一种分类方法。考虑样本空间 X = [ 0 , 1 ] d \mathcal{X}=[0,1]^d X = [ 0 , 1 ] d x ∈ X x\in\mathcal{X} x ∈ X Ω ( x , Z ) \Omega(x,Z) Ω ( x , Z ) x x x N ( x , Z ) N(x,Z) N ( x , Z ) Ω ( x , Z ) \Omega(x,Z) Ω ( x , Z ) N ( x , Z ) = ∑ i = 1 m I ( x i ∈ Ω ( x , Z ) ) N(x,Z)=\sum_{i=1}^m\mathbb{I}(x_i\in\Omega(x,Z)) N ( x , Z ) = ∑ i = 1 m I ( x i ∈ Ω ( x , Z ))
首先证明当 m → ∞ m\rightarrow\infty m → ∞ N ( x , Z ) → ∞ N(x,Z)\rightarrow\infty N ( x , Z ) → ∞ Ω 1 , Ω 2 , … , Ω k + 1 \Omega_1,\Omega_2,\ldots,\Omega_{k+1} Ω 1 , Ω 2 , … , Ω k + 1 k k k k + 1 k+1 k + 1 N 1 , N 2 , … , N k + 1 N_1,N_2, \ldots,N_{k+1} N 1 , N 2 , … , N k + 1 D m \mathcal{D}_m D m D m \mathcal{D}_m D m Z Z Z x x x Ω i \Omega_i Ω i N i / m N_i/m N i / m
对任意给定 t > 0 t>0 t > 0
P ( N ( x , Z ) < t ) = E [ P ( N ( x , Z ) < t ∣ D m , Z ) ] = E [ ∑ i : N i < t N i m ] ⩽ ( t − 1 ) k + 1 m → 0 P(N(x,Z)<t)=\mathbb{E}[P(N(x,Z)<t|D_m,Z)]=\mathbb{E}\left[\sum_{i:N_i<t}\frac{N_i}{m}\right]\leqslant(t−1)\frac{k+1}{m}
\rightarrow 0 P ( N ( x , Z ) < t ) = E [ P ( N ( x , Z ) < t ∣ D m , Z )] = E [ i : N i < t ∑ m N i ] ⩽ ( t − 1 ) m k + 1 → 0 其次证明当 k → ∞ k\rightarrow\infty k → ∞ Ω ( x , Z ) \Omega(x, Z) Ω ( x , Z ) Diam ( Ω ( x , Z ) ) → 0 \text{Diam}(\Omega(x,Z))\rightarrow 0 Diam ( Ω ( x , Z )) → 0 T m T_m T m Ω ( x , Z ) \Omega(x, Z) Ω ( x , Z ) T m = ∑ i = 1 k ξ i T_m=\sum_{i=1}^k\xi_i T m = ∑ i = 1 k ξ i ξ i ∼ Bernoulli ( 1 / i ) \xi_i\sim\text{Bernoulli}(1/i) ξ i ∼ Bernoulli ( 1/ i )
E [ T m ] = ∑ i = 1 k 1 i ≥ ln k , V ( T m ) = ∑ i = 1 k 1 i ( 1 − 1 i ) ≤ ln k + 1 \mathbb{E}\left[T_m\right]=\sum_{i=1}^k \frac{1}{i} \geq \ln k, V\left(T_m\right)=\sum_{i=1}^k \frac{1}{i}\left(1-\frac{1}{i}\right) \leq \ln k+1 E [ T m ] = i = 1 ∑ k i 1 ≥ ln k , V ( T m ) = i = 1 ∑ k i 1 ( 1 − i 1 ) ≤ ln k + 1 根据 Chebyshev 不等式可知，当 k → ∞ k\rightarrow\infty k → ∞
P ( ∣ T m − E [ T m ] ∣ ≥ E [ T m ] 2 ) ≤ 4 V ( T m ) E [ T m ] 2 ≤ 4 ln k + 1 ln 2 k → 0 P\left(\left|T_m-\mathbb{E}\left[T_m\right]\right| \geq \frac{\mathbb{E}\left[T_m\right]}{2}\right) \leq 4 \frac{V\left(T_m\right)}{\mathbb{E}\left[T_m\right]^2} \leq 4 \frac{\ln k+1}{\ln ^2 k} \rightarrow 0 P ( ∣ T m − E [ T m ] ∣ ≥ 2 E [ T m ] ) ≤ 4 E [ T m ] 2 V ( T m ) ≤ 4 ln 2 k ln k + 1 → 0 故 P ( T m ⩽ E [ T m ] 2 ) → 0 P\left(T_m\leqslant\frac{\mathbb{E}[T_m]}{2}\right)\rightarrow 0 P ( T m ⩽ 2 E [ T m ] ) → 0 E [ T m ] ⩾ ln k \mathbb{E}[T_m]\geqslant\ln k E [ T m ] ⩾ ln k P ( T m ⩾ ln k 2 ) → 1 P\left(T_m \geqslant\frac{\ln k}{2}\right)\rightarrow 1 P ( T m ⩾ 2 l n k ) → 1
令 L j L_j L j Ω ( x , Z ) \Omega(x, Z) Ω ( x , Z ) j j j
E [ L j ] ≤ E [ E [ ∏ i = 1 K j max ( U i , 1 − U i ) ∣ K j ] ] \mathbb{E}\left[L_j\right] \leq \mathbb{E}\left[\mathbb{E}\left[\prod_{i=1}^{K_j} \max \left(U_i, 1-U_i\right) \mid K_j\right]\right] E [ L j ] ≤ E ⎣ ⎡ E ⎣ ⎡ i = 1 ∏ K j max ( U i , 1 − U i ) ∣ K j ⎦ ⎤ ⎦ ⎤ 这里的随机变量 k j ∼ B ( T m , 1 / d ) k_j\sim\mathcal{B}(T_m,1/d) k j ∼ B ( T m , 1/ d ) j j j U i ∼ U ( 0 , 1 ) U_i\sim U(0,1) U i ∼ U ( 0 , 1 ) j j j U i ∼ U ( 0 , 1 ) U_i\sim U(0,1) U i ∼ U ( 0 , 1 )
E [ max ( U i , 1 − U i ) ] = 2 ∫ 1 / 2 1 U i d U i = 3 4 \mathbb{E}\left[\max \left(U_i, 1-U_i\right)\right]=2 \int_{1 / 2}^1 U_i d U_i=\frac{3}{4} E [ max ( U i , 1 − U i ) ] = 2 ∫ 1/2 1 U i d U i = 4 3 由此可得
E ( L j ) = E [ E [ Π i = 1 K j max ( U i , 1 − U i ) ∣ K j ] ] = E [ ( 3 4 ) K j ] \mathbb{E}\left(L_j\right)=\mathbb{E}\left[\mathbb{E}\left[\Pi_{i=1}^{K_j} \max \left(U_i, 1-U_i\right) \mid K_j\right]\right]=\mathbb{E}\left[\left(\frac{3}{4}\right)^{K_j}\right] E ( L j ) = E [ E [ Π i = 1 K j max ( U i , 1 − U i ) ∣ K j ] ] = E [ ( 4 3 ) K j ] 再根据 k j ∼ B ( T m , 1 / d ) k_j\sim\mathcal{B}(T_m,1/d) k j ∼ B ( T m , 1/ d )
E [ L j ] = E [ ( 3 4 ) K j ] = E [ ∑ k j = 0 = 0 T m ( 3 4 ) K j ( T k m K j ) ( 1 a ) K j ( 1 − 1 d ) T m − K j ] = E [ ∑ k j = = 0 T T m ( T m ) ( 3 4 d ) K j ( 1 − 1 d ) T m − K j ] = E [ ( 1 − 1 d + 3 4 d ) T m ] = E [ ( 1 − 1 4 d ) T m ] \begin{aligned}
E\left[L_j\right]=E\left[\left(\frac{3}{4}\right)^{K_j}\right] & =E\left[\sum_{k_{j=0}=0}^{T_m}\left(\frac{3}{4}\right)^{K_j}\left(T_{k_m}^{K_j}\right)\left(\frac{1}{a}\right)^{K_j}\left(1-\frac{1}{d}\right)^{T_m-K_j}\right]\\&=E\left[\sum_{k_{j=}=0}^{T_{T_m}}\left(T_m\right)\left(\frac{3}{4 d}\right)^{K_j}\left(1-\frac{1}{d}\right)^{T_m-K_j}\right] \\
& =E\left[\left(1-\frac{1}{d}+\frac{3}{4 d}\right)^{T_m}\right]=E\left[\left(1-\frac{1}{4 d}\right)^{T_m}\right]
\end{aligned} E [ L j ] = E [ ( 4 3 ) K j ] = E ⎣ ⎡ k j = 0 = 0 ∑ T m ( 4 3 ) K j ( T k m K j ) ( a 1 ) K j ( 1 − d 1 ) T m − K j ⎦ ⎤ = E ⎣ ⎡ k j = = 0 ∑ T T m ( T m ) ( 4 d 3 ) K j ( 1 − d 1 ) T m − K j ⎦ ⎤ = E [ ( 1 − d 1 + 4 d 3 ) T m ] = E [ ( 1 − 4 d 1 ) T m ] 又因为 P ( T m ⩾ ln k 2 ) → 1 P\left(T_m\geqslant\frac{\ln k}{2}\right)\rightarrow 1 P ( T m ⩾ 2 l n k ) → 1 k → ∞ k\rightarrow\infty k → ∞ E [ L j ] → 0 \mathbb{E}[L_j]\rightarrow 0 E [ L j ] → 0 E [ Diam ( Ω ( x , Z ) ) ] → 0 \mathbb{E}\left[\text{Diam}(\Omega(x,Z))\right]\rightarrow 0 E [ Diam ( Ω ( x , Z )) ] → 0
根据 Diam ( Ω ( x , Z ) ) ⩾ 0 \text{Diam}(\Omega(x,Z))\geqslant 0 Diam ( Ω ( x , Z )) ⩾ 0 E Diam ( Ω ( x , Z ) ) → 0 \mathbb{E}\text{Diam}(\Omega(x,Z))\rightarrow 0 E Diam ( Ω ( x , Z )) → 0 P ( Ω ( x , Z ) ⩾ ϵ ) → 0 P\left(\Omega(x,Z)\geqslant\epsilon\right)\rightarrow 0 P ( Ω ( x , Z ) ⩾ ϵ ) → 0 Diam ( Ω ( x , Z ) ) → 0 \text{Diam}(\Omega(x,Z))\rightarrow 0 Diam ( Ω ( x , Z )) → 0
根据定理 6.2 可得随机决策树具有一致性。
再根据引理 6.5 可知由随机决策树集成的随机森林也具有一致性。
定理得证。
