反向传播

由上图可知，神经元参数的变化会导致损失函数数值的变化。因此，对于输出层第1个神经元，定义

$w_1^{n+1} = (w_1, w_2, w_3, w_4), b_1^{n+1} = b_1$

由于J是一个标量，标量对向量的求导等于标量对向量中各个元素求导，则：

$dw_1^{n+1} = \frac{\partial J}{\partial w_1^{n+1}} = \left[ \frac{\partial J}{\partial w_1} \quad \frac{\partial J}{\partial w_2} \quad \frac{\partial J}{\partial w_3} \quad \frac{\partial J}{\partial w_4} \right]$

$db_1^{n+1} = \frac{\partial J}{\partial b_1}$

由上图可知，n+1层第一个神经元权值$w_1^{n+1}$的变化引起的连锁变化为：

$w_1^{n+1} -> z_1^{n+1} -> (a_1^{n+1}, a_2^{n+1}, a_3^{n+1}) -> J$

所以：

$dw_1^{n+1} = \frac{\partial J}{\partial w_1^{n+1}} = \frac{\partial J}{\partial a_1^{n+1}} \frac{\partial a_1^{n+1}}{\partial z_1^{n+1}} \frac{\partial z_1^{n+1}}{\partial w_1^{n+1}} + \frac{\partial J}{\partial a_2^{n+1}} \frac{\partial a_2^{n+1}}{\partial z_1^{n+1}} \frac{\partial z_1^{n+1}}{\partial w_1^{n+1}} + \frac{\partial J}{\partial a_3^{n+1}} \frac{\partial a_3^{n+1}}{\partial z_1^{n+1}} \frac{\partial z_1^{n+1}}{\partial w_1^{n+1}}$

定义：

$A = \begin{bmatrix} a_1 & a_2 \\ a_3 & a_4 \\ a_5 & a_6 \end{bmatrix} \quad Z = \begin{bmatrix} z_1 & z_2 \\ z_3 & z_4 \\ z_5 & z_6 \end{bmatrix} \quad Y = \begin{bmatrix} y_1 & y_2 \\ y_3 & y_4 \\ y_5 & y_6 \end{bmatrix}$

$z_1^{n+1} = (z_1, z_2), a_1^{n+1} = (a_1, a_2)$ $z_2^{n+1} = (z_3, z_4), a_2^{n+1} = (a_3, a_4)$ $z_3^{n+1} = (z_5, z_6), a_3^{n+1} = (a_5, a_6)$

Z的第一列是第一个样本对应3个神经元的线性输出。

$\left[ \frac{\partial J}{\partial a_1} \quad \frac{\partial J}{\partial a_2} \right]$这个是横着看，同一个神经元的输出，两个样本，就是$a_1^{n+1}$

分别计算$dw_1^{n+1}$的每一项：

$\frac{\partial J}{\partial a_1^{n+1}} \frac{\partial a_1^{n+1}}{\partial z_1^{n+1}} \frac{\partial z_1^{n+1}}{\partial w_1^{n+1}} = \left[ \frac{\partial J}{\partial a_1} \quad \frac{\partial J}{\partial a_2} \right] \begin{bmatrix} \frac{\partial a_1}{\partial z_1} & \frac{\partial a_1}{\partial z_2} \\ \frac{\partial a_2}{\partial z_1} & \frac{\partial a_2}{\partial z_2} \end{bmatrix} \begin{bmatrix} \frac{\partial z_1}{\partial w_1} & \frac{\partial z_1}{\partial w_2} & \frac{\partial z_1}{\partial w_3} & \frac{\partial z_1}{\partial w_4} \end{bmatrix}$

$\frac{\partial J}{\partial a_3^{n+1}} \frac{\partial a_3^{n+1}}{\partial z_1^{n+1}} \frac{\partial z_1^{n+1}}{\partial w_1^{n+1}} = \left[ \frac{\partial J}{\partial a_5} \quad \frac{\partial J}{\partial a_6} \right] \begin{bmatrix} \frac{\partial a_5}{\partial z_1} & \frac{\partial a_5}{\partial z_2} \\ \frac{\partial a_6}{\partial z_1} & \frac{\partial a_6}{\partial z_2} \end{bmatrix} \begin{bmatrix} \frac{\partial z_1}{\partial w_1} & \frac{\partial z_1}{\partial w_2} & \frac{\partial z_1}{\partial w_3} & \frac{\partial z_1}{\partial w_4} \end{bmatrix}$

$\frac{\partial J}{\partial a_2^{n+1}} \frac{\partial a_2^{n+1}}{\partial z_1^{n+1}} \frac{\partial z_1^{n+1}}{\partial w_1^{n+1}} = \left[ \frac{\partial J}{\partial a_3} \quad \frac{\partial J}{\partial a_4} \right] \begin{bmatrix} \frac{\partial a_3}{\partial z_1} & \frac{\partial a_3}{\partial z_2} \\ \frac{\partial a_4}{\partial z_1} & \frac{\partial a_4}{\partial z_2} \end{bmatrix} \begin{bmatrix} \frac{\partial z_1}{\partial w_1} & \frac{\partial z_1}{\partial w_2} & \frac{\partial z_1}{\partial w_3} & \frac{\partial z_1}{\partial w_4} \end{bmatrix}$

$J = y_1 \cdot \log a_1 + y_3 \log a_3 + y_5 \log a_5 + y_2 \log a_2 + y_4 \log a_4 + y_6 \log a_6$

$a_1 = \frac{e^{z_1}}{e^{z_1} + e^{z_3} + e^{z_5}} \quad a_3 = \frac{e^{z_3}}{e^{z_1} + e^{z_3} + e^{z_5}} \quad a_5 = \frac{e^{z_5}}{e^{z_1} + e^{z_3} + e^{z_5}}$