神经网络基础——矩阵求导运算

HXhlx
5 阅读3分钟

本篇文章不涉及过多理论推导。关于本篇文章，在实践中会用即可。 矩阵求导有以下9种可能情况：

自变量/因变量标量yy向量y\mathbf y矩阵YY
标量xxdydx\frac{dy}{dx}dydx\frac{d\mathbf y}{dx}dYdx\frac{dY}{dx}
向量x\mathbf xdydx\frac{dy}{d\mathbf x}dydx\frac{d\mathbf y}{d\mathbf x}dYdx\frac{dY}{d\mathbf x}
矩阵XXdydX\frac{dy}{dX}dydX\frac{d\mathbf y}{dX}dYdX\frac{dY}{dX}

以下分别讨论。

一阶导数

标量对标量求导

这里是最简单的情况，本篇默认读者已掌握，不再赘述。

向量对标量求导

行向量对标量求导

设行向量y=(y1,y2,⋯ ,yn)\mathbf y=(y_1,y_2,\cdots,y_n)，则有

∂y∂x=[∂y1∂x∂y2∂x⋯∂yn∂x]\frac{\partial\mathbf y}{\partial x}=\begin{bmatrix} \frac{\partial y_1}{\partial x} & \frac{\partial y_2}{\partial x} & \cdots & \frac{\partial y_n}{\partial x} \end{bmatrix}

列向量对标量求导

设列向量y=[y1y2⋮yn]\mathbf y=\begin{bmatrix} y_1\\ y_2\\ \vdots\\ y_n \end{bmatrix}，则有

∂y∂x=[∂y1∂x∂y2∂x⋮∂yn∂x]\frac{\partial\mathbf y}{\partial x}=\begin{bmatrix} \frac{\partial y_1}{\partial x}\\ \frac{\partial y_2}{\partial x}\\ \vdots\\ \frac{\partial y_n}{\partial x} \end{bmatrix}

矩阵对标量求导

设Y=[y11y12⋯y1ny21y22⋯y2n⋮⋮⋱⋮ym1ym2⋯ymn]Y=\begin{bmatrix} y_{11} & y_{12} & \cdots & y_{1n}\\ y_{21} & y_{22} & \cdots & y_{2n}\\ \vdots & \vdots & \ddots & \vdots\\ y_{m1} & y_{m2} & \cdots & y_{mn} \end{bmatrix}，则有

∂Y∂x=[∂y11∂x∂y12∂x⋯∂y1n∂x∂y21∂x∂y22∂x⋯∂y2n∂x⋮⋮⋱⋮∂ym1∂x∂ym2∂x⋯∂ymn∂x]\frac{\partial Y}{\partial x}=\begin{bmatrix} \frac{\partial y_{11}}{\partial x} & \frac{\partial y_{12}}{\partial x} & \cdots & \frac{\partial y_{1n}}{\partial x}\\ \frac{\partial y_{21}}{\partial x} & \frac{\partial y_{22}}{\partial x} & \cdots & \frac{\partial y_{2n}}{\partial x}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y_{m1}}{\partial x} & \frac{\partial y_{m2}}{\partial x} & \cdots & \frac{\partial y_{mn}}{\partial x} \end{bmatrix}

标量对向量求导

标量对行向量求导

设x=(x1,x2,⋯ ,xm)\mathbf x=(x_1,x_2,\cdots,x_m)，则有

∂y∂x=[∂y∂x1∂y∂x2⋯∂y∂xm]\frac{\partial y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial y}{\partial x_1} & \frac{\partial y}{\partial x_2} & \cdots & \frac{\partial y}{\partial x_m} \end{bmatrix}

标量对列向量求导

设x=[x1x2⋮xm]\mathbf x=\begin{bmatrix} x_1\\ x_2\\ \vdots\\ x_m \end{bmatrix}，则有

∂y∂x=[∂y∂x1∂y∂x2⋮∂y∂xm]\frac{\partial y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial y}{\partial x_1}\\ \frac{\partial y}{\partial x_2}\\ \vdots\\ \frac{\partial y}{\partial x_m} \end{bmatrix}

标量对矩阵求导

设X=[x11x12⋯x1nx21x22⋯x2n⋮⋮⋱⋮xm1xm2⋯xmn]X=\begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1n}\\ x_{21} & x_{22} & \cdots & x_{2n}\\ \vdots & \vdots & \ddots & \vdots\\ x_{m1} & x_{m2} & \cdots & x_{mn} \end{bmatrix}，则有

∂y∂X=[∂y∂x11∂y∂x12⋯∂y∂x1n∂y∂x21∂y∂x22⋯∂y∂x2n⋮⋮⋱⋮∂y∂xm1∂y∂xm2⋯∂y∂xmn]\frac{\partial y}{\partial X}=\begin{bmatrix} \frac{\partial y}{\partial x_{11}} & \frac{\partial y}{\partial x_{12}} & \cdots & \frac{\partial y}{\partial x_{1n}}\\ \frac{\partial y}{\partial x_{21}} & \frac{\partial y}{\partial x_{22}} & \cdots & \frac{\partial y}{\partial x_{2n}}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y}{\partial x_{m1}} & \frac{\partial y}{\partial x_{m2}} & \cdots & \frac{\partial y}{\partial x_{mn}} \end{bmatrix}

向量对向量求导

列向量对行向量求导(雅可比矩阵)

设y=[y1y2⋮yn],x=(x1,x2,⋯ ,xm)\mathbf y=\begin{bmatrix} y_1\\ y_2\\ \vdots\\ y_n \end{bmatrix},\mathbf x=(x_1,x_2,\cdots,x_m)，则有

∂y∂x=[∂y1∂x1∂y1∂x2⋯∂y1∂xm∂y2∂x1∂y2∂x2⋯∂y2∂xm⋮⋮⋱⋮∂yn∂x1∂yn∂x2⋯∂yn∂xm]\frac{\partial\mathbf y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial y_1}{\partial x_1} & \frac{\partial y_1}{\partial x_2} & \cdots & \frac{\partial y_1}{\partial x_m}\\ \frac{\partial y_2}{\partial x_1} & \frac{\partial y_2}{\partial x_2} & \cdots & \frac{\partial y_2}{\partial x_m}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y_n}{\partial x_1} & \frac{\partial y_n}{\partial x_2} & \cdots & \frac{\partial y_n}{\partial x_m} \end{bmatrix}

上式也称为雅可比矩阵

行向量对列向量求导

设y=(y1,y2,⋯ ,yn),x=[x1x2⋮xm]\mathbf y=(y_1,y_2,\cdots,y_n),\mathbf x=\begin{bmatrix} x_1\\ x_2\\ \vdots\\ x_m \end{bmatrix}，则有

∂y∂x=[∂y1∂x1∂y2∂x1⋯∂yn∂x1∂y1∂x2∂y2∂x2⋯∂yn∂x2⋮⋮⋱⋮∂y1∂xm∂y2∂xm⋯∂yn∂xm]\frac{\partial\mathbf y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial y_1}{\partial x_1} & \frac{\partial y_2}{\partial x_1} & \cdots & \frac{\partial y_n}{\partial x_1}\\ \frac{\partial y_1}{\partial x_2} & \frac{\partial y_2}{\partial x_2} & \cdots & \frac{\partial y_n}{\partial x_2}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y_1}{\partial x_m} & \frac{\partial y_2}{\partial x_m} & \cdots & \frac{\partial y_n}{\partial x_m} \end{bmatrix}

行向量对行向量求导

设y=(y1,y2,⋯ ,yn),x=(x1,x2,⋯ ,xm)\mathbf y=(y_1,y_2,\cdots,y_n),\mathbf x=(x_1,x_2,\cdots,x_m)，则有

∂y∂x=[∂y∂x1∂y∂x2⋯∂y∂xm]\frac{\partial\mathbf y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial\mathbf{y}}{\partial x_1} & \frac{\partial\mathbf{y}}{\partial x_2} & \cdots & \frac{\partial\mathbf{y}}{\partial x_m} \end{bmatrix}

其中∂yxi\frac{\partial\mathbf{y}}{x_i}的定义见行向量对标量求导

列向量对列向量求导

设y=[y1y2⋮yn],x=[x1x2⋮xm]\mathbf y=\begin{bmatrix} y_1\\ y_2\\ \vdots\\ y_n \end{bmatrix},\mathbf x=\begin{bmatrix} x_1\\ x_2\\ \vdots\\ x_m \end{bmatrix}，则有

∂y∂x=[∂y∂x1∂y∂x2⋮∂y∂xm]\frac{\partial\mathbf y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial\mathbf{y}}{\partial x_1}\\ \frac{\partial\mathbf{y}}{\partial x_2}\\ \vdots\\ \frac{\partial\mathbf{y}}{\partial x_m} \end{bmatrix}

其中∂yxi\frac{\partial\mathbf{y}}{x_i}的定义见列向量对标量求导

矩阵对向量求导

矩阵对行向量求导

设Y=[y11y12⋯y1ny21y22⋯y2n⋮⋮⋱⋮ym1ym2⋯ymn],x=(x1,x2,⋯ ,xc)Y=\begin{bmatrix} y_{11} & y_{12} & \cdots & y_{1n}\\ y_{21} & y_{22} & \cdots & y_{2n}\\ \vdots & \vdots & \ddots & \vdots\\ y_{m1} & y_{m2} & \cdots & y_{mn} \end{bmatrix},\mathbf x=(x_1,x_2,\cdots,x_c)，则有

∂Y∂x=[∂Y∂x1∂Y∂x2⋯∂Y∂xc]\frac{\partial Y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial Y}{\partial x_1} & \frac{\partial Y}{\partial x_2} & \cdots & \frac{\partial Y}{\partial x_c} \end{bmatrix}

其中∂Yxi\frac{\partial Y}{x_i}的定义见矩阵对标量求导

矩阵对列向量求导

设Y=[y11y12⋯y1ny21y22⋯y2n⋮⋮⋱⋮ym1ym2⋯ymn],x=[x1x2⋮xr]Y=\begin{bmatrix} y_{11} & y_{12} & \cdots & y_{1n}\\ y_{21} & y_{22} & \cdots & y_{2n}\\ \vdots & \vdots & \ddots & \vdots\\ y_{m1} & y_{m2} & \cdots & y_{mn} \end{bmatrix},\mathbf x=\begin{bmatrix} x_1\\ x_2\\ \vdots\\ x_r \end{bmatrix}，则有

∂Y∂x=[∂Y∂x1∂Y∂x2⋮∂Y∂xr]\frac{\partial Y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial Y}{\partial x_1}\\ \frac{\partial Y}{\partial x_2}\\ \vdots\\ \frac{\partial Y}{\partial x_r} \end{bmatrix}

其中∂Yxi\frac{\partial Y}{x_i}的定义见矩阵对标量求导

向量对矩阵求导

行向量对矩阵求导

设y=(y1,y2,⋯ ,yn),X=[x11x12⋯x1cx21x22⋯x2c⋮⋮⋱⋮xr1xr2⋯xrc]\mathbf y=(y_1,y_2,\cdots,y_n),X=\begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1c}\\ x_{21} & x_{22} & \cdots & x_{2c}\\ \vdots & \vdots & \ddots & \vdots\\ x_{r1} & x_{r2} & \cdots & x_{rc} \end{bmatrix}，则有

∂y∂X=[∂y∂x11∂y∂x12⋯∂y∂x1c∂y∂x21∂y∂x22⋯∂y∂x2c⋮⋮⋱⋮∂y∂xr1∂y∂xr2⋯∂y∂xrc]\frac{\partial\mathbf y}{\partial X}=\begin{bmatrix} \frac{\partial\mathbf y}{\partial x_{11}} & \frac{\partial\mathbf y}{\partial x_{12}} & \cdots & \frac{\partial\mathbf y}{\partial x_{1c}}\\ \frac{\partial\mathbf y}{\partial x_{21}} & \frac{\partial\mathbf y}{\partial x_{22}} & \cdots & \frac{\partial\mathbf y}{\partial x_{2c}}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial\mathbf y}{\partial x_{r1}} & \frac{\partial\mathbf y}{\partial x_{r2}} & \cdots & \frac{\partial\mathbf y}{\partial x_{rc}} \end{bmatrix}

其中∂yxij\frac{\partial\mathbf y}{x_{ij}}的定义见行向量对标量求导

列向量对矩阵求导

设y=[y1y2⋮yn],X=[x11x12⋯x1cx21x22⋯x2c⋮⋮⋱⋮xr1xr2⋯xrc]\mathbf y=\begin{bmatrix} y_1\\ y_2\\ \vdots\\ y_n \end{bmatrix},X=\begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1c}\\ x_{21} & x_{22} & \cdots & x_{2c}\\ \vdots & \vdots & \ddots & \vdots\\ x_{r1} & x_{r2} & \cdots & x_{rc} \end{bmatrix}，则有

∂y∂X=[∂y∂x11∂y∂x12⋯∂y∂x1c∂y∂x21∂y∂x22⋯∂y∂x2c⋮⋮⋱⋮∂y∂xr1∂y∂xr2⋯∂y∂xrc]\frac{\partial\mathbf y}{\partial X}=\begin{bmatrix} \frac{\partial\mathbf y}{\partial x_{11}} & \frac{\partial\mathbf y}{\partial x_{12}} & \cdots & \frac{\partial\mathbf y}{\partial x_{1c}}\\ \frac{\partial\mathbf y}{\partial x_{21}} & \frac{\partial\mathbf y}{\partial x_{22}} & \cdots & \frac{\partial\mathbf y}{\partial x_{2c}}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial\mathbf y}{\partial x_{r1}} & \frac{\partial\mathbf y}{\partial x_{r2}} & \cdots & \frac{\partial\mathbf y}{\partial x_{rc}} \end{bmatrix}

其中∂yxij\frac{\partial\mathbf y}{x_{ij}}的定义见列向量对标量求导

矩阵对矩阵求导

设Y=[y11y12⋯y1ny21y22⋯y2n⋮⋮⋱⋮ym1ym2⋯ymn],X=[x11x12⋯x1cx21x22⋯x2c⋮⋮⋱⋮xr1xr2⋯xrc]Y=\begin{bmatrix} y_{11} y_{12} \cdots y_{1n}\\ y_{21} y_{22} \cdots y_{2n}\\ \vdots \vdots \ddots \vdots\\ y_{m1} y_{m2} \cdots y_{mn} \end{bmatrix},X=\begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1c}\\ x_{21} & x_{22} & \cdots & x_{2c}\\ \vdots & \vdots & \ddots & \vdots\\ x_{r1} & x_{r2} & \cdots & x_{rc} \end{bmatrix}，则有

∂Y∂X=[∂Y∂x11∂Y∂x12⋯∂Y∂x1c∂Y∂x21∂Y∂x22⋯∂Y∂x2c⋮⋮⋱⋮∂Y∂xr1∂Y∂xr2⋯∂Y∂xrc]\frac{\partial Y}{\partial X}=\begin{bmatrix} \frac{\partial Y}{\partial x_{11}} & \frac{\partial Y}{\partial x_{12}} & \cdots & \frac{\partial Y}{\partial x_{1c}}\\ \frac{\partial Y}{\partial x_{21}} & \frac{\partial Y}{\partial x_{22}} & \cdots & \frac{\partial Y}{\partial x_{2c}}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial Y}{\partial x_{r1}} & \frac{\partial Y}{\partial x_{r2}} & \cdots & \frac{\partial Y}{\partial x_{rc}} \end{bmatrix}

其中∂Yxij\frac{\partial Y}{x_{ij}}的定义见矩阵对标量求导

二阶导数

二阶导数可以视作对一阶导数的导数，从而推导出来。这里要介绍的是一个特别的导数结果，称为海森矩阵（Hessian矩阵）。

海森矩阵

设标量yy和行向量x=(x1,x2,⋯ ,xm)\mathbf{x}=(x_1,x_2,\cdots,x_m)。Hessian矩阵定义为yy对x\mathbf{x}的二阶导（第二次求导要转置行向量x\mathbf{x}），即

H(y)=∂2y∂x2=[∂2y∂x12∂2y∂x1∂x2⋯∂2y∂x1∂xm∂2y∂x2∂x1∂2y∂x22⋯∂2y∂x2∂xm⋮⋮⋱⋮∂2y∂xm∂x1∂2y∂xm∂x2⋯∂2y∂xm2]\begin{aligned} H(y)&=\frac{\partial^2 y}{\partial\mathbf{x}^2}\\ &=\begin{bmatrix} \frac{\partial^2 y}{\partial x_1^2} & \frac{\partial^2 y}{\partial x_1 \partial x_2} & \cdots & \frac{\partial^2 y}{\partial x_1 \partial x_m}\\ \frac{\partial^2 y}{\partial x_2 \partial x_1} & \frac{\partial^2 y}{\partial x_2^2} & \cdots & \frac{\partial^2 y}{\partial x_2 \partial x_m}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial^2 y}{\partial x_m \partial x_1} & \frac{\partial^2 y}{\partial x_m \partial x_2} & \cdots & \frac{\partial^2 y}{\partial x_m^2} \end{bmatrix} \end{aligned}
avatar
文章
阅读
粉丝