Matrix Derivative
一、标量方程对向量的导数
分子： 为 的标量；分母： 为 的向量
1. 分母布局 denominator layout
与分母的行数相同
\dfrac{\partial f(\boldsymbol{y})}{\partial\boldsymbol{y}}=\begin{pmatrix} \dfrac{\partial f(\boldsymbol{y})}{\partial y_{1}}\\ \dfrac{\partial f(\boldsymbol{y})}{\partial y_{2}} \\ \vdots \\ \dfrac{\partial f(\boldsymbol{y})}{\partial y_{n}} \end{pmatrix} \end{align}$$ #### 2. 分子布局 numerator layout 与分子的行数相同 $$\begin{align} \dfrac{\partial f(\boldsymbol{y})}{\partial\boldsymbol{y}}=\begin{pmatrix} \dfrac{\partial f(\boldsymbol{y})}{\partial y_{1}}, \dfrac{\partial f(\boldsymbol{y})}{\partial y_{2}} ,\dots , \dfrac{\partial f(\boldsymbol{y})}{\partial y_{n}} \end{pmatrix} \end{align}$$ > [!important] 注意 > > 以上两种布局没有本质区别，互为转置；为方便起见，以下的推导默认使用**分母布局**。 ### 二、向量方程对标量的导数 $$\begin{align} \boldsymbol{f}(u)=\begin{pmatrix} f_{1}(u) \\ f_{2}(u) \\ \cdots \\ f_{m}(u) \end{pmatrix} \; {\color{red}\Rightarrow} \; \dfrac{\partial \boldsymbol{f}(u)}{\partial u} =\begin{pmatrix} \dfrac{\partial f_{1}(u)}{\partial u}\quad \dfrac{\partial f_{2}(u)}{\partial u} & \cdots & \dfrac{\partial f_{m}(u)}{\partial u} \end{pmatrix} \end{align}$$ ### 三、向量方程对向量的导数 $$ \boldsymbol{f} (\boldsymbol{y})=\begin{pmatrix} f_{1}(\boldsymbol{y})\\ f_{2}(\boldsymbol{y})\\ \vdots\\f_{m}(\boldsymbol{y})\end{pmatrix}_{(m\times1)} \quad\quad \boldsymbol{y}=\begin{pmatrix}y_{1}\\y_{2}\\\vdots\\y_{n}\end{pmatrix}_{(n\times 1)}$$ 分母布局的结果如下：（其转置（分子布局）实际上为[[雅可比矩阵\|雅可比矩阵]]） $$\begin{align} \dfrac{\partial \boldsymbol{f} (\boldsymbol{y})}{\partial \boldsymbol{y}} =\begin{pmatrix} \dfrac{\partial \boldsymbol{f}(\boldsymbol{y})}{\partial y_{1}}\\ \dfrac{\partial \boldsymbol{f}(\boldsymbol{y})}{\partial y_{2}} \\ \vdots \\ \dfrac{\partial \boldsymbol{f}(\boldsymbol{y})}{\partial y_{n}} \end{pmatrix}_{(n\times1)} = \begin{pmatrix} & \dfrac{\partial f_{1}(\boldsymbol{y})}{\partial y_{1}} & \dfrac{\partial f_{2}(\boldsymbol{y})}{\partial y_{1}} & \dots & \dfrac{\partial f_{m}(\boldsymbol{y})}{\partial y_{1}} \\ & \dfrac{\partial f_{1}(\boldsymbol{y})}{\partial y_{2}} & \dfrac{\partial f_{2}(\boldsymbol{y})}{\partial y_{2}} & \dots & \dfrac{\partial f_{m}(\boldsymbol{y})}{\partial y_{2}} \\ & \dots & \dots & \dots & \dots \\ & \dfrac{\partial f_{1}(\boldsymbol{y})}{\partial y_{n}} & \dfrac{\partial f_{2}(\boldsymbol{y})}{\partial y_{n}} & \dots & \dfrac{\partial f_{m}(\boldsymbol{y})}{\partial y_{n}} \end{pmatrix}_{(n\times m)} \end{align}$$ ### 有关结论（分母布局）\boldsymbol{u} =\begin{pmatrix} u_{1} \ u_{2} \ \vdots \ u_{n} \end{pmatrix}{(n\times 1)} \quad\quad \boldsymbol{y}=\begin{pmatrix}y{1}\y_{2}\\vdots\y_{n}\end{pmatrix}_{(n\times 1)}
$$\begin{align} \\ \dfrac{\partial (\boldsymbol{u}^{T} \boldsymbol{f})}{\partial \boldsymbol{u} } =\boldsymbol{f} \end{align}$$ $$\begin{align} \dfrac{\partial A \boldsymbol{y}}{\partial \boldsymbol{y}}&=A^{T} \end{align}$$ [[二次型\|二次型]]的求导： $$\begin{align} \dfrac{\partial \boldsymbol{y}^{T}A \boldsymbol{y}}{\partial \boldsymbol{y}}=A \boldsymbol{y}+ A^{T} \boldsymbol{y} \end{align}$$ 如果 $A$ 为对称阵，则有 $\dfrac{\partial \boldsymbol{y}^{T}A \boldsymbol{y}}{\partial \boldsymbol{y}}= 2A \boldsymbol{y}$ ### 链式法则 [[导数#复合函数的求导法则\|标量的链式求导]]，注意矩阵不满足乘法交换律 使用分母布局： $$\begin{align} \dfrac{\partial J}{\partial \vec{u}}=\dfrac{\partial \boldsymbol{y}}{\partial \vec{u}} \dfrac{\partial J}{\partial \boldsymbol{y}} \end{align}$$ 离散状态空间方程： $\vec{x}_{[k+1]}=A \vec{x}_{[k]}+B \vec{u}_{[k]}$ 代价函数： $J=\vec{x}_{[k+1]}^{T}\vec{x}_{[k+1]}$ $$\dfrac{\partial J}{\partial \vec{u}}=\dfrac{\partial \vec{x}_{[k+1]}}{\partial \vec{u}_{[k]}}\dfrac{\partial J}{\partial \vec{x}_{[k+1]}}=B^{T}\cdot 2\vec{x}_{[k+1]}$$ ### 实际应用 [[线性回归\|线性回归]] $$\begin{align} \hat{z}&=y_{1}+y_{2}x \quad \quad J=\sum\limits_{i=1}^{n} \left[z_{i}-(y_{1}+y_{2}x_{i})\right]^{2} \end{align}$$ 使用矩阵求解 $$\vec{z}=\begin{pmatrix}z_{1}\\ z_{2}\\ \vdots \\ z_{n}\end{pmatrix}\quad x=\begin{pmatrix} 1& x_{1}\\ 1&x_{2} \\ \vdots&\vdots\\ 1&x_{n} \end{pmatrix}\quad y=\begin{pmatrix}y_{1}\\y_{2}\end{pmatrix}\quad \vec{\hat{z}}=\vec{x} \boldsymbol{y}=\begin{pmatrix} y_{1}+y_{2}x_{1}\\ y_{1}+y_{2}x_{2}\\ \vdots\\ y_{1}+y_{2}x_{n} \end{pmatrix}$$ $$\begin{align} \vec{z}-\vec{\hat{z}}&=\begin{pmatrix} z_{1}-(y_{1}+y_{2}x_{1}) \\ z_{2}-( y_{1}+y_{2}x_{2})\\ \vdots\\ z_{n}-(y_{1}+y_{2}x_{n} ) \end{pmatrix} \\ J&=[\vec{z}-\vec{\hat{z}}]^{T}[\vec{z}-\vec{\hat{z}}] \\ &=[\vec{z}^{T}-\boldsymbol{y}^{T}\vec{x}^{T}][\vec{z}-\vec{x}\boldsymbol{y}] \\ &=\vec{z}^{T}\vec{z}-\boldsymbol{y}^{T}\vec{x}^{T}\vec{z}-\vec{z}^{T}\vec{x}\boldsymbol{y}+\boldsymbol{y}^{T}\vec{x}^{T}\vec{x}\boldsymbol{y} \\ &=\vec{z}^{T}\vec{z}-2\vec{z}^{T}\vec{x}\boldsymbol{y}+\boldsymbol{y}^{T}\vec{x}^{T}\vec{x}\boldsymbol{y} \end{align}$$ $$\begin{align} &\dfrac{\partial J}{\partial \boldsymbol{y}}=-2(\vec{z}^{T}\vec{x})+2 \vec{x}^{T}\vec{x}\boldsymbol{y}= 0 \quad \Rightarrow \quad \boldsymbol{y}=(\vec{x}^{T}\vec{x})^{-1} \vec{x}^{T}\vec{z} \end{align}$$ 如果没有解析解，使用机器学习的梯度下降 定义初始值 $\boldsymbol{y}^{*}$ 循环迭代 $\boldsymbol{y}^{*}=\boldsymbol{y}^{*}-\alpha \nabla$ - 梯度： $\nabla=\dfrac{\partial J}{\partial \boldsymbol{y}}$ - 学习率： $\alpha=\begin{pmatrix} \alpha_{1} & 0\\ 0& \alpha_{2}\end{pmatrix}$ 注意如果没有[[数据处理#归一化\|归一化处理]]，学习率的选取要结合未知向量各值的数量级 归一化处理后：各个未知数用同一个学习率，使用标量 $\alpha$ 即可