逻辑回归

2023-02-22 172 阅读1分钟

什么是逻辑回归

逻辑回归是有监督的，判别式的参数模型

核心公式

\eta(\boldsymbol{x})=\frac{1}{1+e^{-\left(\boldsymbol{w}^T \boldsymbol{x}+b\right)}}

对于二分类问题，可以视作标签为1的概率。

求解

首先基于二项分布，可以求出单个样本的概率

p(y \mid \boldsymbol{x} ; \boldsymbol{\theta})=\eta(\boldsymbol{x} ; \boldsymbol{\theta})^y(1-\eta(\boldsymbol{x} ; \boldsymbol{\theta}))^{1-y}, \quad y=0,1,

然后利用最大似然概率求解

L(\boldsymbol{\theta}):=\prod_{i=1}^n p\left(y_i \mid \boldsymbol{x}_i ; \boldsymbol{\theta}\right)

取对数后得

\begin{aligned} \ell(\boldsymbol{\theta}) & :=\log L(\boldsymbol{\theta}) \\ & =\sum_{i=1}^n \log p\left(y_i \mid \boldsymbol{x}_i ; \boldsymbol{\theta}\right) \\ & =\sum_{i=1}^n\left[y_i \log \left(\frac{1}{1+e^{-\boldsymbol{\theta}^T \tilde{\boldsymbol{x}}_i}}\right)+\left(1-y_i\right) \log \left(\frac{e^{-\boldsymbol{\theta}^T \tilde{\boldsymbol{x}}_i}}{1+e^{-\boldsymbol{\theta}^T \tilde{\boldsymbol{x}}_i}}\right)\right] . \end{aligned}

求 $\nabla l=0$

正则化

L1正则化

L1正则化又称Lasso回归，假设 $\omega$ 服从拉普拉斯回归

f(w \mid \mu, b)=\frac{1}{2 b} \exp \left(-\frac{|w-\mu|}{b}\right)

所以原公式写为

\begin{aligned} L(w) & =P(y \mid w, x) P(w) \\ & =\prod_{i=1}^N p\left(x_i\right)^{y_i}\left(1-p\left(x_i\right)\right)^{1-y_i} \prod_{j=1}^d \frac{1}{2 b} \exp \left(-\frac{\left|w_j\right|}{b}\right) \end{aligned}

log后取负得到

-\ln L(w)=-\sum_i\left[y_i \ln p\left(x_i\right)+\left(1-y_i\right) \ln \left(1-p\left(x_i\right)\right)\right]+\frac{1}{2 b^2} \sum_j\left|w_j\right|

L2正则化

L2正则化又称ridge回归（岭回归），假设 $\omega$ 服从高斯分布

f(w \mid \mu, \sigma)=\frac{1}{\sqrt{2 \pi} \sigma} \exp \left(-\frac{(w-\mu)^2}{2 \sigma^2}\right)

所以原公式写为

\begin{aligned} L(w) & =P(y \mid w, x) P(w) \\ & =\prod_{i=1}^N p\left(x_i\right)^{y_i}\left(1-p\left(x_i\right)\right)^{1-y_i} \prod_{j=1}^d \frac{1}{\sqrt{2 \pi} \sigma} \exp \left(-\frac{w_j^2}{2 \sigma^2}\right) \\ & =\prod_{i=1}^N p\left(x_i\right)^{y_i}\left(1-p\left(x_i\right)\right)^{1-y_i} \frac{1}{\sqrt{2 \pi} \sigma} \exp \left(-\frac{w^T w}{2 \sigma^2}\right) \end{aligned}

取 In 再取负，得到目标函数:

-\ln L(w)=-\sum_i\left[y_i \ln p\left(x_i\right)+\left(1-y_i\right) \ln \left(1-p\left(x_i\right)\right)\right]+\frac{1}{2 \sigma^2} w^T w