【机器学习】公式推导与证明：高斯贝叶斯和逻辑回归的关系

情景1：特定形式的高斯朴素贝叶斯分类器与逻辑回归（Specific Gaussian naive Bayes classifiers and logistic regression）

Consider a specific class of Gaussian naive Bayes classifier where:

• $y$ is a boolean variable following a Bernouli distribution, with parameter $\pi=P(y=1)$ and thus $P(Y=0)=1-\pi$.
• $x=[x_1,\dots,x_D]^T$, with each feature $x_i$ a continous random variable. For each $x_i$, $P(x_i|y=k)$ is a Gaussian distribution $N(\mu_{ik},\sigma_i)$. Note that $\sigma_i$ is the standard deviation of the Gaussian distribution, which does not depend on $k$.
• For all $i \neq j$ and $x_i$ and $x_j$ are conditionally independent given $y$ (so called " naice" classifier).

Question: please show that the relationship between a discriminative classifier (say logistic regression) and the above specific class of Gaussian naive Bayes classifiers is precisely the form used by logistic regression.

中文描述： 当 $i \neq j$ 时 $x_i$ 和 $x_j$ 相互独立，$\sigma$ 依赖于 $i$ 而不依赖于 $k$

思路：

尝试将贝叶斯表达式转化为逻辑回归的一般形式：

Answer:

根据上述特定高斯朴素贝叶斯分类器的假设，以及贝叶斯法则，有：

由给定 $x_i, x_j$ 的条件独立性假设，可得：

再根据 $P(x_i|Y=y_k)$ 服从高斯分布 $N(μ_{ik},σ_i)$，可得：

则：

等价于逻辑回归的一般形式：

其中：

所以该特定形式的高斯贝叶斯分类器正是逻辑回归所使用的形式。

情景2：一般形式的高斯朴素贝叶斯分类器与逻辑回归（General Gaussian naive Bayes classifiers and logistic regression）

Removing the assumption that the standard devition $\sigma_i$ of $P(x_i|y=k)$ does not depend on $k$. That is, for each $x_i$, $P(x_i|y=k)$ is a Gaussian distribution $N(\mu_{ik}, \sigma_{ik})$, where $i=1,\dots,D$ and $k=0,1$.

Question: is the new form of $P(y|x)$ implied by this more general Gaussian naive Bayes classifier still the form used by logistic regression? Derive the new form of $P(y|x)$ to prove your answer.

中文描述： 当 $i \neq j$ 时， $x_i$ 和 $x_j$ 相互独立，$\sigma$ 依赖于 $i$ 和 $k$

Answer:

由上题可得：

再根据 $P(x_i|Y=y_k)$ 服从高斯分布 $N(\mu_{ik}, \sigma_{ik})$，可得：

其中含有 $x_i^2$ 项，所以一般高斯朴素贝叶斯分类器不是逻辑回归所使用的形式。

情景3：高斯贝叶斯分类器与逻辑回归（Gaussian Bayes classifiers and logistic regression）

Now, consider the following assumptions for our Gaussian Bayses classifiers (without "naive").

• $y$ is a boolean variable following a Bernouli distribution, with parameter $\pi=P(y=1)$ and thus $P(Y=0)=1-\pi$.

• $x=[x_1,x_2]^T$, i.e., we only consider two features for each sample, with each feature a continous random variable. $x_1$ and $x_2$ are not conditional independent given y. We assume $P(x_1,x_2|y=k)$ is a bivariate Gaussian distribution $N(\mu_{1k},\mu_{2k},\sigma_1,\sigma_2,\rho)$, where $\mu_{1k}$ and $\mu_{2k}$ are means of $x_1$ and $x_2$. The density of the bivariate Gaussian distribution is:
  $P(x_1,x_2|y=k)=\frac{1}{2\pi\sigma_1\sigma_2\sqrt{1-\rho^2}}\exp[-\frac{\sigma^2(x_1-\mu_{1k})^2+\sigma_1^2(x_2-\mu_{2k})^2-2\rho\sigma_1\sigma_2(x_1-\mu_{1k})(x_2-\mu_{2k})}{2(1-\rho^2)\sigma_1^2\sigma2^2}]$

Question: is the form of $P(y|x)$ implied by such not-so-naive Gaussian Bayes classifiers still the form used by logistic regression? Derive the form of $P(y|x)$ to prove your answer.

中文描述：$x_i$ 和 $x_j$ 不相互独立，$\sigma$ 依赖于 $i$ 而不依赖于 $k$

Answer:

根据上述非朴素的高斯贝叶斯分类器的假设，以及贝叶斯法则，有：

其中 $X=[x_1,x_2]^T$ ，且服从二元高斯分布：

$P(x_1,x_2|Y=k)=\frac{1}{2\pi\sigma_1\sigma_2\sqrt{1-\rho^2}}\exp\left[-\frac{\sigma_2^2(x_1-\mu_{1k})^2+\sigma_1^2(x_2-\mu_{2k})^2-2\rho\sigma_1\sigma_2(x_1-\mu_{1k})(x_2-\mu_{2k})}{2(1-\rho^2)\sigma_1^2\sigma_2^2}\right]$

则：

等价于逻辑回归的一般形式：

其中：

所以非朴素的高斯贝叶斯分类器也正是逻辑回归所使用的形式。