Application

• credit scoring
• medical diagnosis
• handwritten character recognition

Regression?

Why not regression

• penalize examples that are "too correct"
• multiple class: 1, 2, 3 (classes 1 and 2 are not necessarily closer than classes 1 and 3)

Ideal alternatives

Gradient descent cannot be applied! - indifferentiable

Two classes

Probabilistic generative model - Bayes' Theorem

$P(C_1|x) = \frac{P(x|C_1) P(C_1)}{P(x|C_1) P(C_1) + P(x|C_2) P(C_2)}$

• estimating the probabilities from training data
• if assume all the dimensions are independent -> Naive Bayes Classifier

$P(C_1)$ & $P(C_2)$

Easily calculated from training data

$P(x|C_1)$ & $P(x|C_2)$

• Assume that the points are sampled from a Gaussian distribution (it's okay to use other distributions as well. e.g., Bernoulli distribution for binary features.)
• Find the Gaussian distribution behind each class (Maximum likelihood)

$L(\mu, \sum) = f_{\mu, \sum} (x^1) f_{\mu, \sum} (x^2) \dots f_{\mu, \sum} (x^n)$

$\mu^*, \sum^* = \arg \max_{\mu, \sum} L(\mu, \sum)$

Easily overfitting if a distribution for each class, since a covariance matrix can have too many parameters (# features * # features) -> the same covariance matrix for each class.

$L(\mu_1, \mu_2, \sum) = f_{\mu^1, \sum} (x^1) f_{\mu^1, \sum} (x^2) \dots f_{\mu^1, \sum} (x^{n^1}) \times f_{\mu^2, \sum} (x^{n^1+1}) f_{\mu^1, \sum} (x^{n^1+2}) \dots f_{\mu^1, \sum} (x^{n^1 + n^2})$

• Find the probability of the new point based on the estimated distribution

Finally

-> How about we directly find w and b?