Analyse Discriminante

Classification

Generative Modeling

Ana Karina Fermin

ISEFAR

http://fermin.perso.math.cnrs.fr/

Ouline:

1 k Nearest-Neighbors X

2 K-Fold cross validation, Model selection X

3 Generative Modeling (Naive Bayes, LDA, QDA)

4 Logistic Modeling

5 SVM

6 Tree Based Methods (M. Zetlaoui course, Apprentissage)

Generative Modeling

Bayes formula

p_k(x) = P{X=x|Y =k}P{Y =k}

P{X=x}

Remark: If one knowsthe law of (X,Y) or equivalently ofX giveny and of Y then everything is easy!

Binary Bayes classifier (the best solution)

f^∗(x) =

(+1 ifp+1(x)≥p−1(x)

−1 otherwise

Heuristic: Estimate those quantities and plug the estimations.

By using different models for P{X|Y}, we get different classifiers.

Remark: You can also use your favorite density estimator...

Discriminant Analysis

Discriminant Analysis (Gaussian model)

The densities are modeled as multivariate normal, i.e., P{X|Y =k} ∼ N_µk,Σk

Discriminants fonctions:

gk(x) = ln(P{X|Y =k}) + ln(P{Y =k}) g_k(x) =−1

2(x−µ_k)^tΣ⁻¹_k (x−µ_k)

−d

2ln(2π)−1

2ln(|Σ_k|) + ln(P{Y =k}) QDA (differents Σ_k in each class) and LDA (Σ_k = Σ for all k) Beware: this model can be false but the methodology remains valid!

Discriminant Analysis

Two-dimensional two-category classifier

The probability densities are Gaussian

The effect of any decision rule is to divide the feature space into c decision regions (ici c=2) R₁,R₂, . . . ,R_c

The regions ared separated by decision boundaries

Discriminant Analysis

Two-dimensional four-category classifier

Discriminant Analysis

Estimation

In pratice, we will need to estimateµ_k, Σ_k andPk :=P{Y =k}

The estimate proportionP^ck = ⁿ_n^k = ¹_n^Pn_i=11_{Yi=k}

Maximum likelihood estimate of µ_ck andΣ^ck (explicit formulas) DA classifier

bf_DA(x) =

(+1 ifbg+1 ≥gb−1

−1 otherwise

Decision boundaries: quadratic = degree 2 polynomials.

If one imposes Σ−1= Σ₁ = Σ then the decision boundaries is an linear hyperplan

Discriminant Analysis

Σω1 = Σω2 = Σ

The decision boundaries is an linear hyperplan

Discriminant Analysis

Σ_ω1 6= Σ_ω2

Arbitrary Gaussian distributions lead to Bayes decision boundaries that are general quadratics.

Example: TwoClass Dataset

Synthetic Dataset

Two features/covariates.

Two classes.

Dataset from Applied Predictive Modeling, M. Kuhn and K.

Johnson, Springer

Numerical experiments withR.

Example: LDA

Example: QDA

Naive Bayes

Naive Bayes

Classical algorithm using a crude modeling for P{X|Y}:

Featureindependenceassumption:

P{X|Y}=

d

Y

i=1

P n

X⁽ⁱ⁾ Yo

Simple featurewise model: binomial if binary, multinomial if finite and Gaussian if continuous

If all features are continuous, similar to the previous Gaussian but with a diagonal covariance matrix!

Very simple learning even invery high dimension!

Naive Bayes with density estimation

Example: Naive Bayes

Example: Naive Bayes