1 Lecture 5 - 07-04-2020

1.1 Tree Classifier

Supposed we groped a tree up to this point and we are wandering how to grow it. S Training set (x_{1}, y_{1})...(x_{m}, y_{m}), x_{1} ∈ X

– DISEGNO

S_{ℓ} ≡ {(x_{1}, y_{1}) x_{t} is router to ℓ}

y_{1} ∈ {−1, 1}

S_{ℓ}^{+} ≡ {(x_{1}, y_{1}) ∈ S_{ℓ} : y_{t} = +1}

S_{ℓ}^{−} ≡ {(x_{1}, y_{1}) ∈ S_{ℓ} : y_{t} = −1} S_{ℓ}^{+} ∩ S_{ℓ}^{−} ≡ 0 S_{ℓ} ≡ S_{ℓ}^{+} ∪ S_{ℓ}^{−}

N_{ℓ} = |S_{ℓ}| N_{ℓ}^{+} = |S_{ℓ}^{+}| N_{ℓ}^{−} = |S_{ℓ}^{−}|
N_{ℓ} = N_{ℓ}^{−} + N_{ℓ}^{+}

leaf ℓ classifies all traning example (S_{ℓ})

Y_{ℓ} = \begin{cases}+1, & If N_{ℓ}^{+} ≥ N_{ℓ}^{−} \\ −1, & If otherwise \end{cases}

ℓ makes a mistake on min{N_{ℓ}^{+}, N_{ℓ}^{−}} example in S_{ℓ}

ℓ̂(h_{T}) = \frac{1}{m} · ∑_{ℓ} min\{\frac{N_{ℓ}^{+}}{N_{ℓ}} , \frac{N_{ℓ}^{−}}{N_{ℓ}} \} · N_{ℓ} =

= \frac{1}{m} · ∑_{ℓ} ψ · (\frac{N_{ℓ}+}{N_{ℓ}}) · N_{ℓ} ⟶ \frac{N_{ℓ}^{+}}{N_{ℓ}} = 1 − \frac{N_{ℓ}}{N_{ℓ}??}

where ψ(a) = min{a, 1 − a} a ∈ [0, 1] I want to replace inner node with other leaves.

– DISEGNO – How is traning error going to change? (when i replace inner nodes with other leaves) I’m hoping my algorithm is not going to overfit (if training error goes to 0 also testing error goes to 0).

1.2 Jensen’s inequality

If ψ is a concave function ⟶ (like log or \sqrt[2]{..} ) Also ψ is a function that map 0 to 1, ⟶ ψ [0, 1] → ℝ

ψ(α·a+(1−α)·b) ≥ α·ψ(a)+(1−α)·ψ(b) Also 2° derivative is negative

– DISEGNO –

ℓ̂(h_{T}) = \frac{1}{m} · ∑_{ℓ} ψ(\frac{N_{ℓ}^{+}}{N_{ℓ}}) · N_{ℓ}

Look a single contribution fo a leaf ℓ to training error

ψ(\frac{N_{ℓ}^{+}}{N_{ℓ}}) · N_{ℓ} = ψ(\frac{N_{ℓ}^{′+}}{N′_{ℓ}} · \frac{N′_{ℓ}}{N_{ℓ}} + \frac{N_{ℓ}”^{+}}{N_{ℓ}”} · \frac{N_{ℓ}”}{N_{ℓ}}) · N_{ℓ}

where \frac{N′_{ℓ}}{N_{ℓ}} = α and \frac{N_{ℓ}”}{N_{ℓ}} = 1 − α  so \frac{N′_{ℓ}}{N_{ℓ}} + \frac{N_{ℓ}”}{N_{ℓ}} = 1  ⟶ α + 1 − α = 1

N_{ℓ′}^{+} + N_{ℓ”}^{+} = N_{ℓ}

I want to check function min concave between 0 and 1.

min(0, 1) = 0 ψ(a) = min(α, 1 − α)

– DISEGNO –

This is a concave function and now I can apply Jensen’s inquality

ψ(\frac{N_{ℓ}^{+}}{N_{ℓ}}) · N_{ℓ} ≥ (\frac{N′_{ℓ}}{N_{ℓ}} · ψ(\frac{N_{ℓ}^{′+}}{N′_{ℓ}}) + \frac{N_{ℓ}”}{N_{ℓ}} · ψ(\frac{N_{ℓ}”^{+}}{N_{ℓ}”} )) · N_{ℓ} = ψ(\frac{N_{ℓ}^{′+}}{N′_{ℓ}}) · N′_{ℓ} + ψ(\frac{N_{ℓ}”^{+}}{N_{ℓ}”} ) · N_{ℓ}”

3