Lecture 9 - 07-04-2020

ĥisERMpredictor

ℓ_{D} \binom{}{ĥ} ≤ min ℓ_{D} (h) +\sqrt{\frac{2}{m} ln \frac{2 H}{δ}} with prob. at least 1 − δ

Now we do it with tree predictors

1.1 Tree predictors

X = {0, 1}^{d} ⟶ Binary classification

h : {0, 1}^{d} ⟶ Binary classification H1

How big is this class? Take the size of codomain power the domain ⟶ |H| = 2^{2^{d}} Can we have a tree predictor that predict every H in this class? For every h : {0, 1}^{d} ⟷ {−1, 1} ∃T

We can build a tree such that h_{T} = h

Figure 1.1: Tree building X = (0, 0, 1, ..., 1) h (x) = −1 x_{1}, x_{2}, x_{3}, ..., x_{d}

I can apply my analisys to this predictors If I run ERM on H

ℓ_{D} \binom{}{ĥ} ≤ min ℓ_{D} \binom{}{ĥ} +\sqrt{\frac{2}{m} 2^{d} ln 2 + ln \frac{2}{δ}} ⟶ ln |H| + ln \frac{2}{δ}

No sense! What we find about training set that we need? Worst case of overfitting m >> 2^{D} = |X| ⇒ training sample larger

PROBLEM: cannot learn from a class to big ( H is too big) I can control H just limiting the number of nodes.

H_{N} ⟶ tree T with at most N node, N << 2^{D} |H_{N}| = ?

|H_{N}| = (# of trees with ≤ N nodes)×(# of test on interval nodes )×( # labels on leaves)

|H_{N}| = ⨂ × d^{M} × 2^{N−M}

N of which N − M are leaves

Figure 1.2: Tree with at most N node

⨂# of binary trees with N nodes, called Catalan Number

1.1.1 Catalan Number

*We are using a binomial *

\frac{1}{N} \binom{2 N − 2}{N − 1} ≤