Lecture 5 - 07-04-2020

1.1 Tree Classifier

Supposed we groped a tree up to this point and we are wandering how to grow it. S Training set (x_{1}, y_{1})...(x_{m}, y_{m}), x_{1} ∈ X

Figure 1.1: Example of domain of K_{NN}

S_{ℓ} ≡ {(x_{1}, y_{1}) x_{t} is router to ℓ}

y_{1} ∈ {−1, 1}

S_{ℓ}^{+} ≡ {(x_{1}, y_{1}) ∈ S_{ℓ} : y_{t} = +1}

S_{ℓ}^{−} ≡ {(x_{1}, y_{1}) ∈ S_{ℓ} : y_{t} = −1} S_{ℓ}^{+} ∩ S_{ℓ}^{−} ≡ 0 S_{ℓ} ≡ S_{ℓ}^{+} ∪ S_{ℓ}^{−}

N_{ℓ} = |S_{ℓ}| N_{ℓ}^{+} = |S_{ℓ}^{+}| N_{ℓ}^{−} = |S_{ℓ}^{−}|
N_{ℓ} = N_{ℓ}^{−} + N_{ℓ}^{+}

leaf ℓ classifies all traning example (S_{ℓ})

Y_{ℓ} = \begin{cases}+1, & If N_{ℓ}^{+} ≥ N_{ℓ}^{−} \\ −1, & If otherwise \end{cases}

ℓ makes a mistake on min{N_{ℓ}^{+}, N_{ℓ}^{−}} example in S_{ℓ}

ℓ̂(h_{T}) = \frac{1}{m} · ∑_{ℓ} min\{\frac{N_{ℓ}^{+}}{N_{ℓ}} , \frac{N_{ℓ}^{−}}{N_{ℓ}} \} · N_{ℓ} =

= \frac{1}{m} · ∑_{ℓ} ψ · (\frac{N_{ℓ}+}{N_{ℓ}}) · N_{ℓ} ⟶ \frac{N_{ℓ}^{+}}{N_{ℓ}} = 1 − \frac{N_{ℓ}}{N_{ℓ}??}

where ψ(a) = min{a, 1 − a} a ∈ [0, 1] I want to replace inner node with other leaves.

Figure 1.2: Example of domain of K_{NN}

How is traning error going to change? (when i replace inner nodes with other leaves) I’m hoping my algorithm is not going to overfit (if training error goes to 0 also testing error goes to 0).

1.2 Jensen’s inequality

If ψ is a concave function ⟶ (like log or \sqrt[2]{..} )

Figure 1.3: Example of domain of K_{NN}

Also ψ is a function that map 0 to 1, ⟶ ψ [0, 1] → ℝ ψ(α·a+(1−α)·b) ≥ α·ψ(a)+(1−α)·ψ(b) Also 2° derivative is negative

Figure 1.4: Example of domain of K_{NN}

ℓ̂(h_{T}) = \frac{1}{m} · ∑_{ℓ} ψ(\frac{N_{ℓ}^{+}}{N_{ℓ}}) · N_{ℓ}

Look a single contribution fo a leaf ℓ to training error

ψ(\frac{N_{ℓ}^{+}}{N_{ℓ}}) · N_{ℓ} = ψ(\frac{N_{ℓ}^{′+}}{N′_{ℓ}} · \frac{N′_{ℓ}}{N_{ℓ}} + \frac{N_{ℓ}”^{+}}{N_{ℓ}”} · \frac{N_{ℓ}”}{N_{ℓ}}) · N_{ℓ}

where \frac{N′_{ℓ}}{N_{ℓ}} = α and \frac{N_{ℓ}”}{N_{ℓ}} = 1 − α so \frac{N′_{ℓ}}{N_{ℓ}} + \frac{N_{ℓ}”}{N_{ℓ}} = 1 ⟶ α + 1 − α = 1 N_{ℓ′}^{+} + N_{ℓ”}^{+} = N_{ℓ} I want to check function min concave between 0 and 1.

min(0, 1) = 0 ψ(a) = min(α, 1 − α)

Figure 1.5: Example of domain of K_{NN}

This is a concave function and now I can apply Jensen’s inquality

ψ(\frac{N_{ℓ}^{+}}{N_{ℓ}}) · N_{ℓ} ≥ (\frac{N′_{ℓ}}{N_{ℓ}} · ψ(\frac{N_{ℓ}^{′+}}{N′_{ℓ}}) + \frac{N_{ℓ}”}{N_{ℓ}} · ψ(\frac{N_{ℓ}”^{+}}{N_{ℓ}”} )) · N_{ℓ} =

= ψ(\frac{N_{ℓ}^{′+}}{N′_{ℓ}}) · N′_{ℓ} + ψ(\frac{N_{ℓ}”^{+}}{N_{ℓ}”} ) · N_{ℓ}”

This are the contribuion of ℓ′ and ℓ” to the training error Every time i split my tree my training error is never going to increase since we have a concave function. Whenever I’m growing my tree training error is going to be smaller.

Every time a leaf is expanded the training error never goes up. (Hopelly will go down) I’ll should always grow the tree by expanding leave that decrease the training error as much as possible. If i take the effort of growing the tree i should get benefits. I can imaging that if i grow the tree at random my training error is going to drop down error (but maybe will derive overfitting). For now is just an intuition since we will introduced statistical learning model.

Could be complicated and tree big may have 100 leave and there could be many way of associating a test with that leaves. I can spent a lot of time to select which leave is the best promising to split.

• Grow the tree by expanding leave that decrease the training error as much as possible

• In general we can assume:

greedy algorithm at each step pick the pair leaf and test that cause (approximative) the largest decrease in training error

In practise we want optimise this all the way since it’s time expensive. That’s the approximately since we are not every time sure.

— MANCA PARTE —

p = 0.8 q = 1 r = 1 α = 60% Net Change in number of mistakes

ψ(p) − (α · ψ(q) + (1 − α) · ψ(r)) =
ℓ −  ℓ′ +  ℓ”

Fraction of example miss classified ℓ− error ℓ′+ error ℓ”

= 0.2 − (\frac{1}{2} · 0.4 + \frac{1}{2} · 0) = 0

— DISEGNO —

Figure 1.6: Example of domain of K_{NN}

Idea is to replace minimum function with convex combination.

ψ(α) = min {α, 1 − α} ψ(a) ≥ ψ(α)

\begin{cases}ψ_{1}(α) = 2 · α · (1 − α) ⟶ GNI \\ ψ_{2}(α) = −\frac{α}{2} · ln α − \frac{1−α}{2} · ln(1 − α) ⟶ ENT ROP Y \\ ψ_{3}(α) =\sqrt{α · (1 − α)} \end{cases}

All this functions has this shape (concave???)

– DISEGNO – In practise Machine Learning algorithm use GNI or entropy to control the split

1.3 Tree Predictor

• Multi class classification |Y | > 2 ⟶ take majority

• Regression Y = ℝ ⟶ take average of labels in S_{ℓ}

I still take majority among different classes. Take average of labels in S_{ℓ} Unless \frac{N_{ℓ}^{+}}{N_{ℓ}} ∈ 0, 1 ∀ leaves ℓ, ℓ̂(h_{T}) > 0 Unless leaves are "pured", the training error will be bigger than 0.

In general, i can always write ℓ̂(h_{t}) to 0 by growing enough the tree unless there are x_{1} in the Time Series such that (x_{t}, y_{t})(x_{t}, y′_{t}) with y_{t} ≠ y′_{t} both occur.

— DISEGNO —-

if(x_{1} = α) ∧ (x_{2} =≥ α) ∨ (x_{1} = b) ∨ (x_{1} = c) ∧ (x_{3} = y)

then predict 1

else

then predict -1

— Picture of tree classifier of iris dataset. — I’m using due attribute at the time. Each data point is a flower and i can measure how petal and sepal are long. I can use two attribute and i test this two. I can see the plot of the tree classifier (second one) making test splitting data space into region that has this sort of “blackish” shape ( like boxes: blue box, red box, yellow box) A good exercise in which I want to reconstruct the tree given this picture.

1.4 Statistical model for Machine Learning

To understand Tree classifier, nearest neighbour and other algorithm... It’s important to understand that the only way to have a guideline in which model to choose.

This mathematical model are developed to learning and choose learning algorithm.

Now let start with theoretical model.

• How example (x, y) are generated to create test set and training set? We get the dataset but we need to have a mathematical model for this process. (x, y) are drawn from a fixed but unknown probability distribution on the pairs X and Y (X data space, Y label set o label space)

• Why X should be random?

In general we assumed that not all the x in X are equally likely to be observed. I have some distribution over my data point and this said that I’m most like to get a datapoint to another.

• How much label?

Often labels are not determined uniquely by their datapoints because labels are given by human that have their subjective thoughts and also natural phenomena. Labels are stochastic phenomena given a datapoint: i will have a distribution.

We’re going to write (in capital) (X, Y) since they are random variable drawn from D on X ·Y A dataset (X_{1}, Y_{1})...(X_{m}, Y_{m}) they are drawn independently from D (distribution on examples) When I get a training the abstraction of process collecting a training set D is a joint probability distribution over X · Y where D_{x} is the marginal over X → D_{y}|x (conditional of Y given X). I can divided my draw in two part. I draw sample and label from conditional.?? Any dataset ( training or test ) is a random sample (campione casuale) in the statistical sense ⟶ so we can use all stastical tools to make inference.