Statistical Methods for Machine Learning

Data Science and Economics

Andrea Ierardi

Lecture notes

Università degli Studi di Milano

Milan 12/04/2020 Contents

1 Lecture 1 - 09-03-2020 3 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Lecture 2 - 07-04-2020 6 2.1 Argomento . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.1 Absolute Loss . . . . . . . . . . . . . . . . . . . . . . . 6 2.2.2 Square Loss . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2.3 Example of information of square loss . . . . . . . . . . 7 2.2.4 labels and losses . . . . . . . . . . . . . . . . . . . . . . 9 2.2.5 Example TF(idf) documents encoding . . . . . . . . . 10

3 Lecture 3 - 07-04-2020 12

3.1 Overfitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.1.1 Noise in the data . . . . . . . . . . . . . . . . . . . . . 14 3.2 Underfitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.3 Nearest neighbour . . . . . . . . . . . . . . . . . . . . . . . . . 16

4 Lecture 4 - 07-04-2020 18 4.1 Computing h_{NN} . . . . . . . . . . . . . . . . . . . . . . . . . . 18 4.2 Tree Predictor . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

5 Lecture 5 - 07-04-2020 22 5.1 Tree Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 5.2 Jensen’s inequality . . . . . . . . . . . . . . . . . . . . . . . . 23 5.3 Tree Predictor . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 5.4 Statistical model for Machine Learning . . . . . . . . . . . . . 26

6 Lecture 6 - 07-04-2020 28

7 Lecture 7 - 07-04-2020 29

8 Lecture 8 - 07-04-2020 30

9 Lecture 9 - 07-04-2020 31 10 Lecture 10 - 07-04-2020 32

10.1 TO BE DEFINE . . . . . . . . . . . . . . . . . . . . . . . . . 32

1 Lecture 1 - 09-03-2020

1.1 Introduction

This is time for all good men to come to the aid of their party!

MACHINE LEARNING In this course we look at the principle behind design of Machine learning. Not just coding but have an idea of algorithm that can work with the data. We have to fix a mathematical framework: some statistic and mathematics. Work on ML on a higher level ML is data inference: make prediction about the future using data about the past Clustering —¿ grouping according to similarity Planning —¿ (robot to learn to interact in a certain environment) Classification —¿ (assign meaning to data) example: Spam filtering I want to predict the outcome of this individual or i want to predict whether a person click or not in a certain advertisement. Examples Classify data into categories: Medical diagnosis: data are medical records and • categories are diseases • Document analysis: data are texts and categories are topics • Image analysts: data are digital images and for categories name of objects in the image (but could be different). • Spam filtering: data are emails, categories are spam vs non spam. • Advertising prediction: data are features of web site visitors and categories could be click/non click on banners. Classification : Different from clustering since we do not have semantically classification (spam or not spam) —¿ like meaning of the image. I have a semantic label. Clustering: i want to group data with similarity function. Planning: Learning what to do next Clustering: Learn similarity function Classification: Learn semantic labels meaning of data Planning: Learn actions given state In classification is an easier than planning task since I’m able to make prediction telling what is the semantic label that goes with data points. If i can do classification i can clustering. If you do planning you probably classify (since you understanding meaning in your position) and then you can also do clustering probably. We will focus on classification because many tasks are about classification. Classify data in categories we can image a set of categories. For instance the tasks: ‘predict income of a person’ ‘Predict tomorrow price for a stock’ The label is a number and not an abstract thing. We can distinguish two cases: The label set —¿ set of possible categories for each data • point. For each of this could be finite set of abstract symbols (case of document classification, medical diagnosis). So the task is classification. • Real number (no bound on how many of them). My prediction will be a real number and is not a category. In this case we talk about a task of regression. Classification: task we want to give a label predefined point in abstract categories (like YES or NO) Regression: task we want to give label to data points but this label are numbers. When we say prediction task: used both for classification and regression tasks. Supervised learning: Label attached to data (classification, regression) Unsupervised learning: No labels attached to data (clustering) In unsupervised the mathematical modelling and way algorithm are score and can learn from mistakes is a little bit harder. Problem of clustering is harder to model mathematically. You can cast planning as supervised learning: i can show the robot which is the right action to do in that state. But that depends on planning task is formalised. Planning is higher level of learning since include task of supervised and unsupervised learning. Why is this important ? Algorithm has to know how to given the label. In ML we want to teach the algorithm to perform prediction correctly. Initially algorithm will make mistakes in classifying data. We want to tell algorithm that classification was wrong and just want to perform a score. Like giving a grade to the algorithm to understand if it did bad or really bad. So we have mistakes! Algorithm predicts and something makes a mistake —¿ we can correct it. Then algorithm can be more precisely. We have to define this mistake. Mistakes in case of classification: If category is the wrong one (in the simple case). We • have a binary signal where we know that category is wrong. How to communicate it? We can use the loss function: we can tell the algorithm whether is wrong or not. Loss function: measure discrepancy between ‘true’ label and predicted label. So we may assume that every datapoint has a true label. If we have a set of topic this is the true topic that document is talking about. It is typical in supervised learning.

How good the algorithm did?

ℓ(y, ŷ) ≤ 0

were y is true label and ŷ is predicted label

We want to build a spam filter were 0 is not spam and 1 is spam and that Classification task:

ℓ(y, ŷ = \begin{cases}0, & if ŷ = y \\ 1, & if ŷ ≠ y \end{cases}

The loss function is the “interface” between algorithm and data. So algorithm know about the data through the loss function. If we give a useless loss function the algorithm will not perform good: is important to have a good loss function. Spam filtering We have two main mistakes: It is the same mistake? No if i have important email and you classify as spam that’s bad and if you show me a spam than it’s ok. So we have to assign a different weight. Even in binary classification, mistakes are not equal. e Iotf.TFprIuos.uos True came razee Cussler aircN TASK spam ACG FIRM ftp.y GO IF F Y n is soon IF FEY 0 Nor spam ZERO CNE Cass n n Span No Seamy Binary Classification I 2 FALSE PEENE Mistake Y NON SPAM J Spam FN Mistake i f SPAM y NO spam 2 IF Fp Meter Airenita f Y F on positive y ye en MISTAKE 0 otherwise 0 otherwise

2.1 Argomento

Classification tasks Semantic label space Y Categorization Y finite and small Regression Y appartiene ad —R How to predict labels? Using the lost function → .. Binary classification Label space is Y = -1, +1 Zero-one loss

ℓ(y, ŷ = \begin{cases}0, & if ŷ = y \\ 1, & if ŷ ≠ y \end{cases}

F P ŷ = 1, y = −1 F N ŷ = −1, y = 1

Losses for regression? y, and ŷ ∈ ℝ, so they are numbers! One example of loss is the absolute loss: absolute difference between numbers

2.2 Loss

2.2.1 Absolute Loss

ℓ(y, ŷ = |y − ŷ| ⇒ absolute loss

— DISEGNO —

Some inconvenient properties:

• ...

• Derivative only two values (not much informations)

2.2.2 Square Loss

ℓ(y, ŷ = (y − ŷ)^{2} ⇒ square loss

– DISEGNO — Derivative :

• more informative

• and differentible

Real numbers as label → regression. Whenever taking difference between two prediction make sense (value are numbers) then we are talking about regression problem. Classification as categorization when we have small finite set.

2.2.3 Example of information of square loss

ℓ(y, ŷ) = (y − ŷ)^{2} = F (y) F′(ŷ) = −2 · (y − ŷ)

• I’m under sho or over and how much

• How much far away from the truth

ℓ(y, ŷ) = |y − ŷ| = F (y′) · F′(y) = Sign(y − ŷ)

Question about the future Will it rain tomorrow? We have a label and this is a binary classification problem. My label space will be Y = “rain”, “no rain” We don’t get a binary prediction, we need another space called prediction space (or decision space). Z = [0, 1] ŷ ∈ Z ŷ is my prediction of rain tomorrow ŷ = ℙ(y = ”rain”) → my guess is tomorrow will rain (not sure)

y ∈ Y  ŷ ∈ Z

quadHow can we manage loss?

Put numbers in our space {1, 0} where 1 is rain and 0 no rain

I measure how much I’m far from reality. So loss behave like this and the punishment is gonna go linearly??

26..

However is pretty annoying. Sometime I prefer to punish more so i going quadratically instead of linearly. There are other way to punish this. I called logarithmic loss We are extending a lot the range of our loss function.

ℓ(y, ŷ) = |y − ŷ| ∈ |0, 1| ℓ(y, ŷ) = (y − ŷ)^{2} ∈ |0, 1|

If i want to expand the punishment i use logarithmic loss

ℓ(y, ŷ = \begin{cases}ln\frac{1}{ŷ}, & if y = 1(rain) \\ ln\frac{1}{1−ŷ}, & if y = 0(no rain \end{cases}

F (ŷ) → can be 0 if i predict with certainty If ŷ = 0.5 ℓ(y, \frac{1}{2}) = ln2 constant losses in each prediction

lim_{ŷ→0^{+}} ℓ(1, ŷ) = +∞

We give a vanishing probability not rain but tomorrow will rain. So this is + ∞ lim_{ŷ→1^{−}} ℓ(0, ŷ) = +∞

The algorithm will be punish high more the prediction is not real. Algorithm will not get 0 and 1 because for example is impossible to get a perfect prediction. This loss is useful to give this information to the algorithm.

Now we talk about labels and losses 2.2.4 labels and losses

Data points: they have some semantic labels that denote some true about this data points and we want to predict this labels. We need to define what data points are: number? Strings? File? Typically they are stored in database records They can have very precise structure or more homogeneously structured A data point can be viewed as a vector in some d dimensional real space. So it’s a vector of number

ℝ^{d}X = (x_{1}, x_{2}..., x_{d}) ∈ ℝ^{c}

Image can be viewed as a vector of pixel values (grey scale 0-255). I can use geometry to learn because point are in my Euclidean space. Data can be represented as point in Euclidean space. Images are list of pixel that are pretty much the same range and structure (from 0 to 255). It’s very natural to put them in a space.

Assume X can be a record with heterogeneous fields: For example medical records, we have several values and each fields has his meaning by it’s own. (Sex, weight, height, age, zip code) Each one has a different range, in some cases is numerical but something have like age .. Does have any sense to see a medical record as a point since coordinates have different meaning. Fields are not comparable. This is something that you do: when you want to solve some inference you have to decide which are the label and what is the label space and we have to encode the data points.

Data algorithm expect some homogenous interface. In this case algorithm has to build records with different values of fields. This is something that we have to pay attention too. You can always each range of values in number. So ages is number, sex you can give 0 and 1, weight number and zip code is number. How ever geometry doesn’t make sense since I cannot compare this coordinates. Linear space i can sum up as vector: i can make linear combination of vectors. Inner product to measure angles! (We will see in linear classifier).

I can scramble the number of my zip code. So we get problems with sex and zip code

Why do we care about geometry? I can use geometry to learn. However there is more to that, geometry will carry some semantically information that I’m going to preserve during prediction. I want to encode my images as vectors in a space. Images with dog.....

PCA doesn’t work because assume we encode in linear space. We hope geometry will help us to predict label correctly and sometimes i hard to convert data into geometry point. Example of comparable data: images, or documents. Assume we have documents with corpus (set of documents). Maybe in English and talk about different thing and different words. X is a document and i want to encode X into a point fix in bidimensional space. There is a way to encode a set of documents in point in a fixed dimensional space in such way it make sense this coordinate are comparable. I can represent fields with [0,1] for Neural network for example. But they have no geometrical meaning

2.2.5 Example TF(idf) documents encoding

TF encoding of docs.

1. Extract where all the words from docs

2. Normalize words (nouns, adjectives, verbs ...)

3. Build a dictionary of normalized words

Doc x = (x_{1}, .., x_{d}) I associate a coordinate for each word in a dictionary. d = number of words in dictionary I can decide that x_{i} = 1 If i-th word of dictionary occurs in doc. x_{i} = 0 Else

X_{i} number of time i-th word occur in doc. Longer documents will have higher value of coordinates that are not zero. Now i can do the TF encoding in which xi = frequency with which i-th word occur in dictionary. You cannot sum dog and cat but we are considering them frequencies so we are summing frequency of words. This encoding works well in real words. I can choose different way of encoding my data and sometime i can encode a real vector

I want

1. A predictor f : X ⟶ Y (in weather X ⟶ Z

2. X is our data space (where points live)

3. X = ℝ^{d} images

4. X = X_{1}x...xX_{d} Medical record

5. ŷ = f(x) predictor for X

(x, y)

We want to predict a label that is much closer to our label. How? Loss function: so this is my setting and is called and example. Data point together with label is a “example” We can get collection of example making measurements or asking people. So we can always recover the true label. We want to replace this process with a predictor (so we don’t have to bored a person). y is the ground truth for x → mean reality! If i want to predict stock for tomorrow, i will wait tomorrow to see the ground truth.

Data point x represented as sequences of measurement and we called this measurements features or attributes.

x = (x_{1}, ..., x_{d}) x_{1} feature valuex ∈ X^{d} X = ℝ^{d} X = X_{1}·x·...·X_{d}·x

Label space Y Predictor f : X → Y

Example (x, y) y is the label associated with x (→ y is the correct label, the ground truth)

Learning with example (x_{1}, y_{1})...(x_{m}, y_{m}) training set

Training set is a set of examples with every algorithm can learn.......

Learning algorithm take training set as input and produces a predictor as output.

......DISEGNO

With image recognition we use as measurement pixels. How do we measure the power of a predictor? A learning algorithm will look at training set, algorithm and generate the predictor. Now the problem is verify the score. Now we can consider a test set collection of example

Test set (x′_{1}, y′_{1})...(x′_{n}, y′_{n})

Typically we collect big dataset and then we split in training set and test set randomly. Training and test are typically disjoint How we measure the score of a predictor? We compute the average loss. The error is the average loss in the element in the test set.

Test error \frac{1}{n} · ∑_{t=1}^{n} ℓ(f(x′_{t}), y′)

In order to simulate we collect the test set and take the average loss of the predictor of the test set. This will give us idea of how the.. Proportion of test and train depends in how big the dataset is in general. Our Goal: A learning algorithm ‘A’ must output f with a small test error. A does not have access to the test set. (Test set is not part of input of A). Now we can think in general on how a learning algorithm should be design. We have a training set so algorithm can say: ‘A’ may choose f based on performance on training set.

Training error  ℓ̂(f) = \frac{1}{m} · ∑_{t=1}^{m} ℓ(f(x_{t}), y_{t})

Given the training set (x_{1}, ..., x_{m})(y_{1}, ..., y_{m}) If ℓ̂(f) for same f, then test of f is also small Fix F set of predictors output f̂

f̂ = arg min ℓ̂(f)f ∈ F

This algorithm is called Empirical Risk Minimiser (ERM) When this strategy (ERM) fails? ERM may fails if for the given training set there are: Many f ∈ F with small ℓ̂(f), but not all of them have small test error

There could be many predictor with small error but some of them may have big test error. Predictor with the smallest training error doesn’t mean we will have the smallest test error. I would like to pick f^{∗} such that:

f^{∗} = arg min\frac{1}{n} · ∑_{t=1}^{m} ℓ(f(x′_{t}), y_{t}) f ∈ F

where ℓ(f(x′_{t}), y_{t}) is the test error ERM works if f^{∗}such that f^{∗} = arg min ℓ̂(f) f ∈ F So minimising training and test????? Check videolecture We can think of f as finite since we are working on a finite computer. We want to see why this can happen and we want to formalise a model in which we can avoid this to happen by design: We want when we run ERM choosing a good predictor with ...... PD

3.1 Overfitting

We called this as overfitting: specific situation in which ‘A’ (where A is the learning algorithm) overfits if f output by A tends to have a training error much smaller than the test error. A is not doing his job (outputting large test error) this happen because test error is misleading. Minimising training error doesn’t mean minimising test error. Overfitting is bad. Why this happens? This happen because we have noise in the data

3.1.1 Noise in the data

Noise in the data: y_{t} is not deterministically associated with x_{i}.

Could be that datapoint appears more times in the same test set. Same datapoint is repeated actually I’m mislead since training and dataset not coincide. Minimising the training error can take me away from the point that minimise the test error. Why this is the case?

• Some human in the loop: label assigned by people.(Like image contains certain object but human are not objective and people may have different opinion)

• Lack of information: in weather prediction i want to predict weather error. Weather is determined by a large complicated system. If i have humidity today is difficult to say for sure that tomorrow will rain.

When data are not noise i should be ok. Labels are not noisy Fix test set and trainign set.

∃f^{∗} ∈ F y′_{t} = f^{∗}(x′_{t}) ∀(x′_{t}, y′_{t}) in test set
y_{t} = f^{+}(x_{t}) ∀(x_{t}, y_{t}) in training set

Think a problem in which we have 5 data points(vectors) : x⃗_{1}, ...x⃗_{5} in some space X We have a binary classification problem Y = {0, 1} {x⃗_{1}, ..., x⃗_{5}} ∈ X Y = {0, 1}

F contains all possible calssifier 2^{5} = 32 f : {x_{1}, ..., x_{5}} → {0, 1}

Example

x_{1}  x_{2}  x_{3}  x_{4}  x_{5}
f  0  0  0  0  0
f′  0  0  0  0  1
f^{”}  ..  ..  ..  ..  ..
Training set x_{1}, x_{2}, x_{3} f^{+}
Test set x_{4}, x_{5} f^{∗}

4 classifier f ∈ F will have ℓ̂(f) = 0

(x_{1}, 0) (x_{2}, 1) (x_{3}, 0) (x_{4},?) (x_{5},?) f^{∗}(x_{4}) f^{∗}(x_{5}) If not noise i will have deterministic data but in this example (worst case) we get problem. I have 32 classifier to choose: i need a larger training set since i can’t distinguish predictor with small and larger training(?) error. So overfitting noisy or can happen with no noisy but few point in the dataset to define which predictor is good.

3.2 Underfitting

‘A’ underfits when f output by A has training error close to test error but they are both large. Close error test and training error is good but the are both large.

A ≡ ERM, then A undefits if F is too small → not containing too much predictors

In general, given a certain training set size:

• Overfitting when |F | is too large (not enough points in training set)

• Underfitting when |F | is too small

Proportion predictors and training set

|F |, i need ln|F | bits of info to uniquely determine f^{∗} ∈ F

m >> ln|F | when |F | < ∞where m is the size of traning set

3.3 Nearest neighbour

This is completely different from ERM and is one of the first learning algorithm. This exploit the geometry of the data. Assume that our data space X is: X ≡ ℝ^{d} x = (x_{1}, ..., x_{d}) y − {−1, 1} S is the traning set (x_{1}, y_{1})...(x_{m}, y_{m}) x_{t} ∈ ℝ^{d} y_{t} ∈ {−1, 1}

d = 2 → 2-dimensional vector

....– DISEGNO –... where + and - are labels

Point of test set If i want to predict this point? Maybe if point is close to point with label i know then. Maybe they have the same label. ŷ = + or ŷ = −

.....– DISEGNO – ... I can came up with some sort of classifier.

Given S training set, i can define h_{NN} X → {−1, 1} h_{NN}(x) = label y_{t} of the point x_{t} in S closest to X (the breaking rule for ties) For the closest we mean euclidian distance X = ℝ^{d}

∥x − x_{t}∥ =\sqrt{∑_{e=1}^{d}(x_{e} − x_{t}, e)^{2}}

ℓ̂(h_{NN}) = 0
h_{NN}(x_{t}) = y_{t}

training error is 0!

We spoke about Knn classifier with voronoi diagram

ℓ̂(h_{NN}) = 0 ∀ Traning set

h_{NN} predictor needs to store entire dataset.

4.1 Computing h_{NN}

Computing h_{NN}(x) requires computing distances between x and points in the traning set.

Θ(d) time for each distance

NN → 1-NN We can generalise NN in K-NN with k = 1, 3, 5, 7 so odd K h_{k−NN}(x) = label corresponding to the majority of labels of the k closet point to x in the training set.

How big could K be if i have n point? I look at the k closest point When k = m? The majority, will be a constant classifier h_{k−NN} is constant and corresponds to the majority of training labels Training error is always 0 for h_{NN}, while for h_{k−NN} will be typically > 0, with k > 1 Image: one dimensional classifier and training set is repeated. Is the plot of 1-NN classifier. Positive and negative. K = 1 error is 0. In the second line we switch to k = 3. Second point doesn’t switch and third will be classify to positive and we have training mistake. Switches corresponds to border of voronoi partition.

K_{NN} For multiclass classification

(|Y | > 2) for regression Y ≡ ℝ Average of labels of K neighbours → i will get a number with prediction. I can weight average by distance You can vary this algorithm as you want.

Let’s go back to Binary classification. The k parameter is the effect of making the structure of classifier more complex and less complex for small value of k.

–.. DISEGNO ..– Fix training set and test set Accury as oppose to the error

Show a plot. Training error is 0 at k = 0. As i go further training error is higher and test error goes down. At some point after which training and set met and then after that training and test error goes up (accuracy goes down). If i run algorithm is going to be overfitting: training error and test error is high and also underfitting since testing and training are close and both high. Trade off point is the point in x = 23 (more or less). There are some heuristic to run NN algorithm without value of k.

History

• K_{NN}: from 1960 → X ≡ ℝ^{d}

• Tree predictor: from 1980

4.2 Tree Predictor

If a give you data not welled defined in a Euclidean space. X = X_{1} · x · ... · X_{d} · x Medical Record X_{1} = {Male, F emale} X_{2} = {Y es, No} so we have different data

I want to avoid comparing x_{i} with x_{j}, i ≠ j so comparing different feature and we want to compare each feature with each self. I don’t want to mix them up. We can use a tree! I have 3 features:

• outlook = {sunny, overcast, rain}

• humidity = {[0, 100]}

• windy = {yes, no}

... – DISEGNO – ...

Tree is a natural way of doing decision and abstraction of decision process of one person. It is a good way to deal with categorical variables. What kind of tree we are talking about? Tree has inner node and leaves. Leaves are associated with labels (Y) and inner nodes are associated with test.

• Inner node → test

• Leaves → label in Y

Test if a function f (NOT A PREDICTOR!) Test f_{i} X_{i} → {1, ..., k} where k is the number of children (inner node) to which test is assigned In a tree predictor we have:

• Root node

• Children are ordered(i know the order of each branch that come out from the node)

X = {Sunny, 50%, No} → are the parameters for {outlook.humidity, windy}

f_{i} = \begin{cases}1, & if x_{2} ∈ [30%, 60%] \\ 2, & if otherwise \end{cases}

where the numbers 1 and 2 are the children A test is partitioning the range of values of a certain attribute in a number of elements equal to number of children of of the node to which the test is assigned. h_{T} (x) is always the label of a leaf of T This leaf is the leaf to which x is routed Data space for this problem (outlook,..) is partitioned in the leaves of the tree. It won’t be like voronoi graph. How do I build a tree given a training set? How do i learn a tree predictor given a training set?

• Decide tree structure (how • many node, leaves ecc..)

• Decide test on inner nodes

• Decide labels on leaves

We have to do this all together and process will be more dynamic. For simplicity binary classification and fix two children for each inner node.

Y = {−1, +1} 2 children for each inner node

What’s the simplest way? Initial tree and correspond to a costant classifier

– DISEGNO –

Majority of all example

– DISEGNO –

(x_{1}, y_{1})...(x_{m}, y_{m}) x_{t} ∈ X y_{t} ∈ {−1, +1} Training set S = {(x, y) ∈ S, x is routed to ℓ} S_{ℓ}^{+}

– DISEGNO –

S_{ℓ} and S′_{ℓ} are given by the result of the test, not the labels and ℓ and ℓ′.

5.1 Tree Classifier

Supposed we groped a tree up to this point and we are wandering how to grow it. S Training set (x_{1}, y_{1})...(x_{m}, y_{m}), x_{1} ∈ X

– DISEGNO

S_{ℓ} ≡ {(x_{1}, y_{1}) x_{t} is router to ℓ}

y_{1} ∈ {−1, 1}

S_{ℓ}^{+} ≡ {(x_{1}, y_{1}) ∈ S_{ℓ} : y_{t} = +1}

S_{ℓ}^{−} ≡ {(x_{1}, y_{1}) ∈ S_{ℓ} : y_{t} = −1} S_{ℓ}^{+} ∩ S_{ℓ}^{−} ≡ 0 S_{ℓ} ≡ S_{ℓ}^{+} ∪ S_{ℓ}^{−}

N_{ℓ} = |S_{ℓ}| N_{ℓ}^{+} = |S_{ℓ}^{+}| N_{ℓ}^{−} = |S_{ℓ}^{−}|
N_{ℓ} = N_{ℓ}^{−} + N_{ℓ}^{+}

leaf ℓ classifies all traning example (S_{ℓ})

Y_{ℓ} = \begin{cases}+1, & If N_{ℓ}^{+} ≥ N_{ℓ}^{−} \\ −1, & If otherwise \end{cases}

ℓ makes a mistake on min{N_{ℓ}^{+}, N_{ℓ}^{−}} example in S_{ℓ}

ℓ̂(h_{T}) = \frac{1}{m} · ∑_{ℓ} min\{\frac{N_{ℓ}^{+}}{N_{ℓ}} , \frac{N_{ℓ}^{−}}{N_{ℓ}} \} · N_{ℓ} =

= \frac{1}{m} · ∑_{ℓ} ψ · (\frac{N_{ℓ}+}{N_{ℓ}}) · N_{ℓ} ⟶ \frac{N_{ℓ}^{+}}{N_{ℓ}} = 1 − \frac{N_{ℓ}}{N_{ℓ}??}

where ψ(a) = min{a, 1 − a} a ∈ [0, 1] I want to replace inner node with other leaves.

– DISEGNO – How is traning error going to change? (when i replace inner nodes with other leaves) I’m hoping my algorithm is not going to overfit (if training error goes to 0 also testing error goes to 0).

5.2 Jensen’s inequality

If ψ is a concave function ⟶ (like log or \sqrt[2]{..} ) Also ψ is a function that map 0 to 1, ⟶ ψ [0, 1] → ℝ

ψ(α·a+(1−α)·b) ≥ α·ψ(a)+(1−α)·ψ(b) Also 2° derivative is negative

– DISEGNO –

ℓ̂(h_{T}) = \frac{1}{m} · ∑_{ℓ} ψ(\frac{N_{ℓ}^{+}}{N_{ℓ}}) · N_{ℓ}

Look a single contribution fo a leaf ℓ to training error

ψ(\frac{N_{ℓ}^{+}}{N_{ℓ}}) · N_{ℓ} = ψ(\frac{N_{ℓ}^{′+}}{N′_{ℓ}} · \frac{N′_{ℓ}}{N_{ℓ}} + \frac{N_{ℓ}”^{+}}{N_{ℓ}”} · \frac{N_{ℓ}”}{N_{ℓ}}) · N_{ℓ}

where \frac{N′_{ℓ}}{N_{ℓ}} = α and \frac{N_{ℓ}”}{N_{ℓ}} = 1 − α  so \frac{N′_{ℓ}}{N_{ℓ}} + \frac{N_{ℓ}”}{N_{ℓ}} = 1  ⟶ α + 1 − α = 1

N_{ℓ′}^{+} + N_{ℓ”}^{+} = N_{ℓ}

I want to check function min concave between 0 and 1.

min(0, 1) = 0 ψ(a) = min(α, 1 − α)

– DISEGNO –

This is a concave function and now I can apply Jensen’s inquality

ψ(\frac{N_{ℓ}^{+}}{N_{ℓ}}) · N_{ℓ} ≥ (\frac{N′_{ℓ}}{N_{ℓ}} · ψ(\frac{N_{ℓ}^{′+}}{N′_{ℓ}}) + \frac{N_{ℓ}”}{N_{ℓ}} · ψ(\frac{N_{ℓ}”^{+}}{N_{ℓ}”} )) · N_{ℓ} =

= ψ(\frac{N_{ℓ}^{′+}}{N′_{ℓ}}) · N′_{ℓ} + ψ(\frac{N_{ℓ}”^{+}}{N_{ℓ}”} ) · N_{ℓ}”

This are the contribuion of ℓ′ and ℓ” to the training error

Every time i split my tree my training error is never going to increase since we have a concave function. Whenever I’m growing my tree training error is going to be smaller.

Every time a leaf is expanded the training error never goes up. (Hopelly will go down) I’ll should always grow the tree by expanding leave that decrease the training error as much as possible. If i take the effort of growing the tree i should get benefits. I can imaging that if i grow the tree at random my training error is going to drop down error (but maybe will derive overfitting). For now is just an intuition since we will introduced statistical learning model.

Could be complicated and tree big may have 100 leave and there could be many way of associating a test with that leaves. I can spent a lot of time to select which leave is the best promising to split.

• Grow the tree by expanding leave that decrease the training error as much as possible

• In general we can assume:

greedy algorithm at each step pick the pair leaf and test that cause (approximative) the largest decrease in training error

In practise we want optimise this all the way since it’s time expensive. That’s the approximately since we are not every time sure.

— MANCA PARTE —

— IMMAGINE — p = 0.8 q = 1 r = 1 α = 60% Net Change in number of mistakes

ψ(p) − (α · ψ(q) + (1 − α) · ψ(r)) =
ℓ −  ℓ′ +  ℓ”

Fraction of example miss classified ℓ− error ℓ′+ error ℓ”

= 0.2 − (\frac{1}{2} · 0.4 + \frac{1}{2} · 0) = 0

— DISEGNO —

Idea is to replace minimum function with convex combination.

ψ(α) = min {α, 1 − α} ψ(a) ≥ ψ(α)

\begin{cases}ψ_{1}(α) = 2 · α · (1 − α) ⟶ GNI \\ ψ_{2}(α) = −\frac{α}{2} · ln α − \frac{1−α}{2} · ln(1 − α) ⟶ ENT ROP Y \\ ψ_{3}(α) =\sqrt{α · (1 − α)} \end{cases}

All this functions has this shape (concave???)

– DISEGNO – In practise Machine Learning algorithm use GNI or entropy to control the split

5.3 Tree Predictor

• Multi class classification |Y | > 2 ⟶ take majority

• Regression Y = ℝ ⟶ take average of labels in S_{ℓ}

I still take majority among different classes. Take average of labels in S_{ℓ} Unless \frac{N_{ℓ}^{+}}{N_{ℓ}} ∈ 0, 1 ∀ leaves ℓ, ℓ̂(h_{T}) > 0 Unless leaves are ”pured”, the training error will be bigger than 0.

In general, i can always write ℓ̂(h_{t}) to 0 by growing enough the tree unless there are x_{1} in the Time Series such that (x_{t}, y_{t})(x_{t}, y′_{t}) with y_{t} ≠ y′_{t} both occur.

— DISEGNO —-

if(x_{1} = α) ∧ (x_{2} =≥ α) ∨ (x_{1} = b) ∨ (x_{1} = c) ∧ (x_{3} = y)

then predict 1

else

then predict -1

— Picture of tree classifier of iris dataset. — I’m using due attribute at the time. Each data point is a flower and i can measure how petal and sepal are long. I can use two attribute and i test this two. I can see the plot of the tree classifier (second one) making test splitting data space into region that has this sort of “blackish” shape ( like boxes: blue box, red box, yellow box) A good exercise in which I want to reconstruct the tree given this picture.

5.4 Statistical model for Machine Learning

To understand Tree classifier, nearest neighbour and other algorithm... It’s important to understand that the only way to have a guideline in which model to choose.

This mathematical model are developed to learning and choose learning algorithm.

Now let start with theoretical model.

• How example (x, y) are generated to create test set and training set? We get the dataset but we need to have a mathematical model for this process. (x, y) are drawn from a fixed but unknown probability distribution on the pairs X and Y (X data space, Y label set o label space)

• Why X should be random? In general we assumed that not all the x in X are equally likely to be observed. I have some distribution over my data point and this said that I’m most like to get a datapoint to another.

• How much label?

Often labels are not determined uniquely by their datapoints because labels are given by human that have their subjective thoughts and also natural phenomena. Labels are stochastic phenomena given a datapoint: i will have a distribution.

We’re going to write (in capital) (X, Y) since they are random variable drawn from D on X ·Y A dataset (X_{1}, Y_{1})...(X_{m}, Y_{m}) they are drawn independently from D (distribution on examples) When I get a training the abstraction of process collecting a training set D is a joint probability distribution over X · Y where D_{x} is the marginal over X → D_{y}|x (conditional of Y given X). I can divided my draw in two part. I draw sample and label from conditional.?? Any dataset ( training or test ) is a random sample (campione casuale) in the statistical sense ⟶ so we can use all stastical tools to make inference.

6 Lecture 6 - 07-04-2020

28

7 Lecture 7 - 07-04-2020

29

8 Lecture 8 - 07-04-2020

30

9 Lecture 9 - 07-04-2020

31

10 Lecture 10 - 07-04-2020

10.1 TO BE DEFINE

|E[z] = |E[|E[z|x]] |E[X] = ∑_{t=1}^{m} |E[xΠ(At)] x ∈ ℝ^{d} ℙ(Y_{Π(s,x)} = 1) = 𝔼[ΠY_{Π(s,x)} = 1] = = ∑_{t=1}^{m} 𝔼[Π{Y_{t} = 1} · ΠP i(s, x) = t] = = ∑_{t=1}^{m} 𝔼[𝔼[Π{Y_{t} = 1} · Π{Π(s, x) = t}|X_{t}]] = giventhefactthatY_{t} ∼ η(X_{t}) ⇒ givemeprobability Y_{t} = 1andΠ(s, x) = tareindependentgivenX_{Y} (e.g.𝔼[Zx] = 𝔼[x] ∗ ·𝔼[z] = ∑_{t=1}^{m} 𝔼[𝔼[Π{Y_{t} = 1}|X_{t}] · 𝔼[Π(s, x) = t|Xt]] = = ∑_{t=1}^{m} 𝔼[η(X_{t}) · Π · {Π(s, x) = t}] = = 𝔼[η(X_{Π(s,x)}]

ℙ(Y_{Π(s,x)}|X = x = 𝔼[η(X_{Π}(s, x))]

ℙ(Y_{Π(s,x)} = 1, y = −1) = = 𝔼[Π{Y_{Π(s,x)} = 1}Π̇{Y = −1|X}]] = = 𝔼[Π{Y_{Π(s,x)} = 1} · Π{y = −1}] = = 𝔼[𝔼[Π{Y_{Π(s,x)} = 1} · Π{y = −1|X}]] =

Y_{Π(s,x)} = 1 y = −1(1 − η(x)) when X = x

= 𝔼[𝔼[Π{Y_{Π}(s, x)} = 1|X] · 𝔼[Π{y = −1}|X]] = = 𝔼[η_{Π(s,x)} · (1 − η(x))] = similarly : ℙ(Y_{Π(s,x)} = −1, y = 1) = 𝔼[(1 − η_{Π(s,x)}) · η(x)] 𝔼[ℓ_{D}(ĥ_{s})] = ℙ(Y_{Π(s,x)} ≠ y) = = ℙ(Y_{Π(s,x)} = 1, y = −1) + ℙ(Y_{P i(s,x)} = −1, y = 1) = = 𝔼[η_{Π(s,x)} · (1 − eta(x))] + 𝔼[(1 − η_{Π(s,x)}) · η(x)] Make assumptions on D_{x} and η:

MANCAAAAAAA ROBAAA

η(x′) <= η(x) + c||X − x′|| − − > euclideandistance 1 − η(x′) <= 1 − η(x) + c||X − x′||

X′ = X_{P i(s,x)} η(X) · (1 − η(x′)) + (1 − η(x)) · η(x′) <= <= η(x) · ((1 − η(x)) + η(x) · c||X − x′|| + (1 − η(x)) · c||X − x′|| = = 2 · η(x) · (1 − η(x)) + c||X − x′|| 𝔼[ℓ_{d} · (ĥ_{s})] <= 2 · 𝔼[η(x) − (1 − η(x))] + c · (E)[||X − x_{Π(s,x)}||] where <= mean at most Compare risk for zero-one loss 𝔼[min{η(x), 1 − η(x)}] = ℓ_{D}(f∗) η(x) · (1 − η(X)) <= min{η(x), 1 − eta(x)} ∀x 𝔼[η(x) · (1 − η(x)] <= ℓ_{D}(f∗) 𝔼[ℓ_{d}(l̂_{s})] <= 2 · ℓ_{D}(f∗) + c · 𝔼[||X − X_{Π(s,x)}||] η(x) ∈ {0, 1} Depends on dimension: curse of dimensionality –DISEGNO– ℓ_{d}(f∗) = 0 ⟺ min{η(x), 1 − η(x)} = 0 with probability = 1 to be true η(x) ∈ {0, 1}

References

35