Statistical Methods for Machine Learning

Andrea Ierardi Data Science and Economcis Università degli Studi di Milano

April 11, 2020

Abstract

This is the paper’s abstract …

1 Lecture 1 - 09-03-2020

1.1 Introduction

This is time for all good men to come to the aid of their party!

MACHINE LEARNING In this course we look at the principle behind design of Machine learning. Not just coding but have an idea of algorithm that can work with the data. We have to fix a mathematical framework: some statistic and mathematics. Work on ML on a higher level ML is data inference: make prediction about the future using data about the past Clustering —¿ grouping according to similarity Planning —¿ (robot to learn to interact in a certain environment) Classification —¿ (assign meaning to data) example: Spam filtering I want to predict the outcome of this individual or i want to predict whether a person click or not in a certain advertisement. Examples Classify data into categories: Medical diagnosis: data are medical records and • categories are diseases • Document analysis: data are texts and categories are topics • Image analysts: data are digital images and for categories name of objects in the image (but could be different). • Spam filtering: data are emails, categories are spam vs non spam. • Advertising prediction: data are features of web site visitors and categories could be click/non click on banners. Classification : Different from clustering since we do not have semantically classification (spam or not spam) —¿ like meaning of the image. I have a semantic label. Clustering: i want to group data with similarity function. Planning: Learning what to do next Clustering: Learn similarity function Classification: Learn semantic labels meaning of data Planning: Learn actions given state In classification is an easier than planning task since I’m able to make prediction telling what is the semantic label that goes with data points. If i can do classification i can clustering. If you do planning you probably classify (since you understanding meaning in your position) and then you can also do clustering probably. We will focus on classification because many tasks are about classification. Classify data in categories we can image a set of categories. For instance the tasks: ‘predict income of a person’ ‘Predict tomorrow price for a stock’ The label is a number and not an abstract thing. We can distinguish two cases: The label set —¿ set of possible categories for each data • point. For each of this could be finite set of abstract symbols (case of document classification, medical diagnosis). So the task is classification. • Real number (no bound on how many of them). My prediction will be a real number and is not a category. In this case we talk about a task of regression. Classification: task we want to give a label predefined point in abstract categories (like YES or NO) Regression: task we want to give label to data points but this label are numbers. When we say prediction task: used both for classification and regression tasks. Supervised learning: Label attached to data (classification, regression) Unsupervised learning: No labels attached to data (clustering) In unsupervised the mathematical modelling and way algorithm are score and can learn from mistakes is a little bit harder. Problem of clustering is harder to model mathematically. You can cast planning as supervised learning: i can show the robot which is the right action to do in that state. But that depends on planning task is formalised. Planning is higher level of learning since include task of supervised and unsupervised learning. Why is this important ? Algorithm has to know how to given the label. In ML we want to teach the algorithm to perform prediction correctly. Initially algorithm will make mistakes in classifying data. We want to tell algorithm that classification was wrong and just want to perform a score. Like giving a grade to the algorithm to understand if it did bad or really bad. So we have mistakes! Algorithm predicts and something makes a mistake —¿ we can correct it. Then algorithm can be more precisely. We have to define this mistake. Mistakes in case of classification: If category is the wrong one (in the simple case). We • have a binary signal where we know that category is wrong. How to communicate it? We can use the loss function: we can tell the algorithm whether is wrong or not. Loss function: measure discrepancy between ‘true’ label and predicted label. So we may assume that every datapoint has a true label. If we have a set of topic this is the true topic that document is talking about. It is typical in supervised learning.

How good the algorithm did?

l(y, ŷ) ≤ 0

were y is true label and ŷ is predicted label

We want to build a spam filter were 0 is not spam and 1 is spam and that Classification task:

f(n) = \begin{cases}0, & if ŷ = y \\ 1, & if ŷ ≠ y \end{cases}

The loss function is the “interface” between algorithm and data. So algorithm know about the data through the loss function. If we give a useless loss function the algorithm will not perform good: is important to have a good loss function. Spam filtering We have two main mistakes: It is the same mistake? No if i have important email and you classify as spam that’s bad and if you show me a spam than it’s ok. So we have to assign a different weight. Even in binary classification, mistakes are not equal. e Iotf.TFprIuos.uos True came razee Cussler aircN TASK spam ACG FIRM ftp.y GO IF F Y n is soon IF FEY 0 Nor spam ZERO CNE Cass n n Span No Seamy Binary Classification I 2 FALSE PEENE Mistake Y NON SPAM J Spam FN Mistake i f SPAM y NO spam 2 IF Fp Meter Airenita f Y F on positive y ye en MISTAKE 0 otherwise

Outline The remainder of this article is organized as follows. Section 3 gives account of previous work. Our new and exciting results are described in Section 4. Finally, Section 5 gives the conclusions.

2 Lecture 10 - 07-04-2020

2.1 TO BE DEFINE

|E[z] = |E[|E[z|x]] |E[X] = ∑_{t=1}^{m} |E[xΠ(At)] x ∈ ℝ^{d} ℙ(Y_{Π(s,x)} = 1) = 𝔼[ΠY_{Π(s,x)} = 1] = = ∑_{t=1}^{m} 𝔼[Π{Y_{t} = 1} · ΠP i(s, x) = t] = = ∑_{t=1}^{m} 𝔼[𝔼[Π{Y_{t} = 1} · Π{Π(s, x) = t}|X_{t}]] = giventhefactthatY_{t} ∼ η(X_{t}) ⇒ givemeprobability Y_{t} = 1andΠ(s, x) = tareindependentgivenX_{Y} (e.g.𝔼[Zx] = 𝔼[x] ∗ ·𝔼[z] = ∑_{t=1}^{m} 𝔼[𝔼[Π{Y_{t} = 1}|X_{t}] · 𝔼[Π(s, x) = t|Xt]] = = ∑_{t=1}^{m} 𝔼[η(X_{t}) · Π · {Π(s, x) = t}] = = 𝔼[η(X_{Π(s,x)}]

ℙ(Y_{Π(s,x)}|X = x = 𝔼[η(X_{Π}(s, x))]

ℙ(Y_{Π(s,x)} = 1, y = −1) = = 𝔼[Π{Y_{Π(s,x)} = 1}Π̇{Y = −1|X}]] = = 𝔼[Π{Y_{Π(s,x)} = 1} · Π{y = −1}] = = 𝔼[𝔼[Π{Y_{Π(s,x)} = 1} · Π{y = −1|X}]] =

Y_{Π(s,x)} = 1 y = −1(1 − η(x)) when X = x

= 𝔼[𝔼[Π{Y_{Π}(s, x)} = 1|X] · 𝔼[Π{y = −1}|X]] = = 𝔼[η_{Π(s,x)} · (1 − η(x))] = similarly : ℙ(Y_{Π(s,x)} = −1, y = 1) = 𝔼[(1 − η_{Π(s,x)}) · η(x)] 𝔼[ℓ_{D}(ĥ_{s})] = ℙ(Y_{Π(s,x)} ≠ y) = = ℙ(Y_{Π(s,x)} = 1, y = −1) + ℙ(Y_{P i(s,x)} = −1, y = 1) = = 𝔼[η_{Π(s,x)} · (1 − eta(x))] + 𝔼[(1 − η_{Π(s,x)}) · η(x)] Make assumptions on D_{x} and η:

MANCAAAAAAA ROBAAA

η(x′) <= η(x) + c||X − x′|| − − > euclideandistance 1 − η(x′) <= 1 − η(x) + c||X − x′||

X′ = X_{P i(s,x)} η(X) · (1 − η(x′)) + (1 − η(x)) · η(x′) <= <= η(x) · ((1 − η(x)) + η(x) · c||X − x′|| + (1 − η(x)) · c||X − x′|| = = 2 · η(x) · (1 − η(x)) + c||X − x′|| 𝔼[ℓ_{d} · (ĥ_{s})] <= 2 · 𝔼[η(x) − (1 − η(x))] + c · (E)[||X − x_{Π(s,x)}||] where <= mean at most Compare risk for zero-one loss 𝔼[min{η(x), 1 − η(x)}] = ℓ_{D}(f∗) η(x) · (1 − η(X)) <= min{η(x), 1 − eta(x)} ∀x 𝔼[η(x) · (1 − η(x)] <= ℓ_{D}(f∗) 𝔼[ℓ_{d}(l̂_{s})] <= 2 · ℓ_{D}(f∗) + c · 𝔼[||X − X_{Π(s,x)}||] η(x) ∈ {0, 1} Depends on dimension: curse of dimensionality –DISEGNO– ℓ_{d}(f∗) = 0 ⟺ min{η(x), 1 − η(x)} = 0 with probability = 1 to be true η(x) ∈ {0, 1}

3 Previous work

A much longer L^{A}TEX 2ε example was written by Gil [?].

4 Results

In this section we describe the results.

5 Conclusions

We worked hard, and achieved very little.