Lecture 6 - 07-04-2020

(X, Y) We random variables drawn iid from D on X ·Y ⟶ where D is fixed but unknown

Independence does not hold. We do not collect datapoints to an independent process. Example: identify new article and i want to put categories. The feed is highly depend on what is happening in the world and there are some news highly correlated. Why do we make an assumption that follows reality? Is very convenient in mathematical term. If you assume Independence you can make a lot of process in mathematical term in making the algorithm. If you have enough data they look independent enough. Statistical learning is not the only way of analyse algorithms —> we will see in linear ML algorithm and at the end you can use both statistical model s

1.1 Bayes Optimal Predictor

f^{∗} : X → Y
f^{∗}(x) = argmin 𝔼 [ ℓ(y, ŷ)|X = x]  ŷ ∈ Y

In general Y given X has distribution D_{y}|X = x Clearly ∀ h X → Y

𝔼 [ ℓ(y, f^{∗}(x))|X = x] ≤ 𝔼 [ ℓ(y, h(x)|X = x]
X, Y 𝔼 [ Y |X = x] = F (x) ⟶ ConditionalExpectation
𝔼 [ 𝔼 [ Y |X] ] = 𝔼(Y)

Now take Expectation for distribution

𝔼 [ ℓ(y, f^{∗}(x)) ] ≤ [ 𝔼(ℓ(y, h(x)) ]

where risk is smaller in f^{∗} I can look at the quantity before l_{d} Bayes risk ⟶ Smallest possible risk given a learning problm

l_{d}(f^{∗}) > 0 because y are still stochastic given X Learning problem can be complem → large risk

1.1.1 Square Loss

ℓ(y, ŷ = (y − ŷ)^{2}

I want to compute bayes optimal predictor ŷ, y ∈ ℝ

f^{∗}(x) = argmin 𝔼 [ (y − ŷ)^{2}|X = x ] =  ŷ ∈ ℝ

we use 𝔼 [ X + Y ] = 𝔼[X]+𝔼[Y] = argmin 𝔼 [ y^{2} + ŷ^{2} − 2 · y · ŷ^{2}|X = x ] =

Dropping y^{2} i remove something that is not important for ŷ

= argmin(𝔼 [ y^{2}|X = x ] + ŷ^{2} − 2 · ŷ · 𝔼 [ y|X = x]) =
= argmin(ŷ^{2} − 2 · ŷ · 𝔼 [ y|X = x]) =

Expectation is a number, so it’s a constant Assume ⊡ = y^{2}

argmin [ ⊡ + ŷ^{2} + 2 · ŷ · 𝔼 [ Y |X = x]]

where redG(ŷ) is equal to the part between [...]

\frac{dG(ŷ)}{dŷ} = 2 · ŷ − 2 · 𝔼 [ y|X = x] = 0 ⟶ So setting derivative to 0

— DISEGNO OPT CURVE —

G′(ŷ) = ŷ^{2} − 2 · b · ŷ

ŷ = 𝔼 [ y|X = x] f^{∗}(x) = 𝔼 [ y|X = x]

Square loss is nice because expected prediction is ... In order to predict the best possibile we have to estimate the value given data point.

𝔼 [ (y − f^{∗}(x))^{2}|X = x ] =

= 𝔼 [ (y − 𝔼 [ y|X = x])^{2}|X = x ] = V ar [ Y |X = x]

1.1.2 Zero-one loss for binary classification

Y = {−1, 1}

ℓ(y, ŷ) = I\{ŷ ≠ y\} I_{A}(x) = \begin{cases}1 x ∈ A \\ 0 x ∉ A \end{cases}

If ŷ ≠ y true, indicator function will give us 1, otherwise it will give 0

D on X · Y D_{x}^{∗} D_{y|x} = D

D_{x} η : X ⟶ [ 0, 1 ] η = ℙ(y = 1|X = x)