Lecture 14 - 28-04-2020

1.1 Linear Regression

Yesterday we look at the problem of emprical risk minismisation for a linear classifier. 0-1 loss is not good: discontinuous jumping from 0 to 1 and it’s dififcult to optimise. Maybe with linear regression we are luckier. Our data point are the form (x, y) x ∈ ℝ^{d} regression, (ŷ − y)^{2} square loss. We are able to pick a much nicer function and we can optimise it in a easier way.

1.1.1 The problem of linear regression

Instead of picking -1 or 1 we just leave it as it is.

h(c) = w^{T} x w ∈ ℝ^{d} x = (x_{1}, ..., x_{d}, 1)

ŵ = arg min_{w∈ℝ^{d}} \frac{1}{m} ∑_{t=1}^{m}(w^{T} x_{t} − y_{t})^{2} ERM for (x_{1}, y_{1})...(x_{m}, y_{m})

How to compute the minimum? We use the vector v of linear prediction v = (w^{T} x_{1}, .., w^{T} x_{m}) and a vector of real valued labels y = (y_{1}, ..., y_{m}) where v, y ∈ ℝ^{m}

∑_{t=1}^{m}(w^{T} x_{t} − y_{t})^{2} = ∥v − y∥^{2}

S is a matrix.

s^{T} = [x_{1}, ..., x_{m}] d × m v = sw = [\begin{matrix}x_{1}^{t} \\ ... \\ x_{m}^{T} \end{matrix}] [w]

So:

∥v − y∥^{2} = ∥sw − y∥^{2}
ŵ = arg min_{w∈ℝ^{d}} ∥sw − y∥^{2} where sw is the design matrix
F (w) = ∥sw − y∥^{2} is convex

∇F (w) = 2̸s^{T} (sw − y) = 0 s^{T} sw = s^{T} y

where s^{T} is d × m and s is m × d and d ≠ m If s^{T} s invertible (non singular) ŵ = (s^{T} s)^{−1} s^{T} y And this is called Least square solutions (OLS)

We can check s^{T} s is non-singular if x_{1}, ..., x_{m} span ℝ^{d}

s^{T} · s may not be always invertible. Also Linear regression is high bias solution. ERM may underfit since linear predictor introduce big bias. ŵ = (s^{T} · s)^{−1} · s^{T} · y is very instable: can change a lot when the the dataset is perturbed. This fact is called instability : variance error It is a good model to see what happens and then try more sofisticated model. Whenever ŵ is invertible we have to prove the instability. But there is a easy fix!

1.1.2 Ridge regression

We want to stabilised our solution. If s^{T} · s non-singular is a problem.

We are gonna change and say something like this:

ŵ = arg min_{w} ∥s · w − y∥^{2} ⇝ ŵ_{α} = arg min_{w} (∥s w − y∥^{2} + α · ∥w∥^{2})

where α is the regularisation term.

ŵ_{α} → ŵ for α → 0 ŵ_{α} → (0, ..., 0) for α → ∞

ŵ_{α} has more bias than ŵ, but also less variance

∇ (∥s w − y∥^{2} + α ∥w∥^{2}) = 2 (s^{T} s w − s^{T} y) + 2 α w = 0
(s^{T} s + α I) w = s^{T} y
(d × m) (m × d) (d × d) (d × m) (d × m) (m × 1)

where I is the identity

ŵ_{α} = (s^{T} s + α I)^{−1} s^{T} y

Figure 1.1:

where y_{1}, ..., y_{α} are eigen-values of s^{T} s y_{1}, ..., y_{α} + α > 0 eigenvalues of s^{T} s + αI In this way we make it positive and semidefinite. We can always compute the inverse and it is a more stable solution and stable means do not overfit.

1.2 Percetron

Now we want to talk about algorithms. Data here are processed in a sequential fashion one by one. Each datapoint is processed in costant time Θ (d) (check y_{t} w^{T} ≤ 0 and in case w ← w + y_{t} x_{t}) and the linear model can be stored in Θ(d) space. Sequential processing scales well with the number of datapoints. But also is good at dealing with scenarios where new data are generated at all times. Several scenario like:

• Sensor data

• Finantial data

• Logs of user

So sequential learning is good when we have lot of data and scenario in which data comes in fits like sensor. We call it Online learning

1.2.1 Online Learning

It is a learning protocol and we can think of it like Batch learning. We have a class H of predictors and a loss function ℓ and we have and algorith that outputs an initial default predictor h_{1} ∈ H.

For t = 1, 2...

1) Next example (x_{t}, y_{t}) is observed

2) The loss ℓ(h_{t}(x_{t}), y_{t}) is observed (y_{t} w^{T} x_{t} ≤ 0)

3) The algorithm updates h_{t} generating h_{t+1} (w ← w + y_{t} x_{t})

The algorithm generates s sequence h_{1}, h_{2}, ... of models It could be that h_{t+1} = h_{t} occasionally The update h_{t} → h_{t+1} is local (it only uses h_{t} and (x_{t}, y_{t})) This is a batch example in which take the training set and generate a new example.

(x_{1}, y_{1}) → A → h_{2}
(x_{1}, y_{1})(x_{2}, y_{2}) → A → h_{3}

But if I have a non-learning algorithm i can look at the updates:

Figure 1.2:

This is a most efficient way and can be done in a costant time. The batch learning usually have single predictor while the online learning uses a sequence of predictors.

How do I evaluate an online learning algorithm A? I cannot use a single model, instead we use a method called Sequential Risk. Suppose that I have h_{1}, h_{2}... on some data sequence.

\frac{1}{m} ∑_{t=1}^{T} ℓ(h_{t}(x), y_{t}) as a function of T

The loss on the next incoming example. 5 I would like something like this:

Figure 1.3:

We need to fix the sequence of data: I absorb the example into the loss of the predictor.

ℓ(h_{t}(x), y_{t}) ⟶ ℓ_{t}(h_{t})

I can write the sequential risk of the algorithm:

\frac{1}{m} ∑_{t=1}^{T} ℓ_{t}(h_{t}) − min_{h∈H} \frac{1}{m} ∑_{t=1}^{T} ℓ_{t}(h)

So the sequencial risk of the algorithm - the sequential risk of best predictor in H (up to T). This is a sequential similar of variance error. ⟶ is called Regret.

h_{T}^{∗} = arg min_{h∈H} \frac{1}{T} ∑_{t} ℓ_{t}(h) \frac{1}{T} ℓ_{t}(h_{t}) − \frac{1}{T} ∑_{t} ℓ_{t}(h_{T}^{∗})

1.2.2 Online Gradiant Descent (OGD)

It is an example of learning algorithm. In optimisation we have one dimension and we want to minimise the function i can compute the gradiant in every point. We start from a point and get the derivative: as I get the derivative I can see if is decreasing or increasing.

Figure 1.4: 7