Lecture 13 - 27-04-2020

1.1 Linear prediction

We had ERM ĥ

S = {(x_{1}, y_{1})...(x_{n}, y_{n})} x_{t} ∈ ℝ^{d} y_{t} ∈ {−1, +1} ℓ_{t}(w) = I{y_{t}w^{T} x_{t} ≤ 0}

ĥ_{S} = arg min_{h∈H_{D}} \frac{1}{m} ∑_{t=1}^{m} I\{y_{t}w^{T} x_{t}\} ≤ 0

The associated decisio problem is a NP problem so cannot be camputed efficientiy unless P ≡ NP Maybe we can approximate it, so a good solution that goes close to minimise error. This is called MinDisOpt

1.1.1 MinDisOpt

Instance: (x_{1}, y_{1})...(x_{n}, y_{n}) ∈ {0, 1}^{d}x{0, 1} Solutio:

w ∈ Q^{D}minimising the number of indicest = 1, ...m s.t. h_{t}w^{T} x_{t} ≤ 0

Opt(S) is the smallest number of mislcassified example in S by any linear classifier in H_{D} where \frac{Opt(S)}{m} is training error of ERM

Theorem : if P ≢ NP, then ∀c > 0 there are no polytime algorithms (with r. t. the input size Θ(m_{d})) that approximately solve every istance S of MinDisOpt with a number of mistakes bounded by C · Opt(S). If I am able to approximate it correclty this approximation will grow with the size of the dataset.

∀A (polytime) and ∀C ∃S  ℓ̂_{S} (A (S)) ≥ c·ℓ̂_{S} \binom{}{ĥ_{S}} (where ĥ_{S} is ERM)
Opt(S) = ℓ̂_{S}(ĥ_{S})

This is not related with free lunch theorem (information we need to get base error for some learning problem). Free lunch: we need arbitrarirally information to get such error. Here is we need a lot of computation to approximate the ERM. Assume Opt(S) = 0 ERM has zero training error on S

∃U ∈ ℝ^{d} s.t. ∀t = 1, ...m  y_{t}U^{T} x_{t} > 0  S is linearly separable

Figure 1.1: Tree building

We can look at the min min_{t=1,...m} y_{t}U^{T} x_{t} = γ(U) > 0 We called this marginn of U on (x_{t}, y_{t})

We called in this way since \frac{γ(U)}{∥U∥} = min ty_{t}∥x_{t}∥cos(Θ) where Θ is the angle where\frac{γ(U)}{∥U∥} is the distance separating hyperplane on closest training example . S linearly separable and if i look at the sistem of this linear inequality:

\begin{cases}y_{t}w_{T} x_{t} > 0 \\ y_{m}w_{T} x_{m} > 0 \end{cases}

Figure 1.2: Tree building

Figure 1.3: Tree building

We can solve it in polytime using a linear solver. So any package of linear programming, and will be solved in linear time.

This is called feasibilty problem. We want a point y that satisfy all my linear inequalities.

When Opt(S) = 0 is we can implememtn ERM efficiently using LP (Linear programming). They may overfitting since a lot of bias. When this condition of Opt is no satisfy we cannot do it efficiently. LP algorithm can be complicated so we figure out another family of algorithm.

Figure 1.4: Feasibilty problem

1.2 The Perception Algorithm

This came from late ’50s and was designed for psicology but have a general utility in othe fields.

Perception Algorithm Input : training set S = {(x_{t}, y_{t})...(x_{m}, y_{m})} x_{t} ∈ ℝ^{d} y_{t} ∈ {−1, +1} Init: w = (0, ...0) Repcat read next (x_{t}, y_{t}) If y_{t}w^{T} x_{t} ≤ then w ⟵ w + y_{t}x_{t} Until margin greater than 0 γ(w) > 0 // w separates S Output w

We know that γ(w) = min_{t} y_{t}w^{T} x_{t} ≤ 0 The question is, will it terminate if S is linearly separable? If y_{t}w^{T} x_{t} ≤ 0, then w ⟵ w + y_{t}x_{t}

Figure 1.5:

For simplicity our x are in this circle. Some are on the circonference on top left with + sign and some in bottom right with − sign.

All minus flipped to the other side and the we can deal the +. U is a separating hyperplane, how can i find it? Maybe i can do something like the average:

U = \frac{1}{m} ∑_{t=1}^{m} y_{t}x_{t} ?

But actually don’t take the average of all of them. So do not take average of all, instead take the one that satisfy y_{t}w^{T} x_{t} ≤ 0 condition. y_{t}w^{T} x_{t} ≤ 0 is a violated consstraint and we want it > 0. Does w ⟵ w + y_{t}x_{t} fix it?

y_{t}(w + y_{t} · x_{t})^{T} x_{t} = y_{t}w^{T} x_{t} + ∥x_{t}∥^{2}

We are trying to see what happen before and after the updates of w. SInce ∥x_{t}∥ > 0 so is positive, the update increase margins, thus going towards fixing violated constraints.

1.2.1 Perception convergence Theorem

dated early 60s On a linearly separable S, perceptron will converge after at most M updates (when they touch in the figure) where:

M ≤ ( min_{U : γ(U) = 1} ∥U∥^{2}) ( max_{t=1,..m} ∥x_{t}∥^{2})

Algorithm is not able to do that. ALgorithm keeps looking till he get a violating constraint and then stops. This is bounded by the number of loops.

We said that γ(U) = min_{t} y_{t}U^{T} x_{t} > 0 when U is separator.

∀t y_{t}U^{T} x_{t} ≥ γ(U) ⇔ ∀t y_{t} (\frac{U}{γ(U)})^{T} x_{t} ≥ 1

If i rescale U i can make the margin bigger (in particolar > 1) The shortest min∥U∥ s.t. y_{t}U^{T} x_{t} ≥ 1 ∀t

Proof: W_{m} is local variable after M updates, I have zero vector W_{0} = (0, ...0) t_{M} is the index of training example that causes the M-th update.

Figure 1.6:

We want to upper bound M (deriving upper and lower bound on a certain quantity ∥W ∥ ∥U∥) where U is any s.t. y_{t}U^{T} x_{t} ≥ 1 ∀t

∥W_{M}∥^{2} = ∥W_{M−1} +y_{tM}x_{tM}∥^{2} = ∥W_{M−1}∥^{2} +∥y_{tM}x_{tM}∥^{2} +2·y_{tM}W_{M−1}^{T}x_{tM} =

= ∥W_{M−1}∥^{2} + ∥x_{tM}∥^{2} + 2 · y_{tM}W_{M−1}^{T}x_{tM} ≤

where y_{tM}W_{M−1}^{T}x_{tM} ≤ 0

≤ ∥w_{M−1}∥^{2} + ∥x_{tM}∥^{2}

∥W_{M}∥^{2} ≤ ∥W_{0}∥^{2} + ∑_{i=2}^{M} ∥x_{t}

........

.....

... MANCA

∥W_{M}∥ ∥U∥ ≤ ∥U∥\sqrt{M} (max_{t} ∥x_{t}∥)

since cos Θ ∈ [−1, 1]

∥W_{M}∥ ∥U∥ ≥ ∥W_{M}∥ ∥U∥ cos Θ = W_{M}^{T}U = (W_{M−1} + y_{tM}x_{tM})^{T} U =

where last passage is the Inner product

W_{M−1}^{T}U + y_{tM}U^{T} x_{tM} ≥ W_{M−1}^{T}U + 1 ≥ W_{0}^{T} U + M = M

where y_{tM}U^{T} x_{tM} is ≥ 1

M ≤ ∥W_{M}∥ ∥U∥ ≤ ∥U∥\sqrt{M} (max_{t} ∥x_{T} ∥)

M ≤ (∥U∥^{2})(max_{t}∥x_{t}∥^{2}) ∀U : min_{t} y_{t}U^{t}x_{t} ≥ 1

M = ( min_{U : γ(U)=1} ∥U∥^{2})(max_{t} ∥∥^{2})

Some number depends on S