Lecture 17 - 11-05-2020

1.1 Strongly convex loss functions

We will saw with OGD but we will see Support Vector Machine(SVM). Very popular learning model. We will see SVM next to see the part of linear predictor and also speak about Kernel function used with linear predictor to obtain non -linear classifier from a linear classifier.

ℓ is σ-SC if ∀u, w:

ℓ(w) − ℓ(u) ≤ ∇ℓ(w)^{T} (w − u) − \frac{σ}{2} | w − u∥^{2}

1.1.1 OGD for Strongly Convex losses

Init: w_{1} = (0, ..., 0) For t = 1, 2... w_{t+1} = w_{t} − \frac{1}{σ t}∇ℓ_{t}(w_{t}) η_{t} = \frac{1}{η t} (no projection steps)

ℓ_{t}(w_{t}) − ℓ_{t}(u) ≤ ∇ℓ_{t}(w_{t})^{T} (w − u) − \frac{σ}{2} ∥w_{t} − u∥^{2} =

= −\frac{1}{η_{t}} (w_{t+1} − w_{t})^{T} (w_{t} − u) − \frac{σ}{2} ∥w_{t} − u∥^{2} =

= \frac{1}{η_{t}} (\frac{1}{2}∥w_{t} − u∥^{2} − \frac{1}{2}∥w_{t+1} − u∥^{2} + \frac{1}{2}∥w_{t+1} − w_{t}∥^{2}) − \frac{σ}{2} ∥w_{t} − u∥^{2}

R_{T} (u) ≤ \frac{1}{2 η_{1}} ∥w_{1}−u∥^{2}−\frac{1}{2 η_{T +1}} ∥w_{T +1}−u∥^{2}−\frac{σ}{2} ∥w_{1}−u∥^{2}+\frac{1}{2} ∑_{t=1}^{T −1} ∥w_{t+1}−u∥^{2} (\frac{1}{η_{t+1}} − \frac{1}{η_{t}} − σ)

x