Lecture 11 - 20-04-2020

1.1 Analysis of K_{NN}

𝔼 [ℓ_{D}(ℓ̂_{s})] ≤ 2 · ℓ_{D} (f^{∗}) + c · 𝔼 [ ∥X = x_{Π(s,x)}∥ ]

At which rate this thing goes down? If number of dimension goes up then a lot of point are far away from X. So this quantity must depend on the space in which X live. Some dependence on number of depends and incresaing number of traning points close to X This expecation is fucniton of random variable X and X_{π(s,x)}

We are going to use the assumption that: |X_{t}| ≤ 1 ∀ cordinates i = 1, ..., d

Figure 1.1: Example of domain of K_{NN}

Hyper box in bydimension. All point live in this box and we exploit that. Look at the little suare in which is divided and we assume that we are dividing the box in small boxes of size ε. Now the training points will be a strincle of point distributed in the big square.

Our training points are distribuited in the box (this is our S). Now we added a point x and given this two things can happned: falls in the square with training points or in a square without training points. What is going to be the distance X_{π(s,x)} in this two cases? We have c_{1} up to c_{r} How big is this when we have this two cases? (We lookjing at specific choices of x and s)

∥X − X_{s,x}∥ ≤ \{ ; \begin{matrix}ε\sqrt{d} C_{i} ∪ S ≠ 0 \\ \sqrt{d} C_{i} ∪ S = 0 \end{matrix}

were X ∈ C_{i} We have to multiply by the lenght of the cube. Will be ε \sqrt{d}

Figure 1.2: Diagonal length

If things go badly can be very far away like the length of the domain. Lenght is 2 and diagonal is \sqrt{d} if close they are going to be εclose or far as domain. We can split that the expression inside the expectation according to the two cases.

𝔼 [ ∥X − X_{Π(s,x)}∥ ] ≤

𝔼 [ε ·\sqrt{d} · ∑_{i=1}^{r} I\{X ∈ C_{i}\} · I\{C_{i} ∩ S ≠ 0\}]+ 2 ·\sqrt{d} · ∑_{i=1}^{r} I\{X ∈ C_{i}\}·I\{C_{i}∩S ≠ 0\} =

= ε·\sqrt{d}· 𝔼 [ ∑_{t=1}^{r} I\{X ∈ C_{i}\}I · \{C_{i} ∩ S ≠ 0\} ]+2·\sqrt{d} · ∑_{i=1}^{r} 𝔼 [ I\{X ∈ C_{1}\} · I\{C_{1} ∩ S ≠ 0\}] ≤

I don’t care about this one ∑_{t=1}^{r} I{X ∈ C_{i}} · I{C_{i} ∩ S ≠ 0} Can be either 0 or 1 (if for some i, X belong to some C_{i} So at most 1 the expectation

≤ ε · \sqrt{d} + □

We can bound this square. Are the event I in the summation of the term after +. If they are indepednt the product will be the product of the two expectation. If I fix the cube. X and S are independent. Now the two events are independent X ∈ C_{i} is independent of C_{i} ∩ S ≠

𝔼 [ I{X ∈ C_{i}} · I{C_{i} ∩ S ≠ 0}] = 𝔼 [I{X ∈ C_{i}}] · 𝔼 [I{C_{i}}]

MANCAAAAAAA 9.26

ℙ (C_{i} ∩ S) = (1 − ℙ (X ∈ C_{1}))^{m} ≤ exp(−m · ℙ(x ∈ C_{1}))

The probability of the point fall there and will be the probability of falling in the cube. Probability of Xs to fall in the cube with a m (samples?) Now use inequality (1 − p)^{m} ∈ e^{−pm} ⟶ 1 + x ≤ e^{x}

Figure 1.3: Shape of the function

∑_{t=1}^{r} 𝔼 [ ℙ (X ∈ C_{1}) · ℙ (C_{1} ∩ S ≠) ] ≤ ∑_{i=1}^{r} p_{i} · e^{−m p_{i}} ≤

given that p_{i} = ℙ(X ∈ C_{i}) I can upper bound this

≤ ∑_{t=1}^{r} ( max_{0≤p≤1} p e^{−m p}) ≤ r max_{0≤p≤1} p e^{−m p} =

where p e^{−m p} is F (p) it is concave function so i’m going to take first order derivative to maximise it.

F′(p) = 0 ⇔ p = \frac{1}{m} check!

F′′(p) ≤ 0

Check this two condition!

= \frac{r}{e m}

Now get expectation

𝔼 [ ∥X − X_{Π(s,x)}∥ ] ≤ ε · \sqrt{d} + (2 ·\sqrt{d}) \frac{r}{e m} =

I have (\frac{2}{ε})^{2} squares. This bring ε in the game

ε · \sqrt{d} + (2 ·\sqrt{d}) \frac{1}{e m} · (\frac{2}{ε})^{d} =

=\sqrt{d} (ε + \frac{2}{e m} · (\frac{2}{ε})^{d})

HE MISS THE "c" costant from the start we can choose ε to take them balanced set ε = 2 m^{\frac{−1}{(d+1)}}

(ε + \frac{2}{e m}) · (\frac{2}{ε}^{d}) ≤ 4 m^{\frac{−1}{(d+1)}} =

𝔼 [ℓ_{d}(ĥ_{s})] ≤ 2ℓ_{d}(f^{∗}) + 4 · c · \sqrt{d} · m^{− \frac{1}{d+1}}

We have that: if m ⟶ ∞ ℓ_{D}(f^{∗}) ≤ 𝔼 [ℓ_{D}(ĥ_{s}]) ≤ 2ℓ_{D}(f^{∗}) I want this smaller than twice risk + some small quantity

𝔼 [ℓ_{d}(ĥ_{S})] ≤ 2ℓ_{D}(f^{∗}) + ε

How big m ? Ignore this part since very small (4 · c · \sqrt{d})

m^{− \frac{1}{d+1}} ≤ ε ⇔ m ≥ (\frac{1}{ε})^{d} + 1

So 1-NN require a training set size exponential "accuracy" 1 − ε

We show that 1 − NN can approach twice based risk 2 · ℓ_{D}(f^{∗}) but it takes a training set exponential in d.

1.1.1 Study of K_{NN}

Maybe we can use the K_{NN}.

𝔼 [ℓ_{D}(ĥ_{s})] ≤ (1 +\sqrt{\frac{8}{k}}) ℓ_{D}(f^{∗}) + 0 (k m^{− \frac{1}{d+1}} )

So is not exponential here. Learning algorithm A is consistent for a certain loss ℓ If ∀ D(distribution) of data we have that A(S_{m}) predictor output by A Now have the risk of that in ℓ_{D}(A(S_{m})) and we look at the expectation 𝔼 [ℓ_{D}(A(S_{m}))] If we give a training set size large ( lim_{m→∞} 𝔼 [ℓ_{D}(A(S_{m}))] = ℓ_{D}(f^{∗}) ) risk will converge in based risk.

K_{NN} where K = K_{m} (is a function of training set size). K_{,} → ∞ as m → ∞. Only way K goes to infinity is sublinearly of training set size. (infinity but so as quicly as m K_{m} = O(m)

For instance K_{m} =\sqrt{m} Then:

lim_{m→∞} 𝔼 [ℓ_{D} (A′ (S_{m}))] = ℓ_{D}(f^{∗}) where A′ is K_{m}-NN

Increasing the size we will converge to this base risk for any distribution and that’s nice.

1.1.2 study of trees

Algorithm that grow tree classifiers can also be made consistent provided two condition:

• The tree keeps growing

• A non-vanishing fraction of traning example is routed to each leaft

Tree has to keep growing but not so fast. Second point is: suppose you have a certain number of leaves and you can look at the fraction. Each leaf ℓ gets N_{ℓ} examples. You want that this fraction at any point of time is not going to 0. The fraction of point every leaf receive a split we are reducing the smallest number of examples.

Example keep growing and leaves too and we want that \frac{N_{ℓ}}{manca} this not going to 0. . since not showed the formula.

Given A, how do I know wheter A could be consistent?

H_{A} ≡ { h : ∃S A(S) = h}

S can be any size. If A is ERM then H_{A} = H, so where ERM minimise it. If ∃f^{∗} : X ⟶ such that f^{∗} ∉ H_{A} and ∃D such that f^{∗} is Bayes optimal for some distribution D. This cannot be consistent because distribution will not be able to generate the Bayes optimal predictor. Maybe is there another predictor f which is not equal to f^{∗} risk.

What’s the intuition? Every time A is such that H_{A} is "restricted" in some sense, then A cannot be consistent. (e.g ERM).

Another way of restricting? Could be tree classifiers with at most N nodes (bound number of nodes). How do i know N is enought to approximate well f^{∗}. I want to converge the risk of f^{∗}. We can introduce a class of algorithm potentially consistent in which space predictor is not restricted.

1.2 Non-parametric Algorithms

When they are potentially consistent. What does it mean? Non-parametric algorithm have the potential of being consistent and do we know if algorithm is parametric or not? A is non-parametric if:

• the description of A(S_{m}) grows with m

Your predictor is a function and let’s assume i can store in any variable a real number with arbitrary precition.

Any algorithm with bias is incosistent. So ability to converge to base risk is this. How do i know if i have bias or not? this is where non parametric algorithm came. Let’s consider K_{NN}, how i can describe it? I have to remember distance is maded by training points and if i give you more S the m will increase. So this is parametric. More training set for tree, then will grow more, even more larger will be ever growing more. Any algorithm as a give training points is no parametric, while growing with parametric will stop a some point.

Figure 1.4: Shape of the function

If algorithm is more parametric as i give training points If a certain point stop growing, f^{∗} will be out and i will grow more. If algorithm is able to generate — MANCA — Then the algorithm is nonparametric and can be potentially consistent and incluse f^{∗} as it grows. If set of predictor stops because I’m not enlarging my set of predictor since description of algortim will not depend on training size at some point → to be consistent. If bias vanashes as i increase the S, then i can be consistent. I generating predictor that description depends on how much points i give them.

Parametric is not precise as consistency. One class of algorithm that has consistency has a predicotr size growing with S growing. Definition of non parametric is more fuzzy, consistency is precise (we demonstrate that mathematically).

1.2.1 Example of parametric algorithms

Neural network is parametric since i give structure of the network. If i give S small or big S my structure will be the same (will fit better on the training points). Other example are algorithm with linear classifier in which number of parameter are just the idmension of the space.