complete lecture 8

1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/lectures/lecture7.log

@@ -1,4 +1,4 @@
-This is pdfTeX, Version 3.14159265-2.6-1.40.21 (MiKTeX 2.9.7300 64-bit) (preloaded format=pdflatex 2020.4.13)  15 APR 2020 12:31
+This is pdfTeX, Version 3.14159265-2.6-1.40.21 (MiKTeX 2.9.7300 64-bit) (preloaded format=pdflatex 2020.4.13)  15 APR 2020 12:37
 entering extended mode
 **./lecture7.tex
 (lecture7.tex

1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/lectures/lecture8.aux

@@ -0,0 +1,14 @@
+\relax 
+\@nameuse{bbl@beforestart}
+\babel@aux{english}{}
+\@writefile{toc}{\contentsline {chapter}{\numberline {1}Lecture 8 - 07-04-2020}{1}\protected@file@percent }
+\@writefile{lof}{\addvspace {10\p@ }}
+\@writefile{lot}{\addvspace {10\p@ }}
+\@writefile{lof}{\contentsline {figure}{\numberline {1.1}{\ignorespaces Representation of $\hat  {h}$, $h^*$ and $f^*$ }}{1}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.2}{\ignorespaces Example}}{2}\protected@file@percent }
+\@writefile{toc}{\contentsline {section}{\numberline {1.1}The problem of estimating risk in practise}{2}\protected@file@percent }
+\@writefile{toc}{\contentsline {section}{\numberline {1.2}Cross-validation}{4}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.3}{\ignorespaces Splitting test and training set}}{4}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.4}{\ignorespaces K-folds}}{5}\protected@file@percent }
+\@writefile{toc}{\contentsline {section}{\numberline {1.3}Nested cross validation}{6}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.5}{\ignorespaces Nested Cross Validation}}{6}\protected@file@percent }

1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/lectures/lecture8.log

@@ -0,0 +1,467 @@
+This is pdfTeX, Version 3.14159265-2.6-1.40.21 (MiKTeX 2.9.7300 64-bit) (preloaded format=pdflatex 2020.4.13)  15 APR 2020 14:36
+entering extended mode
+**./lecture8.tex
+(lecture8.tex
+LaTeX2e <2020-02-02> patch level 5
+L3 programming layer <2020-03-06>
+("C:\Program Files\MiKTeX 2.9\tex/latex/subfiles\subfiles.cls"
+Document Class: subfiles 2020/02/14 v1.6 Multi-file projects (class)
+Preamble taken from file `../main.tex'
+("C:\Program Files\MiKTeX 2.9\tex/latex/tools\verbatim.sty"
+Package: verbatim 2019/11/10 v1.5r LaTeX2e package for verbatim enhancements
+\every@verbatim=\toks15
+\verbatim@line=\toks16
+\verbatim@in@stream=\read2
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/import\import.sty"
+Package: import 2020/04/01  v 6.2
+) (../main.tex
+("C:\Program Files\MiKTeX 2.9\tex/latex/base\report.cls"
+Document Class: report 2019/12/20 v1.4l Standard LaTeX document class
+("C:\Program Files\MiKTeX 2.9\tex/latex/base\size12.clo"
+File: size12.clo 2019/12/20 v1.4l Standard LaTeX file (size option)
+)
+\c@part=\count167
+\c@chapter=\count168
+\c@section=\count169
+\c@subsection=\count170
+\c@subsubsection=\count171
+\c@paragraph=\count172
+\c@subparagraph=\count173
+\c@figure=\count174
+\c@table=\count175
+\abovecaptionskip=\skip47
+\belowcaptionskip=\skip48
+\bibindent=\dimen134
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsmath\amsmath.sty"
+Package: amsmath 2020/01/20 v2.17e AMS math features
+\@mathmargin=\skip49
+
+For additional information on amsmath, use the `?' option.
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsmath\amstext.sty"
+Package: amstext 2000/06/29 v2.01 AMS text
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsmath\amsgen.sty"
+File: amsgen.sty 1999/11/30 v2.0 generic functions
+\@emptytoks=\toks17
+\ex@=\dimen135
+))
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsmath\amsbsy.sty"
+Package: amsbsy 1999/11/29 v1.2d Bold Symbols
+\pmbraise@=\dimen136
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsmath\amsopn.sty"
+Package: amsopn 2016/03/08 v2.02 operator names
+)
+\inf@bad=\count176
+LaTeX Info: Redefining \frac on input line 227.
+\uproot@=\count177
+\leftroot@=\count178
+LaTeX Info: Redefining \overline on input line 389.
+\classnum@=\count179
+\DOTSCASE@=\count180
+LaTeX Info: Redefining \ldots on input line 486.
+LaTeX Info: Redefining \dots on input line 489.
+LaTeX Info: Redefining \cdots on input line 610.
+\Mathstrutbox@=\box45
+\strutbox@=\box46
+\big@size=\dimen137
+LaTeX Font Info:    Redeclaring font encoding OML on input line 733.
+LaTeX Font Info:    Redeclaring font encoding OMS on input line 734.
+\macc@depth=\count181
+\c@MaxMatrixCols=\count182
+\dotsspace@=\muskip16
+\c@parentequation=\count183
+\dspbrk@lvl=\count184
+\tag@help=\toks18
+\row@=\count185
+\column@=\count186
+\maxfields@=\count187
+\andhelp@=\toks19
+\eqnshift@=\dimen138
+\alignsep@=\dimen139
+\tagshift@=\dimen140
+\tagwidth@=\dimen141
+\totwidth@=\dimen142
+\lineht@=\dimen143
+\@envbody=\toks20
+\multlinegap=\skip50
+\multlinetaggap=\skip51
+\mathdisplay@stack=\toks21
+LaTeX Info: Redefining \[ on input line 2859.
+LaTeX Info: Redefining \] on input line 2860.
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/systeme\systeme.sty"
+("C:\Program Files\MiKTeX 2.9\tex/latex/xstring\xstring.sty"
+("C:\Program Files\MiKTeX 2.9\tex/generic/xstring\xstring.tex"
+\integerpart=\count188
+\decimalpart=\count189
+)
+Package: xstring 2019/02/06 v1.83 String manipulations (CT)
+)
+("C:\Program Files\MiKTeX 2.9\tex/generic/systeme\systeme.tex"
+\SYS_systemecode=\toks22
+\SYS_systempreamble=\toks23
+\SYSeqnum=\count190
+)
+Package: systeme 2019/01/13 v0.32 Mise en forme de systemes d'equations (CT)
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsfonts\amssymb.sty"
+Package: amssymb 2013/01/14 v3.01 AMS font symbols
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsfonts\amsfonts.sty"
+Package: amsfonts 2013/01/14 v3.01 Basic AMSFonts support
+\symAMSa=\mathgroup4
+\symAMSb=\mathgroup5
+LaTeX Font Info:    Redeclaring math symbol \hbar on input line 98.
+LaTeX Font Info:    Overwriting math alphabet `\mathfrak' in version `bold'
+(Font)                  U/euf/m/n --> U/euf/b/n on input line 106.
+))
+("C:\Program Files\MiKTeX 2.9\tex/latex/subfiles\subfiles.sty"
+Package: subfiles 2020/02/14 v1.6 Multi-file projects (package)
+)
+("C:\Program Files\MiKTeX 2.9\tex/generic/babel\babel.sty"
+Package: babel 2020/02/28 3.41 The Babel package
+
+("C:\Program Files\MiKTeX 2.9\tex/generic/babel\switch.def"
+File: switch.def 2020/02/28 3.41 Babel switching mechanism
+)
+*************************************
+* Local config file bblopts.cfg used
+*
+("C:\Program Files\MiKTeX 2.9\tex/latex/arabi\bblopts.cfg"
+File: bblopts.cfg 2005/09/08 v0.1 add Arabic and Farsi to "declared" options of
+ babel
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/babel-english\english.ldf"
+Language: english 2017/06/06 v3.3r English support from the babel system
+
+("C:\Program Files\MiKTeX 2.9\tex/generic/babel\babel.def"
+File: babel.def 2020/02/28 3.41 Babel common definitions
+\babel@savecnt=\count191
+\U@D=\dimen144
+
+("C:\Program Files\MiKTeX 2.9\tex/generic/babel\txtbabel.def")
+\bbl@readstream=\read3
+\bbl@dirlevel=\count192
+)
+Package babel Info: \l@canadian = using hyphenrules for english
+(babel)             (\language0) on input line 102.
+Package babel Info: \l@australian = using hyphenrules for ukenglish
+(babel)             (\language72) on input line 105.
+Package babel Info: \l@newzealand = using hyphenrules for ukenglish
+(babel)             (\language72) on input line 108.
+))
+("C:\Program Files\MiKTeX 2.9\tex/latex/xcolor\xcolor.sty"
+Package: xcolor 2016/05/11 v2.12 LaTeX color extensions (UK)
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics-cfg\color.cfg"
+File: color.cfg 2016/01/02 v1.6 sample color configuration
+)
+Package xcolor Info: Driver file: pdftex.def on input line 225.
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics-def\pdftex.def"
+File: pdftex.def 2018/01/08 v1.0l Graphics/color driver for pdftex
+)
+Package xcolor Info: Model `cmy' substituted by `cmy0' on input line 1348.
+Package xcolor Info: Model `hsb' substituted by `rgb' on input line 1352.
+Package xcolor Info: Model `RGB' extended on input line 1364.
+Package xcolor Info: Model `HTML' substituted by `rgb' on input line 1366.
+Package xcolor Info: Model `Hsb' substituted by `hsb' on input line 1367.
+Package xcolor Info: Model `tHsb' substituted by `hsb' on input line 1368.
+Package xcolor Info: Model `HSB' substituted by `hsb' on input line 1369.
+Package xcolor Info: Model `Gray' substituted by `gray' on input line 1370.
+Package xcolor Info: Model `wave' substituted by `hsb' on input line 1371.
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics\dvipsnam.def"
+File: dvipsnam.def 2016/06/17 v3.0m Driver-dependent file (DPC,SPQR)
+))
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics\graphicx.sty"
+Package: graphicx 2019/11/30 v1.2a Enhanced LaTeX Graphics (DPC,SPQR)
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics\keyval.sty"
+Package: keyval 2014/10/28 v1.15 key=value parser (DPC)
+\KV@toks@=\toks24
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics\graphics.sty"
+Package: graphics 2019/11/30 v1.4a Standard LaTeX Graphics (DPC,SPQR)
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics\trig.sty"
+Package: trig 2016/01/03 v1.10 sin cos tan (DPC)
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics-cfg\graphics.cfg"
+File: graphics.cfg 2016/06/04 v1.11 sample graphics configuration
+)
+Package graphics Info: Driver file: pdftex.def on input line 105.
+)
+\Gin@req@height=\dimen145
+\Gin@req@width=\dimen146
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/sectsty\sectsty.sty"
+Package: sectsty 2002/02/25 v2.0.2 Commands to change all sectional heading sty
+les
+
+
+LaTeX Warning: Command \underbar  has changed.
+               Check if current package is valid.
+
+
+LaTeX Warning: Command \underline  has changed.
+               Check if current package is valid.
+
+) ("C:\Program Files\MiKTeX 2.9\tex/latex/framed\framed.sty"
+Package: framed 2011/10/22 v 0.96: framed or shaded text with page breaks
+\OuterFrameSep=\skip52
+\fb@frw=\dimen147
+\fb@frh=\dimen148
+\FrameRule=\dimen149
+\FrameSep=\dimen150
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/titlesec\titlesec.sty"
+Package: titlesec 2019/10/16 v2.13 Sectioning titles
+\ttl@box=\box47
+\beforetitleunit=\skip53
+\aftertitleunit=\skip54
+\ttl@plus=\dimen151
+\ttl@minus=\dimen152
+\ttl@toksa=\toks25
+\titlewidth=\dimen153
+\titlewidthlast=\dimen154
+\titlewidthfirst=\dimen155
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/base\fontenc.sty"
+Package: fontenc 2020/02/11 v2.0o Standard LaTeX package
+)))
+("C:\Program Files\MiKTeX 2.9\tex/latex/l3backend\l3backend-pdfmode.def"
+File: l3backend-pdfmode.def 2020-03-12 L3 backend support: PDF mode
+\l__kernel_color_stack_int=\count193
+\l__pdf_internal_box=\box48
+)
+(lecture8.aux)
+\openout1 = `lecture8.aux'.
+
+LaTeX Font Info:    Checking defaults for OML/cmm/m/it on input line 2.
+LaTeX Font Info:    ... okay on input line 2.
+LaTeX Font Info:    Checking defaults for OMS/cmsy/m/n on input line 2.
+LaTeX Font Info:    ... okay on input line 2.
+LaTeX Font Info:    Checking defaults for OT1/cmr/m/n on input line 2.
+LaTeX Font Info:    ... okay on input line 2.
+LaTeX Font Info:    Checking defaults for T1/cmr/m/n on input line 2.
+LaTeX Font Info:    ... okay on input line 2.
+LaTeX Font Info:    Checking defaults for TS1/cmr/m/n on input line 2.
+LaTeX Font Info:    ... okay on input line 2.
+LaTeX Font Info:    Checking defaults for OMX/cmex/m/n on input line 2.
+LaTeX Font Info:    ... okay on input line 2.
+LaTeX Font Info:    Checking defaults for U/cmr/m/n on input line 2.
+LaTeX Font Info:    ... okay on input line 2.
+ ("C:\Program Files\MiKTeX 2.9\tex/context/base/mkii\supp-pdf.mkii"
+[Loading MPS to PDF converter (version 2006.09.02).]
+\scratchcounter=\count194
+\scratchdimen=\dimen156
+\scratchbox=\box49
+\nofMPsegments=\count195
+\nofMParguments=\count196
+\everyMPshowfont=\toks26
+\MPscratchCnt=\count197
+\MPscratchDim=\dimen157
+\MPnumerator=\count198
+\makeMPintoPDFobject=\count199
+\everyMPtoPDFconversion=\toks27
+) ("C:\Program Files\MiKTeX 2.9\tex/latex/epstopdf-pkg\epstopdf-base.sty"
+Package: epstopdf-base 2020-01-24 v2.11 Base part for package epstopdf
+
+("C:\Program Files\MiKTeX 2.9\tex/generic/infwarerr\infwarerr.sty"
+Package: infwarerr 2019/12/03 v1.5 Providing info/warning/error messages (HO)
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/grfext\grfext.sty"
+Package: grfext 2019/12/03 v1.3 Manage graphics extensions (HO)
+
+("C:\Program Files\MiKTeX 2.9\tex/generic/kvdefinekeys\kvdefinekeys.sty"
+Package: kvdefinekeys 2019-12-19 v1.6 Define keys (HO)
+))
+("C:\Program Files\MiKTeX 2.9\tex/latex/kvoptions\kvoptions.sty"
+Package: kvoptions 2019/11/29 v3.13 Key value format for package options (HO)
+
+("C:\Program Files\MiKTeX 2.9\tex/generic/ltxcmds\ltxcmds.sty"
+Package: ltxcmds 2019/12/15 v1.24 LaTeX kernel commands for general use (HO)
+)
+("C:\Program Files\MiKTeX 2.9\tex/generic/kvsetkeys\kvsetkeys.sty"
+Package: kvsetkeys 2019/12/15 v1.18 Key value parser (HO)
+))
+("C:\Program Files\MiKTeX 2.9\tex/latex/pdftexcmds\pdftexcmds.sty"
+Package: pdftexcmds 2019/11/24 v0.31 Utility functions of pdfTeX for LuaTeX (HO
+)
+
+("C:\Program Files\MiKTeX 2.9\tex/generic/iftex\iftex.sty"
+Package: iftex 2020/03/06 v1.0d TeX engine tests
+)
+Package pdftexcmds Info: \pdf@primitive is available.
+Package pdftexcmds Info: \pdf@ifprimitive is available.
+Package pdftexcmds Info: \pdfdraftmode found.
+)
+Package epstopdf-base Info: Redefining graphics rule for `.eps' on input line 4
+85.
+Package grfext Info: Graphics extension search list:
+(grfext)             [.pdf,.png,.jpg,.mps,.jpeg,.jbig2,.jb2,.PDF,.PNG,.JPG,.JPE
+G,.JBIG2,.JB2,.eps]
+(grfext)             \AppendGraphicsExtensions on input line 504.
+)
+Chapter 1.
+LaTeX Font Info:    Trying to load font information for U+msa on input line 6.
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsfonts\umsa.fd"
+File: umsa.fd 2013/01/14 v3.01 AMS symbols A
+)
+LaTeX Font Info:    Trying to load font information for U+msb on input line 6.
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsfonts\umsb.fd"
+File: umsb.fd 2013/01/14 v3.01 AMS symbols B
+)
+Underfull \hbox (badness 10000) in paragraph at lines 10--13
+
+ []
+
+
+Underfull \hbox (badness 10000) in paragraph at lines 21--25
+
+ []
+
+
+Overfull \hbox (74.73193pt too wide) detected at line 27
+\OML/cmm/m/it/12 `[] []  \OMS/cmsy/m/n/12   \OML/cmm/m/it/12 `[] [] \OT1/cmr/m
+/n/12 + [] []
+ []
+
+<../img/lez8-img1.JPG, id=1, 214.55156pt x 144.54pt>
+File: ../img/lez8-img1.JPG Graphic file (type jpg)
+<use ../img/lez8-img1.JPG>
+Package pdftex.def Info: ../img/lez8-img1.JPG  used on input line 32.
+(pdftex.def)             Requested size: 195.0pt x 131.37057pt.
+
+Underfull \hbox (badness 10000) in paragraph at lines 27--38
+
+ []
+
+
+Underfull \hbox (badness 10000) in paragraph at lines 27--38
+
+ []
+
+
+Underfull \hbox (badness 10000) in paragraph at lines 42--47
+
+ []
+
+[1
+
+
+{C:/Users/AndreDany/AppData/Local/MiKTeX/2.9/pdftex/config/pdftex.map} <../img/
+lez8-img1.JPG>]
+Overfull \hbox (22.02261pt too wide) detected at line 51
+\OMS/cmsy/m/n/12 8\OML/cmm/m/it/12 h \OMS/cmsy/m/n/12 2 \OML/cmm/m/it/12 H []\O
+T1/cmr/m/n/12 (\OML/cmm/m/it/12 h\OT1/cmr/m/n/12 ) \OMS/cmsy/m/n/12  \OML/cmm/
+m/it/12 `[] []  \OMS/cmsy/m/n/12   [] []
+ []
+
+<../img/lez8-img2.JPG, id=18, 215.30438pt x 57.96657pt>
+File: ../img/lez8-img2.JPG Graphic file (type jpg)
+<use ../img/lez8-img2.JPG>
+Package pdftex.def Info: ../img/lez8-img2.JPG  used on input line 55.
+(pdftex.def)             Requested size: 195.0pt x 52.5011pt.
+
+Underfull \hbox (badness 10000) in paragraph at lines 51--61
+
+ []
+
+
+Underfull \hbox (badness 10000) in paragraph at lines 62--71
+
+ []
+
+
+Underfull \hbox (badness 10000) in paragraph at lines 62--71
+
+ []
+
+
+Underfull \hbox (badness 10000) in paragraph at lines 74--104
+
+ []
+
+
+Underfull \hbox (badness 10000) in paragraph at lines 74--104
+
+ []
+
+
+Underfull \hbox (badness 10000) in paragraph at lines 74--104
+
+ []
+
+
+Underfull \hbox (badness 10000) in paragraph at lines 74--104
+
+ []
+
+[2 <../img/lez8-img2.JPG>] [3]
+Underfull \hbox (badness 10000) in paragraph at lines 125--133
+
+ []
+
+<../img/lez8-img3.JPG, id=27, 149.8097pt x 45.92157pt>
+File: ../img/lez8-img3.JPG Graphic file (type jpg)
+<use ../img/lez8-img3.JPG>
+Package pdftex.def Info: ../img/lez8-img3.JPG  used on input line 144.
+(pdftex.def)             Requested size: 195.0pt x 59.7786pt.
+<../img/lez8-img4.JPG, id=28, 258.9675pt x 94.10156pt>
+File: ../img/lez8-img4.JPG Graphic file (type jpg)
+<use ../img/lez8-img4.JPG>
+Package pdftex.def Info: ../img/lez8-img4.JPG  used on input line 165.
+(pdftex.def)             Requested size: 234.00238pt x 85.03378pt.
+[4 <../img/lez8-img3.JPG>]
+Underfull \hbox (badness 10000) in paragraph at lines 183--186
+
+ []
+
+
+Underfull \hbox (badness 10000) in paragraph at lines 188--195
+
+ []
+
+<../img/lez8-img5.JPG, id=33, 441.14813pt x 118.94438pt>
+File: ../img/lez8-img5.JPG Graphic file (type jpg)
+<use ../img/lez8-img5.JPG>
+Package pdftex.def Info: ../img/lez8-img5.JPG  used on input line 206.
+(pdftex.def)             Requested size: 312.00119pt x 84.1262pt.
+[5 <../img/lez8-img4.JPG>] [6 <../img/lez8-img5.JPG>] (lecture8.aux) ) 
+Here is how much of TeX's memory you used:
+ 5133 strings out of 480934
+ 69435 string characters out of 2909670
+ 334073 words of memory out of 3000000
+ 20878 multiletter control sequences out of 15000+200000
+ 546968 words of font info for 58 fonts, out of 3000000 for 9000
+ 1141 hyphenation exceptions out of 8191
+ 42i,7n,50p,332b,238s stack positions out of 5000i,500n,10000p,200000b,50000s
+ <C:\User
+s\AndreDany\AppData\Local\MiKTeX\2.9\fonts/pk/ljfour/jknappen/ec/dpi600\ecbx172
+8.pk> <C:\Users\AndreDany\AppData\Local\MiKTeX\2.9\fonts/pk/ljfour/jknappen/ec/
+dpi600\tcrm1200.pk> <C:\Users\AndreDany\AppData\Local\MiKTeX\2.9\fonts/pk/ljfou
+r/jknappen/ec/dpi600\ecti1200.pk> <C:\Users\AndreDany\AppData\Local\MiKTeX\2.9\
+fonts/pk/ljfour/jknappen/ec/dpi600\ecrm1200.pk> <C:\Users\AndreDany\AppData\Loc
+al\MiKTeX\2.9\fonts/pk/ljfour/jknappen/ec/dpi600\ecbx1200.pk> <C:\Users\AndreDa
+ny\AppData\Local\MiKTeX\2.9\fonts/pk/ljfour/jknappen/ec/dpi600\ecbx2488.pk><C:/
+Program Files/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmex10.pfb><C:/Program 
+Files/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmmi12.pfb><C:/Program Files/Mi
+KTeX 2.9/fonts/type1/public/amsfonts/cm/cmmi6.pfb><C:/Program Files/MiKTeX 2.9/
+fonts/type1/public/amsfonts/cm/cmmi8.pfb><C:/Program Files/MiKTeX 2.9/fonts/typ
+e1/public/amsfonts/cm/cmr12.pfb><C:/Program Files/MiKTeX 2.9/fonts/type1/public
+/amsfonts/cm/cmr8.pfb><C:/Program Files/MiKTeX 2.9/fonts/type1/public/amsfonts/
+cm/cmsy10.pfb><C:/Program Files/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmsy6
+.pfb><C:/Program Files/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmsy8.pfb><C:/
+Program Files/MiKTeX 2.9/fonts/type1/public/amsfonts/symbols/msbm10.pfb>
+Output written on lecture8.pdf (6 pages, 231902 bytes).
+PDF statistics:
+ 254 PDF objects out of 1000 (max. 8388607)
+ 0 named destinations out of 1000 (max. 500000)
+ 26 words of extra memory for PDF output out of 10000 (max. 10000000)
+

1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/lectures/lecture8.tex

@@ -2,4 +2,235 @@
 \begin{document}
 
 \chapter{Lecture 8 - 07-04-2020}
+
+$$
+|H| < \infty \qquad \hat{h} = argmin \, \hat{\ell}_S \left(h\right) \quad h^* = argmin \, \ell_D\left(h\right)
+$$
+
+\blue{minimise risk}
+\\\\
+\bred{Bias-Variance decomposition}
+$$
+\ell_d\left(\hat{h}_S\right) = \quad \ell_D\left(\hat{h}_S\right) - \ell_d\left(h^*\right) + \qquad \longrightarrow \red{\textbf{ Variance error $\Rightarrow$ Overfitting}}
+$$
+$$
+\qquad \quad +\,  \ell_d\left(h^+\right) - \ell_d\left(f^*\right) + \qquad \longrightarrow \red{\textbf{ Bias error $\Rightarrow$ Underfitting}}
+$$
+$$ \qquad \qquad \quad 
++ \, \ell_D\left(f^*\right)\qquad \qquad \quad  \longrightarrow \red{\textbf{ Bayes risk $\Rightarrow$ Unavoidable}} 
+$$\\
+We state this 
+for all algorithm but we studied for ERM.
+\\
+$$
+\ell_D\left(\hat{h}_S\right) \quad \leq \quad \ell_D \left(h^* \right) + \sqrt[]{ \red{\frac{2}{m} \, \ln \, \frac{2\, |H|}{\delta}}} \qquad \textit{with probability at least $1-\delta$ over the draw of $S$}
+$$
+we want \bred{this} to be small when $m>> \ln|H|$
+\\\\
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.5\linewidth]{../img/lez8-img1.JPG}
+    \caption{Representation of $\hat{h}$, $h^*$ and $f^*$ }
+    %\label{fig:}
+\end{figure}\\
+Size of model fix, increase $m$ (size of sample) when $m$ is bigger $\longrightarrow$ variance goes down.\\
+When $|H|$ $\longrightarrow$ big model will be closer to optimal\\
+\begin{itemize}
+\item $m$ grows $\Rightarrow$ variance error goes down \qquad (if not overfitting)
+\item $|H|$ grows $\Rightarrow$ bias error goes down \qquad (if not underfitting)
+\end{itemize}
+ERM with $|H| < \infty$
+\\
+$A$ \qquad $H$ such that $\forall s$ $A(S) \in H$
+\\\\
+We controlled this event:
+$$
+\forall h \in H \qquad \hat{\ell}_S(h) - \ell_D \left(h\right) \quad \leq \quad \sqrt[]{\frac{2}{m}\, \ln \, \frac{2\,|H|}{\delta}} 
+\qquad 
+\textit{ with probability at least $1-\delta$}
+$$\\
+We assure that training error is a good proxy for the true risk.
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.5\linewidth]{../img/lez8-img2.JPG}
+    \caption{Example}
+    %\label{fig:}
+\end{figure}\\
+\bred{If I do the empirical way(for a specific training set) i get something different.}
+\\
+\section{The problem of estimating risk in practise}
+Test error is a good estimate for a risk, provided $\frac{1}{\sqrt[]{n}}$ is small\\\\
+It’s usually good to take a small test set.\\
+$80/20$ big data, if low data better to look at good estimate...=??\\
+Usually small test set is still ok\\\\
+Typically we are not given the predictor, instead we start from a learning
+algorithm. It’s true that A has parameter (how many nodes for a tree classifier
+for example).\\
+In general, if I have parameters then i get a set of different classifier
+Imaging Algorithm that has parameter $\theta$:
+$$ 
+A \qquad  \{ A_{\theta}: \theta \in \Theta \} \qquad
+A_{\theta} \left(S\right) = \hat{h}_S
+$$
+$\barra{E} \left[ \ell_D \left( A_\theta \left(S \right) \right) \right]
+$ \qquad \bred{typically averaged over $S$ of size $m$ for fixed $m$}
+\\\\In general you may also be interest looking at the best choice of parameter for
+your algorithm:
+Suppose now that i want to look at mini sing the risk with the respect to the
+choice of the parameter\\\\
+$\barra{E} \left[ \, min \, \ell_D \left( A_\theta \left(S \right) \right) \right]
+$ \quad i want to choose parameter that minimise risk running algorithm on training set.
+\\\\
+I would like to choose best possible value for my parameter k in the k-NN and
+i want to estimate the risk.\\
+$\theta$ can be a set of parameters (so more than 1 parameter), so theta are
+the \textbf{hyper parameters of A}.\\
+In general this parameter are the choices that algorithm that can make:
+parameter that define a classifier (example nodes and test in the internal
+nodes and label in the leaves)\\\\
+\bred{Hyper parameters} are not determined by the training set $\rightarrow$ chosen
+before the training set.\\
+I fix $k$ for $\knn$ and I choose an upper bound of number of nodes after the
+training set is given.\\
+So, some parameters are given after training set is given and some parameter
+fixed before.\\
+Now the idea is that parameter are determined after training set is given, while
+hyper parameters are given before training set to get then a predictor.\\
+I have a family of algorithms, so I choose a hyper parameters to get a one
+algorithm.\\
+Most algorithm are given as family. We have first decide hyper parameters not
+determined on the training set.\\
+One way to move is to take you dataset and the split that in 3 parts:
+\begin{itemize}
+\item Training set
+\item Development (or validation) test
+\item Test sets
+\end{itemize}
+There should not be a leak of information between test and train and
+development.\\
+We get family of algorithm, train with train set and then use dev set to test the
+parameter and choice the parameters.
+Once i found the parameter I retrain the algorithm with a part of train and dev
+set to being then tested.
+\\
+\bred{Development set is like a fake test set } $\rightarrow$ it usefull to choose parameters.
+\\
+Algorithm steps:
+\begin{enumerate}
+\item  Train $A_\theta$ on training set for each $\theta \in \Theta $ (\bred{grid search}) 
+\item Find $\theta$ such that $\hat{h}= A_{\hat{\theta}}\left(S\right)$ \, minimise developement error
+\item Train $A_{\hat{\theta}}$ on training $+$ development set
+\item Test resulting predictor on test set
+\end{enumerate}
+(There's theory about this but it's difficult and we are not going to do that)
+\\\\
+It’s heuristic and kinda simple to do. This technics work for every learning
+algorithm.\\
+One parameter: grid on this\\
+Two parameters ecc..\\
+\textbf{It’s quadratic!}\\
+\bred{Good learning algorithm should have small number of hyperparameters.}
+
+\section{Cross-validation}
+
+Solving an easier problem: suppose you have a dataset and what you do is
+that you can choose training set and test set.\\
+Algorithm $A$ with no hyper parameters and we would like to check how good is
+the predictor: how good can $A$ be? $A$ is good if the predictor that generate has
+low risk.\\
+I split dataset in training and test set.
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.5\linewidth]{../img/lez8-img3.JPG}
+    \caption{Splitting test and training set}
+    %\label{fig:}
+\end{figure}
+$$
+h = A\left(A\right) \qquad \hat{\ell}_{S'}\left(h\right) \approx \ell_D\left(h\right)
+$$
+I can use Cross-validation! (CV). It helps estimate the risk.
+$$
+\bred{CV:} \qquad \barra{E}\left[ \ell_D \left(A\left(S \right) \right) \right]
+$$
+I would like to average .... \\
+How do i do this estimate? Super easy, take my data assuming CV as
+parameter not of the algorithm A, but intrinsic to cross validation.\\
+Parameter k-fold CV typically $k = 5$ or $10$.\\
+What do you do ? If i take a specific training set, i got a ruff estimate of A.
+I shouldn’t split one but several time.\\
+There are different way but he give us another:\\
+\textbf{split data in k folds randomly!}
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.6\linewidth]{../img/lez8-img4.JPG}
+    \caption{K-folds}
+    %\label{fig:}
+\end{figure}\\
+For each $k$ we define $S^{(k)}
+ \qquad \bred{take out k-st D}$
+$$
+S^{(k)} = S\backslash D_k
+$$
+$$
+S^{(1)} = D_2 \cup D_3 \cup ...\cup D_k
+$$
+$$
+S^{(2)} = D_1 \cup D_3 \cup ... \cup D_k
+$$
+For each $k-1, ... k$ \bred{folds} \qquad $S^{(k)}$ training part and $D_k$ test part
+$$
+h_K = A \left( S^{(k)} \right) \qquad \hat{\ell}_{DK} \left(h_K\right) = \frac{k}{m} \, \sum_{ (x,y) \in D_k}{} \ell \left( y, h_K\left( x \right) \right ) 
+$$
+Repeat the procedure for $k = 1 ... k$ \quad get $h_1, ... , h_k$
+\\
+$$
+Compute \qquad \frac{1}{k} \cdot \sum_{k=1}^{K} \hat{\ell}_{DK} \left(h_k\right)
+$$
+where $CV$ is $\barra{E}\left[ \ell_D \left(A\left(S \right) \right) \right]$ and this is called ..... ---MANCA ---  Estimate
+\\\\
+It’s used a lot!\\
+You get data from internet, then what to do? I want to try an algorithm, try $\knn$ so you do Cross validation and will give you the risk.\\
+In some other cases you get a splitted dataset in training and testing set.
+You don’t use CV since the dataset is already splitted in training and test.
+
+\section{Nested cross validation}
+The use of CV to solve the hyper parameters choice problem.
+If you are given test and training set you can solve splitting in training set and
+dev set.\\
+Suppose not given train and test, you can assign arbitrary .....\\
+Now the idea: you give me a way to optimise splitting training set and test set
+and then avoiding using a ...\\
+\textbf{This is what cross validation is doing.}
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.8\linewidth]{../img/lez8-img5.JPG}
+    \caption{Nested Cross Validation}
+    %\label{fig:}
+\end{figure}\\
+$ \{ A_\theta : \theta \in \Theta \}$
+which i run in each fold?
+\\
+The idea is \bred{running interval CV for the folds}\\
+I have fold $D_1$ and $S(1)$ and i perform external validation, then i run internal
+CV in $S(1)$.\\
+On each training part of the external CV, run a internal CV for each $A_\theta$.
+\\When $\theta \in $ grid ($\Theta$):
+\begin{itemize}
+\item I have a CV-estimate and for each $A_\theta$ pick the $\theta$ with best CV-estimate
+\item Run $A_{\hat{\theta}}$ on the entire training part of current external fold
+\end{itemize}
+\textbf{Basically what I’m choosing the best hyper parameter on each fold of the
+external CV.}\\
+External CV is not testing $A_{\hat{\theta}}$ for a given $\hat{\theta} \in \Theta$
+\\
+I am not measuring a goodness of predictor generate by algorithm for given
+value of hyper parameters but what I’m estimating is the average risk of the
+predictor output by learning algorithm when hyper parameters are optimise on
+the training set. This optimisation on training set is done into a internal CV. To
+avoid be depend i run and external CV.\\
+Many platform like sklearn allow you to do that in two lines of code. So i can
+specify the grid, predictor, number of falls for internal and external.
+It took a bit but that’s it in in two lines of code\\
+Cross validation can be done in every order of $D_1$ or $D_2$ or $D_k$. So doesn’t
+matter the order we start fold D2 or D1.
 \end{document}