Last lecture!

1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.aux

@@ -0,0 +1,26 @@
+\relax 
+\@nameuse{bbl@beforestart}
+\babel@aux{english}{}
+\@writefile{toc}{\contentsline {chapter}{\numberline {1}Lecture 24 - 09-06-2020}{1}\protected@file@percent }
+\@writefile{lof}{\addvspace {10\p@ }}
+\@writefile{lot}{\addvspace {10\p@ }}
+\@writefile{toc}{\contentsline {section}{\numberline {1.1}Neural Networks}{1}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {1.1.1}Feedforward NN}{1}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.1}{\ignorespaces }}{1}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.2}{\ignorespaces }}{1}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.3}{\ignorespaces }}{2}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.4}{\ignorespaces }}{2}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.5}{\ignorespaces }}{3}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.6}{\ignorespaces }}{3}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.7}{\ignorespaces }}{3}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.8}{\ignorespaces }}{4}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.9}{\ignorespaces }}{4}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.10}{\ignorespaces }}{5}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.11}{\ignorespaces }}{5}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.12}{\ignorespaces }}{6}\protected@file@percent }
+\@writefile{toc}{\contentsline {section}{\numberline {1.2}Deep Learning}{7}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.13}{\ignorespaces }}{7}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {1.2.1}Convolution Neural Network}{8}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.14}{\ignorespaces }}{8}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {1.2.2}General infos about Neural Networks}{8}\protected@file@percent }
+\@writefile{lof}{\contentsline {figure}{\numberline {1.15}{\ignorespaces }}{9}\protected@file@percent }

1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.log

@@ -0,0 +1,507 @@
+This is pdfTeX, Version 3.14159265-2.6-1.40.21 (MiKTeX 2.9.7300 64-bit) (preloaded format=pdflatex 2020.4.13)  9 JUN 2020 10:29
+entering extended mode
+**./lecture24.tex
+(lecture24.tex
+LaTeX2e <2020-02-02> patch level 5
+L3 programming layer <2020-03-06>
+("C:\Program Files\MiKTeX 2.9\tex/latex/subfiles\subfiles.cls"
+Document Class: subfiles 2020/02/14 v1.6 Multi-file projects (class)
+Preamble taken from file `../main.tex'
+("C:\Program Files\MiKTeX 2.9\tex/latex/tools\verbatim.sty"
+Package: verbatim 2019/11/10 v1.5r LaTeX2e package for verbatim enhancements
+\every@verbatim=\toks15
+\verbatim@line=\toks16
+\verbatim@in@stream=\read2
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/import\import.sty"
+Package: import 2020/04/01  v 6.2
+) (../main.tex
+("C:\Program Files\MiKTeX 2.9\tex/latex/base\report.cls"
+Document Class: report 2019/12/20 v1.4l Standard LaTeX document class
+("C:\Program Files\MiKTeX 2.9\tex/latex/base\size12.clo"
+File: size12.clo 2019/12/20 v1.4l Standard LaTeX file (size option)
+)
+\c@part=\count167
+\c@chapter=\count168
+\c@section=\count169
+\c@subsection=\count170
+\c@subsubsection=\count171
+\c@paragraph=\count172
+\c@subparagraph=\count173
+\c@figure=\count174
+\c@table=\count175
+\abovecaptionskip=\skip47
+\belowcaptionskip=\skip48
+\bibindent=\dimen134
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsmath\amsmath.sty"
+Package: amsmath 2020/01/20 v2.17e AMS math features
+\@mathmargin=\skip49
+
+For additional information on amsmath, use the `?' option.
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsmath\amstext.sty"
+Package: amstext 2000/06/29 v2.01 AMS text
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsmath\amsgen.sty"
+File: amsgen.sty 1999/11/30 v2.0 generic functions
+\@emptytoks=\toks17
+\ex@=\dimen135
+))
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsmath\amsbsy.sty"
+Package: amsbsy 1999/11/29 v1.2d Bold Symbols
+\pmbraise@=\dimen136
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsmath\amsopn.sty"
+Package: amsopn 2016/03/08 v2.02 operator names
+)
+\inf@bad=\count176
+LaTeX Info: Redefining \frac on input line 227.
+\uproot@=\count177
+\leftroot@=\count178
+LaTeX Info: Redefining \overline on input line 389.
+\classnum@=\count179
+\DOTSCASE@=\count180
+LaTeX Info: Redefining \ldots on input line 486.
+LaTeX Info: Redefining \dots on input line 489.
+LaTeX Info: Redefining \cdots on input line 610.
+\Mathstrutbox@=\box45
+\strutbox@=\box46
+\big@size=\dimen137
+LaTeX Font Info:    Redeclaring font encoding OML on input line 733.
+LaTeX Font Info:    Redeclaring font encoding OMS on input line 734.
+\macc@depth=\count181
+\c@MaxMatrixCols=\count182
+\dotsspace@=\muskip16
+\c@parentequation=\count183
+\dspbrk@lvl=\count184
+\tag@help=\toks18
+\row@=\count185
+\column@=\count186
+\maxfields@=\count187
+\andhelp@=\toks19
+\eqnshift@=\dimen138
+\alignsep@=\dimen139
+\tagshift@=\dimen140
+\tagwidth@=\dimen141
+\totwidth@=\dimen142
+\lineht@=\dimen143
+\@envbody=\toks20
+\multlinegap=\skip50
+\multlinetaggap=\skip51
+\mathdisplay@stack=\toks21
+LaTeX Info: Redefining \[ on input line 2859.
+LaTeX Info: Redefining \] on input line 2860.
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/systeme\systeme.sty"
+("C:\Program Files\MiKTeX 2.9\tex/latex/xstring\xstring.sty"
+("C:\Program Files\MiKTeX 2.9\tex/generic/xstring\xstring.tex"
+\integerpart=\count188
+\decimalpart=\count189
+)
+Package: xstring 2019/02/06 v1.83 String manipulations (CT)
+)
+("C:\Program Files\MiKTeX 2.9\tex/generic/systeme\systeme.tex"
+\SYS_systemecode=\toks22
+\SYS_systempreamble=\toks23
+\SYSeqnum=\count190
+)
+Package: systeme 2019/01/13 v0.32 Mise en forme de systemes d'equations (CT)
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsfonts\amssymb.sty"
+Package: amssymb 2013/01/14 v3.01 AMS font symbols
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsfonts\amsfonts.sty"
+Package: amsfonts 2013/01/14 v3.01 Basic AMSFonts support
+\symAMSa=\mathgroup4
+\symAMSb=\mathgroup5
+LaTeX Font Info:    Redeclaring math symbol \hbar on input line 98.
+LaTeX Font Info:    Overwriting math alphabet `\mathfrak' in version `bold'
+(Font)                  U/euf/m/n --> U/euf/b/n on input line 106.
+))
+("C:\Program Files\MiKTeX 2.9\tex/latex/subfiles\subfiles.sty"
+Package: subfiles 2020/02/14 v1.6 Multi-file projects (package)
+)
+("C:\Program Files\MiKTeX 2.9\tex/generic/babel\babel.sty"
+Package: babel 2020/02/28 3.41 The Babel package
+
+("C:\Program Files\MiKTeX 2.9\tex/generic/babel\switch.def"
+File: switch.def 2020/02/28 3.41 Babel switching mechanism
+)
+*************************************
+* Local config file bblopts.cfg used
+*
+("C:\Program Files\MiKTeX 2.9\tex/latex/arabi\bblopts.cfg"
+File: bblopts.cfg 2005/09/08 v0.1 add Arabic and Farsi to "declared" options of
+ babel
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/babel-english\english.ldf"
+Language: english 2017/06/06 v3.3r English support from the babel system
+
+("C:\Program Files\MiKTeX 2.9\tex/generic/babel\babel.def"
+File: babel.def 2020/02/28 3.41 Babel common definitions
+\babel@savecnt=\count191
+\U@D=\dimen144
+
+("C:\Program Files\MiKTeX 2.9\tex/generic/babel\txtbabel.def")
+\bbl@readstream=\read3
+\bbl@dirlevel=\count192
+)
+Package babel Info: \l@canadian = using hyphenrules for english
+(babel)             (\language0) on input line 102.
+Package babel Info: \l@australian = using hyphenrules for ukenglish
+(babel)             (\language72) on input line 105.
+Package babel Info: \l@newzealand = using hyphenrules for ukenglish
+(babel)             (\language72) on input line 108.
+))
+("C:\Program Files\MiKTeX 2.9\tex/latex/xcolor\xcolor.sty"
+Package: xcolor 2016/05/11 v2.12 LaTeX color extensions (UK)
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics-cfg\color.cfg"
+File: color.cfg 2016/01/02 v1.6 sample color configuration
+)
+Package xcolor Info: Driver file: pdftex.def on input line 225.
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics-def\pdftex.def"
+File: pdftex.def 2018/01/08 v1.0l Graphics/color driver for pdftex
+)
+Package xcolor Info: Model `cmy' substituted by `cmy0' on input line 1348.
+Package xcolor Info: Model `hsb' substituted by `rgb' on input line 1352.
+Package xcolor Info: Model `RGB' extended on input line 1364.
+Package xcolor Info: Model `HTML' substituted by `rgb' on input line 1366.
+Package xcolor Info: Model `Hsb' substituted by `hsb' on input line 1367.
+Package xcolor Info: Model `tHsb' substituted by `hsb' on input line 1368.
+Package xcolor Info: Model `HSB' substituted by `hsb' on input line 1369.
+Package xcolor Info: Model `Gray' substituted by `gray' on input line 1370.
+Package xcolor Info: Model `wave' substituted by `hsb' on input line 1371.
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics\dvipsnam.def"
+File: dvipsnam.def 2016/06/17 v3.0m Driver-dependent file (DPC,SPQR)
+))
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics\graphicx.sty"
+Package: graphicx 2019/11/30 v1.2a Enhanced LaTeX Graphics (DPC,SPQR)
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics\keyval.sty"
+Package: keyval 2014/10/28 v1.15 key=value parser (DPC)
+\KV@toks@=\toks24
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics\graphics.sty"
+Package: graphics 2019/11/30 v1.4a Standard LaTeX Graphics (DPC,SPQR)
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics\trig.sty"
+Package: trig 2016/01/03 v1.10 sin cos tan (DPC)
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/graphics-cfg\graphics.cfg"
+File: graphics.cfg 2016/06/04 v1.11 sample graphics configuration
+)
+Package graphics Info: Driver file: pdftex.def on input line 105.
+)
+\Gin@req@height=\dimen145
+\Gin@req@width=\dimen146
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/sectsty\sectsty.sty"
+Package: sectsty 2002/02/25 v2.0.2 Commands to change all sectional heading sty
+les
+
+
+LaTeX Warning: Command \underbar  has changed.
+               Check if current package is valid.
+
+
+LaTeX Warning: Command \underline  has changed.
+               Check if current package is valid.
+
+) ("C:\Program Files\MiKTeX 2.9\tex/latex/framed\framed.sty"
+Package: framed 2011/10/22 v 0.96: framed or shaded text with page breaks
+\OuterFrameSep=\skip52
+\fb@frw=\dimen147
+\fb@frh=\dimen148
+\FrameRule=\dimen149
+\FrameSep=\dimen150
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/titlesec\titlesec.sty"
+Package: titlesec 2019/10/16 v2.13 Sectioning titles
+\ttl@box=\box47
+\beforetitleunit=\skip53
+\aftertitleunit=\skip54
+\ttl@plus=\dimen151
+\ttl@minus=\dimen152
+\ttl@toksa=\toks25
+\titlewidth=\dimen153
+\titlewidthlast=\dimen154
+\titlewidthfirst=\dimen155
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/base\fontenc.sty"
+Package: fontenc 2020/02/11 v2.0o Standard LaTeX package
+)))
+("C:\Program Files\MiKTeX 2.9\tex/latex/l3backend\l3backend-pdfmode.def"
+File: l3backend-pdfmode.def 2020-03-12 L3 backend support: PDF mode
+\l__kernel_color_stack_int=\count193
+\l__pdf_internal_box=\box48
+)
+(lecture24.aux)
+\openout1 = `lecture24.aux'.
+
+LaTeX Font Info:    Checking defaults for OML/cmm/m/it on input line 2.
+LaTeX Font Info:    ... okay on input line 2.
+LaTeX Font Info:    Checking defaults for OMS/cmsy/m/n on input line 2.
+LaTeX Font Info:    ... okay on input line 2.
+LaTeX Font Info:    Checking defaults for OT1/cmr/m/n on input line 2.
+LaTeX Font Info:    ... okay on input line 2.
+LaTeX Font Info:    Checking defaults for T1/cmr/m/n on input line 2.
+LaTeX Font Info:    ... okay on input line 2.
+LaTeX Font Info:    Checking defaults for TS1/cmr/m/n on input line 2.
+LaTeX Font Info:    ... okay on input line 2.
+LaTeX Font Info:    Checking defaults for OMX/cmex/m/n on input line 2.
+LaTeX Font Info:    ... okay on input line 2.
+LaTeX Font Info:    Checking defaults for U/cmr/m/n on input line 2.
+LaTeX Font Info:    ... okay on input line 2.
+
+("C:\Program Files\MiKTeX 2.9\tex/context/base/mkii\supp-pdf.mkii"
+[Loading MPS to PDF converter (version 2006.09.02).]
+\scratchcounter=\count194
+\scratchdimen=\dimen156
+\scratchbox=\box49
+\nofMPsegments=\count195
+\nofMParguments=\count196
+\everyMPshowfont=\toks26
+\MPscratchCnt=\count197
+\MPscratchDim=\dimen157
+\MPnumerator=\count198
+\makeMPintoPDFobject=\count199
+\everyMPtoPDFconversion=\toks27
+) ("C:\Program Files\MiKTeX 2.9\tex/latex/epstopdf-pkg\epstopdf-base.sty"
+Package: epstopdf-base 2020-01-24 v2.11 Base part for package epstopdf
+
+("C:\Program Files\MiKTeX 2.9\tex/generic/infwarerr\infwarerr.sty"
+Package: infwarerr 2019/12/03 v1.5 Providing info/warning/error messages (HO)
+)
+("C:\Program Files\MiKTeX 2.9\tex/latex/grfext\grfext.sty"
+Package: grfext 2019/12/03 v1.3 Manage graphics extensions (HO)
+
+("C:\Program Files\MiKTeX 2.9\tex/generic/kvdefinekeys\kvdefinekeys.sty"
+Package: kvdefinekeys 2019-12-19 v1.6 Define keys (HO)
+))
+("C:\Program Files\MiKTeX 2.9\tex/latex/kvoptions\kvoptions.sty"
+Package: kvoptions 2019/11/29 v3.13 Key value format for package options (HO)
+
+("C:\Program Files\MiKTeX 2.9\tex/generic/ltxcmds\ltxcmds.sty"
+Package: ltxcmds 2019/12/15 v1.24 LaTeX kernel commands for general use (HO)
+)
+("C:\Program Files\MiKTeX 2.9\tex/generic/kvsetkeys\kvsetkeys.sty"
+Package: kvsetkeys 2019/12/15 v1.18 Key value parser (HO)
+))
+("C:\Program Files\MiKTeX 2.9\tex/latex/pdftexcmds\pdftexcmds.sty"
+Package: pdftexcmds 2019/11/24 v0.31 Utility functions of pdfTeX for LuaTeX (HO
+)
+
+("C:\Program Files\MiKTeX 2.9\tex/generic/iftex\iftex.sty"
+Package: iftex 2020/03/06 v1.0d TeX engine tests
+)
+Package pdftexcmds Info: \pdf@primitive is available.
+Package pdftexcmds Info: \pdf@ifprimitive is available.
+Package pdftexcmds Info: \pdfdraftmode found.
+)
+Package epstopdf-base Info: Redefining graphics rule for `.eps' on input line 4
+85.
+Package grfext Info: Graphics extension search list:
+(grfext)             [.pdf,.png,.jpg,.mps,.jpeg,.jbig2,.jb2,.PDF,.PNG,.JPG,.JPE
+G,.JBIG2,.JB2,.eps]
+(grfext)             \AppendGraphicsExtensions on input line 504.
+)
+Chapter 1.
+LaTeX Font Info:    Trying to load font information for U+msa on input line 8.
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsfonts\umsa.fd"
+File: umsa.fd 2013/01/14 v3.01 AMS symbols A
+)
+LaTeX Font Info:    Trying to load font information for U+msb on input line 8.
+
+("C:\Program Files\MiKTeX 2.9\tex/latex/amsfonts\umsb.fd"
+File: umsb.fd 2013/01/14 v3.01 AMS symbols B
+)
+<../img/lez24-img1.JPG, id=1, 176.91093pt x 86.57344pt>
+File: ../img/lez24-img1.JPG Graphic file (type jpg)
+<use ../img/lez24-img1.JPG>
+Package pdftex.def Info: ../img/lez24-img1.JPG  used on input line 21.
+(pdftex.def)             Requested size: 117.00119pt x 57.2562pt.
+<../img/lez24-img2.JPG, id=2, 224.33812pt x 121.20282pt>
+File: ../img/lez24-img2.JPG Graphic file (type jpg)
+<use ../img/lez24-img2.JPG>
+Package pdftex.def Info: ../img/lez24-img2.JPG  used on input line 28.
+(pdftex.def)             Requested size: 117.00119pt x 63.21075pt.
+ [1
+
+
+{C:/Users/AndreDany/AppData/Local/MiKTeX/2.9/pdftex/config/pdftex.map} <../img/
+lez24-img1.JPG> <../img/lez24-img2.JPG>]
+<../img/lez24-img3.JPG, id=18, 139.27031pt x 98.61844pt>
+File: ../img/lez24-img3.JPG Graphic file (type jpg)
+<use ../img/lez24-img3.JPG>
+Package pdftex.def Info: ../img/lez24-img3.JPG  used on input line 58.
+(pdftex.def)             Requested size: 117.00119pt x 82.85094pt.
+
+Underfull \hbox (badness 10000) in paragraph at lines 55--65
+
+ []
+
+<../img/lez24-img4.JPG, id=19, 143.03438pt x 113.67468pt>
+File: ../img/lez24-img4.JPG Graphic file (type jpg)
+<use ../img/lez24-img4.JPG>
+Package pdftex.def Info: ../img/lez24-img4.JPG  used on input line 70.
+(pdftex.def)             Requested size: 117.00119pt x 92.9866pt.
+<../img/lez24-img5.JPG, id=20, 336.50719pt x 115.93312pt>
+File: ../img/lez24-img5.JPG Graphic file (type jpg)
+<use ../img/lez24-img5.JPG>
+Package pdftex.def Info: ../img/lez24-img5.JPG  used on input line 76.
+(pdftex.def)             Requested size: 117.00119pt x 40.3083pt.
+
+Underfull \hbox (badness 10000) in paragraph at lines 67--80
+
+ []
+
+
+LaTeX Warning: `h' float specifier changed to `ht'.
+
+[2 <../img/lez24-img3.JPG> <../img/lez24-img4.JPG>]
+<../img/lez24-img6.JPG, id=24, 570.63188pt x 163.3603pt>
+File: ../img/lez24-img6.JPG Graphic file (type jpg)
+<use ../img/lez24-img6.JPG>
+Package pdftex.def Info: ../img/lez24-img6.JPG  used on input line 89.
+(pdftex.def)             Requested size: 195.0pt x 55.82343pt.
+
+Underfull \hbox (badness 10000) in paragraph at lines 86--93
+
+ []
+
+<../img/lez24-img8.JPG, id=25, 168.63pt x 105.39375pt>
+File: ../img/lez24-img8.JPG Graphic file (type jpg)
+<use ../img/lez24-img8.JPG>
+Package pdftex.def Info: ../img/lez24-img8.JPG  used on input line 106.
+(pdftex.def)             Requested size: 117.00119pt x 73.12701pt.
+
+Underfull \hbox (badness 10000) in paragraph at lines 102--110
+
+ []
+
+[3 <../img/lez24-img5.JPG> <../img/lez24-img6.JPG> <../img/lez24-img8.JPG>]
+<../img/lez24-img9.JPG, id=30, 215.30438pt x 96.36pt>
+File: ../img/lez24-img9.JPG Graphic file (type jpg)
+<use ../img/lez24-img9.JPG>
+Package pdftex.def Info: ../img/lez24-img9.JPG  used on input line 114.
+(pdftex.def)             Requested size: 117.00119pt x 52.36443pt.
+<../img/lez24-img10.JPG, id=31, 465.99094pt x 182.93344pt>
+File: ../img/lez24-img10.JPG Graphic file (type jpg)
+<use ../img/lez24-img10.JPG>
+Package pdftex.def Info: ../img/lez24-img10.JPG  used on input line 131.
+(pdftex.def)             Requested size: 195.0pt x 76.54959pt.
+
+[4 <../img/lez24-img9.JPG> <../img/lez24-img10.JPG>]
+<../img/lez24-img11.JPG, id=35, 240.9pt x 136.25906pt>
+File: ../img/lez24-img11.JPG Graphic file (type jpg)
+<use ../img/lez24-img11.JPG>
+Package pdftex.def Info: ../img/lez24-img11.JPG  used on input line 154.
+(pdftex.def)             Requested size: 195.0pt x 110.3006pt.
+
+Underfull \hbox (badness 10000) in paragraph at lines 151--158
+
+ []
+
+<../img/lez24-img12.JPG, id=36, 137.01187pt x 103.88812pt>
+File: ../img/lez24-img12.JPG Graphic file (type jpg)
+<use ../img/lez24-img12.JPG>
+Package pdftex.def Info: ../img/lez24-img12.JPG  used on input line 172.
+(pdftex.def)             Requested size: 117.00119pt x 88.71747pt.
+
+Underfull \hbox (badness 10000) in paragraph at lines 169--176
+
+ []
+
+[5 <../img/lez24-img11.JPG> <../img/lez24-img12.JPG>]
+<../img/lez24-img13.JPG, id=42, 252.945pt x 162.6075pt>
+File: ../img/lez24-img13.JPG Graphic file (type jpg)
+<use ../img/lez24-img13.JPG>
+Package pdftex.def Info: ../img/lez24-img13.JPG  used on input line 191.
+(pdftex.def)             Requested size: 155.99762pt x 100.28462pt.
+
+Underfull \hbox (badness 10000) in paragraph at lines 205--214
+
+ []
+
+[6 <../img/lez24-img13.JPG>]
+<../img/lez24-img14.JPG, id=48, 260.47313pt x 173.89969pt>
+File: ../img/lez24-img14.JPG Graphic file (type jpg)
+<use ../img/lez24-img14.JPG>
+Package pdftex.def Info: ../img/lez24-img14.JPG  used on input line 230.
+(pdftex.def)             Requested size: 155.99762pt x 104.14955pt.
+
+Underfull \hbox (badness 10000) in paragraph at lines 227--235
+
+ []
+
+[7 <../img/lez24-img14.JPG>]
+<../img/lez24-img15.JPG, id=55, 277.78781pt x 214.55156pt>
+File: ../img/lez24-img15.JPG Graphic file (type jpg)
+<use ../img/lez24-img15.JPG>
+Package pdftex.def Info: ../img/lez24-img15.JPG  used on input line 247.
+(pdftex.def)             Requested size: 155.99762pt x 120.48524pt.
+
+Underfull \hbox (badness 10000) in paragraph at lines 251--257
+
+ []
+
+
+Underfull \hbox (badness 10000) in paragraph at lines 251--257
+
+ []
+
+
+Underfull \hbox (badness 10000) in paragraph at lines 251--257
+
+ []
+
+
+Underfull \hbox (badness 10000) in paragraph at lines 261--264
+
+ []
+
+<../img/lez24-img16.JPG, id=56, 291.33844pt x 123.46124pt>
+File: ../img/lez24-img16.JPG Graphic file (type jpg)
+<use ../img/lez24-img16.JPG>
+Package pdftex.def Info: ../img/lez24-img16.JPG  used on input line 273.
+(pdftex.def)             Requested size: 155.99762pt x 66.10855pt.
+
+LaTeX Warning: `h' float specifier changed to `ht'.
+
+[8 <../img/lez24-img15.JPG>] [9 <../img/lez24-img16.JPG>] (lecture24.aux) ) 
+Here is how much of TeX's memory you used:
+ 5198 strings out of 480934
+ 71003 string characters out of 2909670
+ 331085 words of memory out of 3000000
+ 20932 multiletter control sequences out of 15000+200000
+ 548174 words of font info for 59 fonts, out of 3000000 for 9000
+ 1141 hyphenation exceptions out of 8191
+ 42i,7n,50p,333b,236s stack positions out of 5000i,500n,10000p,200000b,50000s
+ <C:
+\Users\AndreDany\AppData\Local\MiKTeX\2.9\fonts/pk/ljfour/jknappen/ec/dpi600\ec
+ti1200.pk> <C:\Users\AndreDany\AppData\Local\MiKTeX\2.9\fonts/pk/ljfour/jknappe
+n/ec/dpi600\ecbx1200.pk> <C:\Users\AndreDany\AppData\Local\MiKTeX\2.9\fonts/pk/
+ljfour/jknappen/ec/dpi600\ecrm1200.pk> <C:\Users\AndreDany\AppData\Local\MiKTeX
+\2.9\fonts/pk/ljfour/jknappen/ec/dpi600\ecbx1440.pk> <C:\Users\AndreDany\AppDat
+a\Local\MiKTeX\2.9\fonts/pk/ljfour/jknappen/ec/dpi600\ecbx1728.pk> <C:\Users\An
+dreDany\AppData\Local\MiKTeX\2.9\fonts/pk/ljfour/jknappen/ec/dpi600\ecbx2488.pk
+><C:/Program Files/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmex10.pfb><C:/Pro
+gram Files/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmmi12.pfb><C:/Program Fil
+es/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmmi6.pfb><C:/Program Files/MiKTeX
+ 2.9/fonts/type1/public/amsfonts/cm/cmmi8.pfb><C:/Program Files/MiKTeX 2.9/font
+s/type1/public/amsfonts/cm/cmr12.pfb><C:/Program Files/MiKTeX 2.9/fonts/type1/p
+ublic/amsfonts/cm/cmr6.pfb><C:/Program Files/MiKTeX 2.9/fonts/type1/public/amsf
+onts/cm/cmr8.pfb><C:/Program Files/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cm
+sy10.pfb><C:/Program Files/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmsy6.pfb>
+<C:/Program Files/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmsy8.pfb><C:/Progr
+am Files/MiKTeX 2.9/fonts/type1/public/amsfonts/symbols/msam10.pfb><C:/Program 
+Files/MiKTeX 2.9/fonts/type1/public/amsfonts/symbols/msbm10.pfb>
+Output written on lecture24.pdf (9 pages, 450062 bytes).
+PDF statistics:
+ 288 PDF objects out of 1000 (max. 8388607)
+ 0 named destinations out of 1000 (max. 500000)
+ 76 words of extra memory for PDF output out of 10000 (max. 10000000)
+

1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.tex

@@ -2,5 +2,281 @@
 \begin{document}
 
 \chapter{Lecture 24 - 09-06-2020}
+\section{Neural Networks}
 
+\subsection{Feedforward NN}
+$$
+f : \barra{R}^d \rightarrow \barra{R}^n
+$$
+Done with a combination of preditors.
+$$
+g(x) = \sigma(w^T \, x) \qquad \textbf{$\sigma$ is  a nonòinear activation function}
+$$
+Entire structure is a DAG:
+$$
+G= (V,E) \quad DAG \textit{  directed and acyclic graph}
+$$
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.3\linewidth]{../img/lez24-img1.JPG}
+    \caption{}
+    %\label{fig:}
+\end{figure}\\
+Multilayer NN (special case of Feedforward)
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.3\linewidth]{../img/lez24-img2.JPG}
+    \caption{}
+    %\label{fig:}
+\end{figure}\\
+Computing a function:
+$$ g(\cdot) = \sigma(w^T \cdot)$$
+where $\cdot$ is the input of the previous layer.
+\\ 
+We are going to split the nodes:
+$$
+V = V_m \cup V_{hid} \cup V_{out} \qquad |V_{in}| = d
+$$
+If $(i,j)$ is an edge:
+$$
+(i,j) \in E \Rightarrow w_{ij} \in \barra{R} \qquad \textit{ where $i$ and $j$ are nodes}
+$$
+$W$ weighted matrix:
+$$
+w_{ij}= 0 \quad if \quad (i,j) \not \in E
+$$
+We have then $G,w, \sigma$ parameters that define a function compute by the network:
+$$
+f_{G,w,\sigma} : \barra{R}^d \rightarrow \barra{R}^n
+$$
+Let's pick any node $j$ that $ j \in V \setminus V_{in}$:
+$$
+w(j) : \{ w_{ij} : (i,j)
+$$
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.3\linewidth]{../img/lez24-img3.JPG}
+    \caption{}
+    %\label{fig:}
+\end{figure}\\
+MANCAAAA
+\\\\
+Each node $j$ has a state value which is evaluated during the computation of the function.
+$$
+V_j = \sigma(w(j)^T v(j))
+$$
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.3\linewidth]{../img/lez24-img4.JPG}
+    \caption{}
+    %\label{fig:}
+\end{figure}\\
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.3\linewidth]{../img/lez24-img5.JPG}
+    \caption{}
+    %\label{fig:}
+\end{figure}\\
+\newpage
+$$
+V_i = \sigma(w(i)^T x) \qquad x \in \barra{R}^d
+$$
+$$
+f_{G,w,\sigma}(x) \qquad v_i = \sigma(w(j)^T v(j))
+$$
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.5\linewidth]{../img/lez24-img6.JPG}
+    \caption{}
+    %\label{fig:}
+\end{figure}\\
+$$
+f_{G,w,\sigma} (x) = (v_1,...,v_n) \qquad v1= \sigma (w(j)^T v(j))
+$$
+What are the parameters I learn?\\
+$G, \sigma$ fixed and $w$ is trained.
+\\
+I can define a class $F$:
+$$
+F_{G,\sigma} \textbf{class of all predictors $f_{G,w,\sigma}$ for $w$ variable}
+$$
+$\sigma$ is the identity and we can do other model like linear regression etc.
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.3\linewidth]{../img/lez24-img8.JPG}
+    \caption{}
+    %\label{fig:}
+\end{figure}\\
+\newpage
+\bred{Case of single output node:}
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.3\linewidth]{../img/lez24-img9.JPG}
+    \caption{}
+    %\label{fig:}
+\end{figure}\\
+In classification:
+$$
+v_0 = f_{G,w,\sigma}(x) = sgn (w(0)^T v(0))
+$$
+In regression:
+$$
+w(0)^T v(0)
+$$
+This depends on the problem. How do I choose the activation function $\sigma$?
+Could be a Sigmoid or Relu (0 in negative and 1 positive part) or other variant of this like Leaky Relu. \\
+The first one is bounded the other one are not.
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.5\linewidth]{../img/lez24-img10.JPG}
+    \caption{}
+    %\label{fig:}
+\end{figure}\\
+How many layers? 
+\\ How many nodes in each layer?
+\\ Patter connectivity between layers?
+\\ Activation function?
+\\
+In order to define our class $F_{G,\sigma}$:
+we are gonna ask ourself some structural question. How many layers do I need in general? Is there a minimum range of layers?
+\newpage
+
+\bred{Theorem}
+$$
+\forall \ d \ \exists \ G = (V,E) \textit{\ with $d+1$ input nodes, one hidden layer, one output node }
+$$
+$$ s.t \ F_{G,sgn} \textit{ contains all function of the form} f:\{-1,1\}^d \rightarrow \{-1,1\}
+$$
+
+\bred{Proof}
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.5\linewidth]{../img/lez24-img11.JPG}
+    \caption{}
+    %\label{fig:}
+\end{figure}\\
+$$
+f: \{-1,1\}^d \rightarrow \{-1,1\}
+$$
+$$
+\bar{x}_1,...,\bar{x}_N \qquad f(\bar{x}_i) = 1 \qquad N \leq 2^d
+$$
+The $i$-$th$ hidden node computes the function: $$g_i(x) = sgn(\bar{x}^T \bar{x}_i - d +1) \qquad \bar{x} \rightsquigarrow \hat{x}(\bar{x},1) 
+$$
+$$
+w(i) = (\bar{x}_i, -d+1) \qquad 
+w(i)^T \hat{x} = \bar{x}^T \bar{x}_i -d +1
+$$
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.3\linewidth]{../img/lez24-img12.JPG}
+    \caption{}
+    %\label{fig:}
+\end{figure}\\
+$$
+x^T x_i = \begin{cases}
+d \qquad \qquad \ \ if \ x=x_i
+\\
+\leq d-2 \qquad if \ x \neq x_i
+\end{cases}
+$$
+
+\newpage
+Let's see the output layer
+$$
+f(x) = g_1(x) \vee ... \vee g_N(x)
+$$
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.4\linewidth]{../img/lez24-img13.JPG}
+    \caption{}
+    %\label{fig:}
+\end{figure}\\
+The function compute by the output node is :
+$$
+sgn(w(0)^T v(0)) = \sum_{i=1}^N g_i(x) w_{i,0}  + N -1
+$$
+where $\sum_{i=1}^N g_i(x) w_{i,0}$ is equal to $-N +1$ to obtain $1$.
+$$
+x \ s.t \ f(x) = -1 \qquad g(x) = -1 \qquad i = 1,...,N
+$$
+$$
+x \ s.t. \ f(x) = 1 \qquad \exists i : g_i(x) = 1
+$$\\
+I can definetely use more than one layer since exponential is not good. Maybe you can use fewer node, still potentially able to generate a classifier.
+\\\\
+\bred{Theorem}\\
+$
+\forall d \in \barra{N}
+$ let $s(d)$ be the smaller integer \\ s.t. $\exists \ G = (V,E)$ with $|V| = s(d)$ \\ s.t. $F_{G,sgn}$ contains all functions $f: \{-1,1\}^d \rightarrow \{-1,1\}$ then $|V|= \Omega (2^{\frac{d}{3}})$
+\\
+If inputs are not binary we will use a sigmoid function. So 1 hidden layer is enought to approximate all function of the form:
+$$
+f: [-1,1]^d \rightarrow [-1,1]
+$$
+If you want to learn everything you need an exponential number of nodes.
+
+\newpage
+\section{Deep Learning}
+I have many hidden layers, not too many nodes in each layer. 
+\\
+I am choosing $G$ with a specific profile (tall and skinny network).
+
+$$
+F_{G_{dept},\sigma} \quad compared \ with \quad F_{G_{fat},\sigma}
+$$
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.4\linewidth]{../img/lez24-img14.JPG}
+    \caption{}
+    %\label{fig:}
+\end{figure}\\
+Usually, the first one should be the better one. \\
+$$ |V'| > |V|  \qquad 
+G \rightarrow G' \qquad F_{G,\sigma}
+$$
+Either increase size of 2 layer or add more layers:
+$$
+|F_{G', \sigma}| \gg |F_{G", \sigma}|
+$$
+where $G'$ are new layers and $G"$ are the fatter layers.
+\newpage
+\subsection{Convolution Neural Network}
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.4\linewidth]{../img/lez24-img15.JPG}
+    \caption{}
+    %\label{fig:}
+\end{figure}
+== MANCAAAA FOTO GiUSTA ====
+\\\\
+Using convolutional NN.
+We will recognise some shape (like presence of angles or simple geometric form obtained by combine the edge detector in the previous layers.
+If you have several layers you can have more more specific shapes combine the previous simpler layers.
+\\\\
+\subsection{General infos about Neural Networks}
+NN are trained using stochastic gradient descent.
+$$
+W \qquad w_{ij} \leftarrow w_{ij} - \eta_t \frac{\eth \ell_{z_t}(w)}{\eth w_{ij}} 
+$$
+$Z_t$ is randomly drawn from training set.
+\\
+$$
+\ell_t(w) = \ell(f_{G,w,\sigma}(x_t),y_t) \quad \forall (i,j) \in E
+$$
+We have $ \nabla \ell_t(w)$
+\\ The gradient component $\frac{\eth \ell_{z_t}(w)}{\eth w_{ij}} \quad (i,j) \in \sqrt[]{s}$ is computed using a technique called "error backpropagation"
+\\
+$\ell_t(w)$ is not convex!
+\begin{figure}[h]
+    \centering
+    \includegraphics[width=0.4\linewidth]{../img/lez24-img16.JPG}
+    \caption{}
+    %\label{fig:}
+\end{figure}
+Gradient descent will not terminate in: $$
+w^* = arg \min_w \hat{\ell}_s (w)
+$$
+\\
+\bred{AutoML to determine hyperparameter!}
 \end{document}