diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img1.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img1.JPG new file mode 100644 index 000000000..2b42c3c55 Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img1.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img10.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img10.JPG new file mode 100644 index 000000000..d20c838a5 Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img10.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img11.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img11.JPG new file mode 100644 index 000000000..3fbaf899f Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img11.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img12.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img12.JPG new file mode 100644 index 000000000..61927dd2d Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img12.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img13.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img13.JPG new file mode 100644 index 000000000..0beeb6c9b Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img13.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img14.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img14.JPG new file mode 100644 index 000000000..e3532c93c Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img14.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img15.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img15.JPG new file mode 100644 index 000000000..0b8d61c6d Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img15.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img16.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img16.JPG new file mode 100644 index 000000000..e15e6ed19 Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img16.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img2.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img2.JPG new file mode 100644 index 000000000..edbfd3aa8 Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img2.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img3.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img3.JPG new file mode 100644 index 000000000..43946f699 Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img3.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img4.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img4.JPG new file mode 100644 index 000000000..cb8805b85 Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img4.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img5.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img5.JPG new file mode 100644 index 000000000..7c605d5f4 Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img5.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img6.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img6.JPG new file mode 100644 index 000000000..4198c2cf8 Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img6.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img7.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img7.JPG new file mode 100644 index 000000000..7df8becd8 Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img7.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img8.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img8.JPG new file mode 100644 index 000000000..3d03c033f Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img8.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img9.JPG b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img9.JPG new file mode 100644 index 000000000..f971f246d Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/img/lez24-img9.JPG differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.aux b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.aux new file mode 100644 index 000000000..dc668186d --- /dev/null +++ b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.aux @@ -0,0 +1,26 @@ +\relax +\@nameuse{bbl@beforestart} +\babel@aux{english}{} +\@writefile{toc}{\contentsline {chapter}{\numberline {1}Lecture 24 - 09-06-2020}{1}\protected@file@percent } +\@writefile{lof}{\addvspace {10\p@ }} +\@writefile{lot}{\addvspace {10\p@ }} +\@writefile{toc}{\contentsline {section}{\numberline {1.1}Neural Networks}{1}\protected@file@percent } +\@writefile{toc}{\contentsline {subsection}{\numberline {1.1.1}Feedforward NN}{1}\protected@file@percent } +\@writefile{lof}{\contentsline {figure}{\numberline {1.1}{\ignorespaces }}{1}\protected@file@percent } +\@writefile{lof}{\contentsline {figure}{\numberline {1.2}{\ignorespaces }}{1}\protected@file@percent } +\@writefile{lof}{\contentsline {figure}{\numberline {1.3}{\ignorespaces }}{2}\protected@file@percent } +\@writefile{lof}{\contentsline {figure}{\numberline {1.4}{\ignorespaces }}{2}\protected@file@percent } +\@writefile{lof}{\contentsline {figure}{\numberline {1.5}{\ignorespaces }}{3}\protected@file@percent } +\@writefile{lof}{\contentsline {figure}{\numberline {1.6}{\ignorespaces }}{3}\protected@file@percent } +\@writefile{lof}{\contentsline {figure}{\numberline {1.7}{\ignorespaces }}{3}\protected@file@percent } +\@writefile{lof}{\contentsline {figure}{\numberline {1.8}{\ignorespaces }}{4}\protected@file@percent } +\@writefile{lof}{\contentsline {figure}{\numberline {1.9}{\ignorespaces }}{4}\protected@file@percent } +\@writefile{lof}{\contentsline {figure}{\numberline {1.10}{\ignorespaces }}{5}\protected@file@percent } +\@writefile{lof}{\contentsline {figure}{\numberline {1.11}{\ignorespaces }}{5}\protected@file@percent } +\@writefile{lof}{\contentsline {figure}{\numberline {1.12}{\ignorespaces }}{6}\protected@file@percent } +\@writefile{toc}{\contentsline {section}{\numberline {1.2}Deep Learning}{7}\protected@file@percent } +\@writefile{lof}{\contentsline {figure}{\numberline {1.13}{\ignorespaces }}{7}\protected@file@percent } +\@writefile{toc}{\contentsline {subsection}{\numberline {1.2.1}Convolution Neural Network}{8}\protected@file@percent } +\@writefile{lof}{\contentsline {figure}{\numberline {1.14}{\ignorespaces }}{8}\protected@file@percent } +\@writefile{toc}{\contentsline {subsection}{\numberline {1.2.2}General infos about Neural Networks}{8}\protected@file@percent } +\@writefile{lof}{\contentsline {figure}{\numberline {1.15}{\ignorespaces }}{9}\protected@file@percent } diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.log b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.log new file mode 100644 index 000000000..cd8af1c16 --- /dev/null +++ b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.log @@ -0,0 +1,507 @@ +This is pdfTeX, Version 3.14159265-2.6-1.40.21 (MiKTeX 2.9.7300 64-bit) (preloaded format=pdflatex 2020.4.13) 9 JUN 2020 10:29 +entering extended mode +**./lecture24.tex +(lecture24.tex +LaTeX2e <2020-02-02> patch level 5 +L3 programming layer <2020-03-06> +("C:\Program Files\MiKTeX 2.9\tex/latex/subfiles\subfiles.cls" +Document Class: subfiles 2020/02/14 v1.6 Multi-file projects (class) +Preamble taken from file `../main.tex' +("C:\Program Files\MiKTeX 2.9\tex/latex/tools\verbatim.sty" +Package: verbatim 2019/11/10 v1.5r LaTeX2e package for verbatim enhancements +\every@verbatim=\toks15 +\verbatim@line=\toks16 +\verbatim@in@stream=\read2 +) +("C:\Program Files\MiKTeX 2.9\tex/latex/import\import.sty" +Package: import 2020/04/01 v 6.2 +) (../main.tex +("C:\Program Files\MiKTeX 2.9\tex/latex/base\report.cls" +Document Class: report 2019/12/20 v1.4l Standard LaTeX document class +("C:\Program Files\MiKTeX 2.9\tex/latex/base\size12.clo" +File: size12.clo 2019/12/20 v1.4l Standard LaTeX file (size option) +) +\c@part=\count167 +\c@chapter=\count168 +\c@section=\count169 +\c@subsection=\count170 +\c@subsubsection=\count171 +\c@paragraph=\count172 +\c@subparagraph=\count173 +\c@figure=\count174 +\c@table=\count175 +\abovecaptionskip=\skip47 +\belowcaptionskip=\skip48 +\bibindent=\dimen134 +) +("C:\Program Files\MiKTeX 2.9\tex/latex/amsmath\amsmath.sty" +Package: amsmath 2020/01/20 v2.17e AMS math features +\@mathmargin=\skip49 + +For additional information on amsmath, use the `?' option. +("C:\Program Files\MiKTeX 2.9\tex/latex/amsmath\amstext.sty" +Package: amstext 2000/06/29 v2.01 AMS text + +("C:\Program Files\MiKTeX 2.9\tex/latex/amsmath\amsgen.sty" +File: amsgen.sty 1999/11/30 v2.0 generic functions +\@emptytoks=\toks17 +\ex@=\dimen135 +)) +("C:\Program Files\MiKTeX 2.9\tex/latex/amsmath\amsbsy.sty" +Package: amsbsy 1999/11/29 v1.2d Bold Symbols +\pmbraise@=\dimen136 +) +("C:\Program Files\MiKTeX 2.9\tex/latex/amsmath\amsopn.sty" +Package: amsopn 2016/03/08 v2.02 operator names +) +\inf@bad=\count176 +LaTeX Info: Redefining \frac on input line 227. +\uproot@=\count177 +\leftroot@=\count178 +LaTeX Info: Redefining \overline on input line 389. +\classnum@=\count179 +\DOTSCASE@=\count180 +LaTeX Info: Redefining \ldots on input line 486. +LaTeX Info: Redefining \dots on input line 489. +LaTeX Info: Redefining \cdots on input line 610. +\Mathstrutbox@=\box45 +\strutbox@=\box46 +\big@size=\dimen137 +LaTeX Font Info: Redeclaring font encoding OML on input line 733. +LaTeX Font Info: Redeclaring font encoding OMS on input line 734. +\macc@depth=\count181 +\c@MaxMatrixCols=\count182 +\dotsspace@=\muskip16 +\c@parentequation=\count183 +\dspbrk@lvl=\count184 +\tag@help=\toks18 +\row@=\count185 +\column@=\count186 +\maxfields@=\count187 +\andhelp@=\toks19 +\eqnshift@=\dimen138 +\alignsep@=\dimen139 +\tagshift@=\dimen140 +\tagwidth@=\dimen141 +\totwidth@=\dimen142 +\lineht@=\dimen143 +\@envbody=\toks20 +\multlinegap=\skip50 +\multlinetaggap=\skip51 +\mathdisplay@stack=\toks21 +LaTeX Info: Redefining \[ on input line 2859. +LaTeX Info: Redefining \] on input line 2860. +) +("C:\Program Files\MiKTeX 2.9\tex/latex/systeme\systeme.sty" +("C:\Program Files\MiKTeX 2.9\tex/latex/xstring\xstring.sty" +("C:\Program Files\MiKTeX 2.9\tex/generic/xstring\xstring.tex" +\integerpart=\count188 +\decimalpart=\count189 +) +Package: xstring 2019/02/06 v1.83 String manipulations (CT) +) +("C:\Program Files\MiKTeX 2.9\tex/generic/systeme\systeme.tex" +\SYS_systemecode=\toks22 +\SYS_systempreamble=\toks23 +\SYSeqnum=\count190 +) +Package: systeme 2019/01/13 v0.32 Mise en forme de systemes d'equations (CT) +) +("C:\Program Files\MiKTeX 2.9\tex/latex/amsfonts\amssymb.sty" +Package: amssymb 2013/01/14 v3.01 AMS font symbols + +("C:\Program Files\MiKTeX 2.9\tex/latex/amsfonts\amsfonts.sty" +Package: amsfonts 2013/01/14 v3.01 Basic AMSFonts support +\symAMSa=\mathgroup4 +\symAMSb=\mathgroup5 +LaTeX Font Info: Redeclaring math symbol \hbar on input line 98. +LaTeX Font Info: Overwriting math alphabet `\mathfrak' in version `bold' +(Font) U/euf/m/n --> U/euf/b/n on input line 106. +)) +("C:\Program Files\MiKTeX 2.9\tex/latex/subfiles\subfiles.sty" +Package: subfiles 2020/02/14 v1.6 Multi-file projects (package) +) +("C:\Program Files\MiKTeX 2.9\tex/generic/babel\babel.sty" +Package: babel 2020/02/28 3.41 The Babel package + +("C:\Program Files\MiKTeX 2.9\tex/generic/babel\switch.def" +File: switch.def 2020/02/28 3.41 Babel switching mechanism +) +************************************* +* Local config file bblopts.cfg used +* +("C:\Program Files\MiKTeX 2.9\tex/latex/arabi\bblopts.cfg" +File: bblopts.cfg 2005/09/08 v0.1 add Arabic and Farsi to "declared" options of + babel +) +("C:\Program Files\MiKTeX 2.9\tex/latex/babel-english\english.ldf" +Language: english 2017/06/06 v3.3r English support from the babel system + +("C:\Program Files\MiKTeX 2.9\tex/generic/babel\babel.def" +File: babel.def 2020/02/28 3.41 Babel common definitions +\babel@savecnt=\count191 +\U@D=\dimen144 + +("C:\Program Files\MiKTeX 2.9\tex/generic/babel\txtbabel.def") +\bbl@readstream=\read3 +\bbl@dirlevel=\count192 +) +Package babel Info: \l@canadian = using hyphenrules for english +(babel) (\language0) on input line 102. +Package babel Info: \l@australian = using hyphenrules for ukenglish +(babel) (\language72) on input line 105. +Package babel Info: \l@newzealand = using hyphenrules for ukenglish +(babel) (\language72) on input line 108. +)) +("C:\Program Files\MiKTeX 2.9\tex/latex/xcolor\xcolor.sty" +Package: xcolor 2016/05/11 v2.12 LaTeX color extensions (UK) + +("C:\Program Files\MiKTeX 2.9\tex/latex/graphics-cfg\color.cfg" +File: color.cfg 2016/01/02 v1.6 sample color configuration +) +Package xcolor Info: Driver file: pdftex.def on input line 225. + +("C:\Program Files\MiKTeX 2.9\tex/latex/graphics-def\pdftex.def" +File: pdftex.def 2018/01/08 v1.0l Graphics/color driver for pdftex +) +Package xcolor Info: Model `cmy' substituted by `cmy0' on input line 1348. +Package xcolor Info: Model `hsb' substituted by `rgb' on input line 1352. +Package xcolor Info: Model `RGB' extended on input line 1364. +Package xcolor Info: Model `HTML' substituted by `rgb' on input line 1366. +Package xcolor Info: Model `Hsb' substituted by `hsb' on input line 1367. +Package xcolor Info: Model `tHsb' substituted by `hsb' on input line 1368. +Package xcolor Info: Model `HSB' substituted by `hsb' on input line 1369. +Package xcolor Info: Model `Gray' substituted by `gray' on input line 1370. +Package xcolor Info: Model `wave' substituted by `hsb' on input line 1371. + +("C:\Program Files\MiKTeX 2.9\tex/latex/graphics\dvipsnam.def" +File: dvipsnam.def 2016/06/17 v3.0m Driver-dependent file (DPC,SPQR) +)) +("C:\Program Files\MiKTeX 2.9\tex/latex/graphics\graphicx.sty" +Package: graphicx 2019/11/30 v1.2a Enhanced LaTeX Graphics (DPC,SPQR) + +("C:\Program Files\MiKTeX 2.9\tex/latex/graphics\keyval.sty" +Package: keyval 2014/10/28 v1.15 key=value parser (DPC) +\KV@toks@=\toks24 +) +("C:\Program Files\MiKTeX 2.9\tex/latex/graphics\graphics.sty" +Package: graphics 2019/11/30 v1.4a Standard LaTeX Graphics (DPC,SPQR) + +("C:\Program Files\MiKTeX 2.9\tex/latex/graphics\trig.sty" +Package: trig 2016/01/03 v1.10 sin cos tan (DPC) +) +("C:\Program Files\MiKTeX 2.9\tex/latex/graphics-cfg\graphics.cfg" +File: graphics.cfg 2016/06/04 v1.11 sample graphics configuration +) +Package graphics Info: Driver file: pdftex.def on input line 105. +) +\Gin@req@height=\dimen145 +\Gin@req@width=\dimen146 +) +("C:\Program Files\MiKTeX 2.9\tex/latex/sectsty\sectsty.sty" +Package: sectsty 2002/02/25 v2.0.2 Commands to change all sectional heading sty +les + + +LaTeX Warning: Command \underbar has changed. + Check if current package is valid. + + +LaTeX Warning: Command \underline has changed. + Check if current package is valid. + +) ("C:\Program Files\MiKTeX 2.9\tex/latex/framed\framed.sty" +Package: framed 2011/10/22 v 0.96: framed or shaded text with page breaks +\OuterFrameSep=\skip52 +\fb@frw=\dimen147 +\fb@frh=\dimen148 +\FrameRule=\dimen149 +\FrameSep=\dimen150 +) +("C:\Program Files\MiKTeX 2.9\tex/latex/titlesec\titlesec.sty" +Package: titlesec 2019/10/16 v2.13 Sectioning titles +\ttl@box=\box47 +\beforetitleunit=\skip53 +\aftertitleunit=\skip54 +\ttl@plus=\dimen151 +\ttl@minus=\dimen152 +\ttl@toksa=\toks25 +\titlewidth=\dimen153 +\titlewidthlast=\dimen154 +\titlewidthfirst=\dimen155 +) +("C:\Program Files\MiKTeX 2.9\tex/latex/base\fontenc.sty" +Package: fontenc 2020/02/11 v2.0o Standard LaTeX package +))) +("C:\Program Files\MiKTeX 2.9\tex/latex/l3backend\l3backend-pdfmode.def" +File: l3backend-pdfmode.def 2020-03-12 L3 backend support: PDF mode +\l__kernel_color_stack_int=\count193 +\l__pdf_internal_box=\box48 +) +(lecture24.aux) +\openout1 = `lecture24.aux'. + +LaTeX Font Info: Checking defaults for OML/cmm/m/it on input line 2. +LaTeX Font Info: ... okay on input line 2. +LaTeX Font Info: Checking defaults for OMS/cmsy/m/n on input line 2. +LaTeX Font Info: ... okay on input line 2. +LaTeX Font Info: Checking defaults for OT1/cmr/m/n on input line 2. +LaTeX Font Info: ... okay on input line 2. +LaTeX Font Info: Checking defaults for T1/cmr/m/n on input line 2. +LaTeX Font Info: ... okay on input line 2. +LaTeX Font Info: Checking defaults for TS1/cmr/m/n on input line 2. +LaTeX Font Info: ... okay on input line 2. +LaTeX Font Info: Checking defaults for OMX/cmex/m/n on input line 2. +LaTeX Font Info: ... okay on input line 2. +LaTeX Font Info: Checking defaults for U/cmr/m/n on input line 2. +LaTeX Font Info: ... okay on input line 2. + +("C:\Program Files\MiKTeX 2.9\tex/context/base/mkii\supp-pdf.mkii" +[Loading MPS to PDF converter (version 2006.09.02).] +\scratchcounter=\count194 +\scratchdimen=\dimen156 +\scratchbox=\box49 +\nofMPsegments=\count195 +\nofMParguments=\count196 +\everyMPshowfont=\toks26 +\MPscratchCnt=\count197 +\MPscratchDim=\dimen157 +\MPnumerator=\count198 +\makeMPintoPDFobject=\count199 +\everyMPtoPDFconversion=\toks27 +) ("C:\Program Files\MiKTeX 2.9\tex/latex/epstopdf-pkg\epstopdf-base.sty" +Package: epstopdf-base 2020-01-24 v2.11 Base part for package epstopdf + +("C:\Program Files\MiKTeX 2.9\tex/generic/infwarerr\infwarerr.sty" +Package: infwarerr 2019/12/03 v1.5 Providing info/warning/error messages (HO) +) +("C:\Program Files\MiKTeX 2.9\tex/latex/grfext\grfext.sty" +Package: grfext 2019/12/03 v1.3 Manage graphics extensions (HO) + +("C:\Program Files\MiKTeX 2.9\tex/generic/kvdefinekeys\kvdefinekeys.sty" +Package: kvdefinekeys 2019-12-19 v1.6 Define keys (HO) +)) +("C:\Program Files\MiKTeX 2.9\tex/latex/kvoptions\kvoptions.sty" +Package: kvoptions 2019/11/29 v3.13 Key value format for package options (HO) + +("C:\Program Files\MiKTeX 2.9\tex/generic/ltxcmds\ltxcmds.sty" +Package: ltxcmds 2019/12/15 v1.24 LaTeX kernel commands for general use (HO) +) +("C:\Program Files\MiKTeX 2.9\tex/generic/kvsetkeys\kvsetkeys.sty" +Package: kvsetkeys 2019/12/15 v1.18 Key value parser (HO) +)) +("C:\Program Files\MiKTeX 2.9\tex/latex/pdftexcmds\pdftexcmds.sty" +Package: pdftexcmds 2019/11/24 v0.31 Utility functions of pdfTeX for LuaTeX (HO +) + +("C:\Program Files\MiKTeX 2.9\tex/generic/iftex\iftex.sty" +Package: iftex 2020/03/06 v1.0d TeX engine tests +) +Package pdftexcmds Info: \pdf@primitive is available. +Package pdftexcmds Info: \pdf@ifprimitive is available. +Package pdftexcmds Info: \pdfdraftmode found. +) +Package epstopdf-base Info: Redefining graphics rule for `.eps' on input line 4 +85. +Package grfext Info: Graphics extension search list: +(grfext) [.pdf,.png,.jpg,.mps,.jpeg,.jbig2,.jb2,.PDF,.PNG,.JPG,.JPE +G,.JBIG2,.JB2,.eps] +(grfext) \AppendGraphicsExtensions on input line 504. +) +Chapter 1. +LaTeX Font Info: Trying to load font information for U+msa on input line 8. +("C:\Program Files\MiKTeX 2.9\tex/latex/amsfonts\umsa.fd" +File: umsa.fd 2013/01/14 v3.01 AMS symbols A +) +LaTeX Font Info: Trying to load font information for U+msb on input line 8. + +("C:\Program Files\MiKTeX 2.9\tex/latex/amsfonts\umsb.fd" +File: umsb.fd 2013/01/14 v3.01 AMS symbols B +) +<../img/lez24-img1.JPG, id=1, 176.91093pt x 86.57344pt> +File: ../img/lez24-img1.JPG Graphic file (type jpg) + +Package pdftex.def Info: ../img/lez24-img1.JPG used on input line 21. +(pdftex.def) Requested size: 117.00119pt x 57.2562pt. +<../img/lez24-img2.JPG, id=2, 224.33812pt x 121.20282pt> +File: ../img/lez24-img2.JPG Graphic file (type jpg) + +Package pdftex.def Info: ../img/lez24-img2.JPG used on input line 28. +(pdftex.def) Requested size: 117.00119pt x 63.21075pt. + [1 + + +{C:/Users/AndreDany/AppData/Local/MiKTeX/2.9/pdftex/config/pdftex.map} <../img/ +lez24-img1.JPG> <../img/lez24-img2.JPG>] +<../img/lez24-img3.JPG, id=18, 139.27031pt x 98.61844pt> +File: ../img/lez24-img3.JPG Graphic file (type jpg) + +Package pdftex.def Info: ../img/lez24-img3.JPG used on input line 58. +(pdftex.def) Requested size: 117.00119pt x 82.85094pt. + +Underfull \hbox (badness 10000) in paragraph at lines 55--65 + + [] + +<../img/lez24-img4.JPG, id=19, 143.03438pt x 113.67468pt> +File: ../img/lez24-img4.JPG Graphic file (type jpg) + +Package pdftex.def Info: ../img/lez24-img4.JPG used on input line 70. +(pdftex.def) Requested size: 117.00119pt x 92.9866pt. +<../img/lez24-img5.JPG, id=20, 336.50719pt x 115.93312pt> +File: ../img/lez24-img5.JPG Graphic file (type jpg) + +Package pdftex.def Info: ../img/lez24-img5.JPG used on input line 76. +(pdftex.def) Requested size: 117.00119pt x 40.3083pt. + +Underfull \hbox (badness 10000) in paragraph at lines 67--80 + + [] + + +LaTeX Warning: `h' float specifier changed to `ht'. + +[2 <../img/lez24-img3.JPG> <../img/lez24-img4.JPG>] +<../img/lez24-img6.JPG, id=24, 570.63188pt x 163.3603pt> +File: ../img/lez24-img6.JPG Graphic file (type jpg) + +Package pdftex.def Info: ../img/lez24-img6.JPG used on input line 89. +(pdftex.def) Requested size: 195.0pt x 55.82343pt. + +Underfull \hbox (badness 10000) in paragraph at lines 86--93 + + [] + +<../img/lez24-img8.JPG, id=25, 168.63pt x 105.39375pt> +File: ../img/lez24-img8.JPG Graphic file (type jpg) + +Package pdftex.def Info: ../img/lez24-img8.JPG used on input line 106. +(pdftex.def) Requested size: 117.00119pt x 73.12701pt. + +Underfull \hbox (badness 10000) in paragraph at lines 102--110 + + [] + +[3 <../img/lez24-img5.JPG> <../img/lez24-img6.JPG> <../img/lez24-img8.JPG>] +<../img/lez24-img9.JPG, id=30, 215.30438pt x 96.36pt> +File: ../img/lez24-img9.JPG Graphic file (type jpg) + +Package pdftex.def Info: ../img/lez24-img9.JPG used on input line 114. +(pdftex.def) Requested size: 117.00119pt x 52.36443pt. +<../img/lez24-img10.JPG, id=31, 465.99094pt x 182.93344pt> +File: ../img/lez24-img10.JPG Graphic file (type jpg) + +Package pdftex.def Info: ../img/lez24-img10.JPG used on input line 131. +(pdftex.def) Requested size: 195.0pt x 76.54959pt. + +[4 <../img/lez24-img9.JPG> <../img/lez24-img10.JPG>] +<../img/lez24-img11.JPG, id=35, 240.9pt x 136.25906pt> +File: ../img/lez24-img11.JPG Graphic file (type jpg) + +Package pdftex.def Info: ../img/lez24-img11.JPG used on input line 154. +(pdftex.def) Requested size: 195.0pt x 110.3006pt. + +Underfull \hbox (badness 10000) in paragraph at lines 151--158 + + [] + +<../img/lez24-img12.JPG, id=36, 137.01187pt x 103.88812pt> +File: ../img/lez24-img12.JPG Graphic file (type jpg) + +Package pdftex.def Info: ../img/lez24-img12.JPG used on input line 172. +(pdftex.def) Requested size: 117.00119pt x 88.71747pt. + +Underfull \hbox (badness 10000) in paragraph at lines 169--176 + + [] + +[5 <../img/lez24-img11.JPG> <../img/lez24-img12.JPG>] +<../img/lez24-img13.JPG, id=42, 252.945pt x 162.6075pt> +File: ../img/lez24-img13.JPG Graphic file (type jpg) + +Package pdftex.def Info: ../img/lez24-img13.JPG used on input line 191. +(pdftex.def) Requested size: 155.99762pt x 100.28462pt. + +Underfull \hbox (badness 10000) in paragraph at lines 205--214 + + [] + +[6 <../img/lez24-img13.JPG>] +<../img/lez24-img14.JPG, id=48, 260.47313pt x 173.89969pt> +File: ../img/lez24-img14.JPG Graphic file (type jpg) + +Package pdftex.def Info: ../img/lez24-img14.JPG used on input line 230. +(pdftex.def) Requested size: 155.99762pt x 104.14955pt. + +Underfull \hbox (badness 10000) in paragraph at lines 227--235 + + [] + +[7 <../img/lez24-img14.JPG>] +<../img/lez24-img15.JPG, id=55, 277.78781pt x 214.55156pt> +File: ../img/lez24-img15.JPG Graphic file (type jpg) + +Package pdftex.def Info: ../img/lez24-img15.JPG used on input line 247. +(pdftex.def) Requested size: 155.99762pt x 120.48524pt. + +Underfull \hbox (badness 10000) in paragraph at lines 251--257 + + [] + + +Underfull \hbox (badness 10000) in paragraph at lines 251--257 + + [] + + +Underfull \hbox (badness 10000) in paragraph at lines 251--257 + + [] + + +Underfull \hbox (badness 10000) in paragraph at lines 261--264 + + [] + +<../img/lez24-img16.JPG, id=56, 291.33844pt x 123.46124pt> +File: ../img/lez24-img16.JPG Graphic file (type jpg) + +Package pdftex.def Info: ../img/lez24-img16.JPG used on input line 273. +(pdftex.def) Requested size: 155.99762pt x 66.10855pt. + +LaTeX Warning: `h' float specifier changed to `ht'. + +[8 <../img/lez24-img15.JPG>] [9 <../img/lez24-img16.JPG>] (lecture24.aux) ) +Here is how much of TeX's memory you used: + 5198 strings out of 480934 + 71003 string characters out of 2909670 + 331085 words of memory out of 3000000 + 20932 multiletter control sequences out of 15000+200000 + 548174 words of font info for 59 fonts, out of 3000000 for 9000 + 1141 hyphenation exceptions out of 8191 + 42i,7n,50p,333b,236s stack positions out of 5000i,500n,10000p,200000b,50000s + + +Output written on lecture24.pdf (9 pages, 450062 bytes). +PDF statistics: + 288 PDF objects out of 1000 (max. 8388607) + 0 named destinations out of 1000 (max. 500000) + 76 words of extra memory for PDF output out of 10000 (max. 10000000) + diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.pdf b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.pdf new file mode 100644 index 000000000..61d0d8153 Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.pdf differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.synctex.gz b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.synctex.gz new file mode 100644 index 000000000..aa06c6ea2 Binary files /dev/null and b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.synctex.gz differ diff --git a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.tex b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.tex index be76522b4..77ebfe2b1 100644 --- a/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.tex +++ b/1year/3trimester/Machine Learning, Statistical Learning, Deep Learning and Artificial Intelligence/Machine Learning/Notes/lectures/lecture24.tex @@ -2,5 +2,281 @@ \begin{document} \chapter{Lecture 24 - 09-06-2020} +\section{Neural Networks} +\subsection{Feedforward NN} +$$ +f : \barra{R}^d \rightarrow \barra{R}^n +$$ +Done with a combination of preditors. +$$ +g(x) = \sigma(w^T \, x) \qquad \textbf{$\sigma$ is a nonòinear activation function} +$$ +Entire structure is a DAG: +$$ +G= (V,E) \quad DAG \textit{ directed and acyclic graph} +$$ +\begin{figure}[h] + \centering + \includegraphics[width=0.3\linewidth]{../img/lez24-img1.JPG} + \caption{} + %\label{fig:} +\end{figure}\\ +Multilayer NN (special case of Feedforward) +\begin{figure}[h] + \centering + \includegraphics[width=0.3\linewidth]{../img/lez24-img2.JPG} + \caption{} + %\label{fig:} +\end{figure}\\ +Computing a function: +$$ g(\cdot) = \sigma(w^T \cdot)$$ +where $\cdot$ is the input of the previous layer. +\\ +We are going to split the nodes: +$$ +V = V_m \cup V_{hid} \cup V_{out} \qquad |V_{in}| = d +$$ +If $(i,j)$ is an edge: +$$ +(i,j) \in E \Rightarrow w_{ij} \in \barra{R} \qquad \textit{ where $i$ and $j$ are nodes} +$$ +$W$ weighted matrix: +$$ +w_{ij}= 0 \quad if \quad (i,j) \not \in E +$$ +We have then $G,w, \sigma$ parameters that define a function compute by the network: +$$ +f_{G,w,\sigma} : \barra{R}^d \rightarrow \barra{R}^n +$$ +Let's pick any node $j$ that $ j \in V \setminus V_{in}$: +$$ +w(j) : \{ w_{ij} : (i,j) +$$ +\begin{figure}[h] + \centering + \includegraphics[width=0.3\linewidth]{../img/lez24-img3.JPG} + \caption{} + %\label{fig:} +\end{figure}\\ +MANCAAAA +\\\\ +Each node $j$ has a state value which is evaluated during the computation of the function. +$$ +V_j = \sigma(w(j)^T v(j)) +$$ +\begin{figure}[h] + \centering + \includegraphics[width=0.3\linewidth]{../img/lez24-img4.JPG} + \caption{} + %\label{fig:} +\end{figure}\\ +\begin{figure}[h] + \centering + \includegraphics[width=0.3\linewidth]{../img/lez24-img5.JPG} + \caption{} + %\label{fig:} +\end{figure}\\ +\newpage +$$ +V_i = \sigma(w(i)^T x) \qquad x \in \barra{R}^d +$$ +$$ +f_{G,w,\sigma}(x) \qquad v_i = \sigma(w(j)^T v(j)) +$$ +\begin{figure}[h] + \centering + \includegraphics[width=0.5\linewidth]{../img/lez24-img6.JPG} + \caption{} + %\label{fig:} +\end{figure}\\ +$$ +f_{G,w,\sigma} (x) = (v_1,...,v_n) \qquad v1= \sigma (w(j)^T v(j)) +$$ +What are the parameters I learn?\\ +$G, \sigma$ fixed and $w$ is trained. +\\ +I can define a class $F$: +$$ +F_{G,\sigma} \textbf{class of all predictors $f_{G,w,\sigma}$ for $w$ variable} +$$ +$\sigma$ is the identity and we can do other model like linear regression etc. +\begin{figure}[h] + \centering + \includegraphics[width=0.3\linewidth]{../img/lez24-img8.JPG} + \caption{} + %\label{fig:} +\end{figure}\\ +\newpage +\bred{Case of single output node:} +\begin{figure}[h] + \centering + \includegraphics[width=0.3\linewidth]{../img/lez24-img9.JPG} + \caption{} + %\label{fig:} +\end{figure}\\ +In classification: +$$ +v_0 = f_{G,w,\sigma}(x) = sgn (w(0)^T v(0)) +$$ +In regression: +$$ +w(0)^T v(0) +$$ +This depends on the problem. How do I choose the activation function $\sigma$? +Could be a Sigmoid or Relu (0 in negative and 1 positive part) or other variant of this like Leaky Relu. \\ +The first one is bounded the other one are not. +\begin{figure}[h] + \centering + \includegraphics[width=0.5\linewidth]{../img/lez24-img10.JPG} + \caption{} + %\label{fig:} +\end{figure}\\ +How many layers? +\\ How many nodes in each layer? +\\ Patter connectivity between layers? +\\ Activation function? +\\ +In order to define our class $F_{G,\sigma}$: +we are gonna ask ourself some structural question. How many layers do I need in general? Is there a minimum range of layers? +\newpage + +\bred{Theorem} +$$ +\forall \ d \ \exists \ G = (V,E) \textit{\ with $d+1$ input nodes, one hidden layer, one output node } +$$ +$$ s.t \ F_{G,sgn} \textit{ contains all function of the form} f:\{-1,1\}^d \rightarrow \{-1,1\} +$$ + +\bred{Proof} +\begin{figure}[h] + \centering + \includegraphics[width=0.5\linewidth]{../img/lez24-img11.JPG} + \caption{} + %\label{fig:} +\end{figure}\\ +$$ +f: \{-1,1\}^d \rightarrow \{-1,1\} +$$ +$$ +\bar{x}_1,...,\bar{x}_N \qquad f(\bar{x}_i) = 1 \qquad N \leq 2^d +$$ +The $i$-$th$ hidden node computes the function: $$g_i(x) = sgn(\bar{x}^T \bar{x}_i - d +1) \qquad \bar{x} \rightsquigarrow \hat{x}(\bar{x},1) +$$ +$$ +w(i) = (\bar{x}_i, -d+1) \qquad +w(i)^T \hat{x} = \bar{x}^T \bar{x}_i -d +1 +$$ +\begin{figure}[h] + \centering + \includegraphics[width=0.3\linewidth]{../img/lez24-img12.JPG} + \caption{} + %\label{fig:} +\end{figure}\\ +$$ +x^T x_i = \begin{cases} +d \qquad \qquad \ \ if \ x=x_i +\\ +\leq d-2 \qquad if \ x \neq x_i +\end{cases} +$$ + +\newpage +Let's see the output layer +$$ +f(x) = g_1(x) \vee ... \vee g_N(x) +$$ +\begin{figure}[h] + \centering + \includegraphics[width=0.4\linewidth]{../img/lez24-img13.JPG} + \caption{} + %\label{fig:} +\end{figure}\\ +The function compute by the output node is : +$$ +sgn(w(0)^T v(0)) = \sum_{i=1}^N g_i(x) w_{i,0} + N -1 +$$ +where $\sum_{i=1}^N g_i(x) w_{i,0}$ is equal to $-N +1$ to obtain $1$. +$$ +x \ s.t \ f(x) = -1 \qquad g(x) = -1 \qquad i = 1,...,N +$$ +$$ +x \ s.t. \ f(x) = 1 \qquad \exists i : g_i(x) = 1 +$$\\ +I can definetely use more than one layer since exponential is not good. Maybe you can use fewer node, still potentially able to generate a classifier. +\\\\ +\bred{Theorem}\\ +$ +\forall d \in \barra{N} +$ let $s(d)$ be the smaller integer \\ s.t. $\exists \ G = (V,E)$ with $|V| = s(d)$ \\ s.t. $F_{G,sgn}$ contains all functions $f: \{-1,1\}^d \rightarrow \{-1,1\}$ then $|V|= \Omega (2^{\frac{d}{3}})$ +\\ +If inputs are not binary we will use a sigmoid function. So 1 hidden layer is enought to approximate all function of the form: +$$ +f: [-1,1]^d \rightarrow [-1,1] +$$ +If you want to learn everything you need an exponential number of nodes. + +\newpage +\section{Deep Learning} +I have many hidden layers, not too many nodes in each layer. +\\ +I am choosing $G$ with a specific profile (tall and skinny network). + +$$ +F_{G_{dept},\sigma} \quad compared \ with \quad F_{G_{fat},\sigma} +$$ +\begin{figure}[h] + \centering + \includegraphics[width=0.4\linewidth]{../img/lez24-img14.JPG} + \caption{} + %\label{fig:} +\end{figure}\\ +Usually, the first one should be the better one. \\ +$$ |V'| > |V| \qquad +G \rightarrow G' \qquad F_{G,\sigma} +$$ +Either increase size of 2 layer or add more layers: +$$ +|F_{G', \sigma}| \gg |F_{G", \sigma}| +$$ +where $G'$ are new layers and $G"$ are the fatter layers. +\newpage +\subsection{Convolution Neural Network} +\begin{figure}[h] + \centering + \includegraphics[width=0.4\linewidth]{../img/lez24-img15.JPG} + \caption{} + %\label{fig:} +\end{figure} +== MANCAAAA FOTO GiUSTA ==== +\\\\ +Using convolutional NN. +We will recognise some shape (like presence of angles or simple geometric form obtained by combine the edge detector in the previous layers. +If you have several layers you can have more more specific shapes combine the previous simpler layers. +\\\\ +\subsection{General infos about Neural Networks} +NN are trained using stochastic gradient descent. +$$ +W \qquad w_{ij} \leftarrow w_{ij} - \eta_t \frac{\eth \ell_{z_t}(w)}{\eth w_{ij}} +$$ +$Z_t$ is randomly drawn from training set. +\\ +$$ +\ell_t(w) = \ell(f_{G,w,\sigma}(x_t),y_t) \quad \forall (i,j) \in E +$$ +We have $ \nabla \ell_t(w)$ +\\ The gradient component $\frac{\eth \ell_{z_t}(w)}{\eth w_{ij}} \quad (i,j) \in \sqrt[]{s}$ is computed using a technique called "error backpropagation" +\\ +$\ell_t(w)$ is not convex! +\begin{figure}[h] + \centering + \includegraphics[width=0.4\linewidth]{../img/lez24-img16.JPG} + \caption{} + %\label{fig:} +\end{figure} +Gradient descent will not terminate in: $$ +w^* = arg \min_w \hat{\ell}_s (w) +$$ +\\ +\bred{AutoML to determine hyperparameter!} \end{document} \ No newline at end of file