Introduction to Machine Learning - ML-chap1.pdf

Introduction to Machine Learning Chap I: Generic concepts Julien Donini LPC/Université Clermont Auvergne  Outline of the ML course Lectures Chap I: Basic concepts on ML Chap II:...

Introduction to Machine Learning Chap I: Generic concepts Julien Donini LPC/Université Clermont Auvergne  Outline of the ML course Lectures Chap I: Basic concepts on ML Chap II: Regression Chap III: Classification Practice sessions Introduction to Machine Learning Resources on Moodle: https://moodle2024.uca.fr/course/view.php?id=9405 Sections marked with (*) are more advanced → non examinable for DU students Julien Donini 2  This week Lectures Julien Donini 3  Buzzwords Machine Learning: statistics + computing + “learn” parameters from data Big Data: same as above + techniques to handle lots of data Artificial intelligence: same as above but sounds smarter Data Science: same as above but sounds more scientific Differentiable programming: Julien Donini 4  Warm up Julien Donini 5  What is Machine Learning Data Output weights y(x,w) = t Target Julien Donini 6  What is Machine Learning Data Output weights y(x,w) = t Target Examples X = {age, year, education, …} → t: income X = {image pixel values} → t: face recognition X = {list of words} → t: spam detection X = {E, p, …} → t: particle detection... Julien Donini 7  Machine Learning Basics Data Algorithm Objective (features x) y = f(x) (target t) Regression Julien Donini 8  Machine Learning Basics Data Algorithm Objective (features x) y = f(x) (target t) Classification Dog Julien Donini 9  Machine Learning Basics Data Algorithm Objective (features x) y = f(x) (target t) Training Julien Donini 10  Machine Learning Basics Data Algorithm Objective (features x) y = f(x) (target t) Test Julien Donini 11  Example: Neural Networks Data Objective (features x) (target t) y = f(x, W) Cost function “how close is the network output f : non-linear functions to the objective ?” W : parameters (weights) Julien Donini 12  Example: Neural Networks Data Objective (features x) (target t) Update of weights W Julien Donini 13  Machine Learning Not new ! Used since the 80’s in Physics: NN, then Boosted Decision Trees “era” Julien Donini - UCA/LPCA 14  Very active field nowadays Modern Machine learning since about 10-15 years → Breakthrough ideas supported by statisticians, computer scientists, etc → Increasing computing power to run efficiently complex algorithms → Fast execution of code (Graphics Processing Units,...) → ML librairies became much accessible Extremely large area of applications: industry, science,... Julien Donini – UCA/LPC 15  A decade of ML breakthrough VAE: Variational Autoencoders (arXiv:1312.6114) Julien Donini – UCA/LPC 16  A decade of ML breakthrough GAN: Generative Adversarial Networks (arXiv:1406.2661) Real images Random noise Julien Donini – UCA/LPC 17  A decade of ML breakthrough Transformers (arXiv:1706.03762) LLM (ChatGPT) Foundation models... Julien Donini – UCA/LPC 18  A decade of ML breakthrough DDPM: Denoising Diffusion Probabilistic Models (arXiv:2006.11239) Generative models inspired by Statistical Physics Julien Donini - UCA/LPC 19  A decade of ML breakthrough KAN: Kolmogorov-Arnold Networks (arXiv:2404.19756) Julien Donini - UCA/LPC 20  Great ! But let's not get carried away Yann Lecun Harvard lecture 28/03/24 Savannah Thais, EUCaifCon24 Julien Donini - UCA/LPC 21  Spiking Neural Networks (*) Particular type of artificial neural network : neurons and the synapses in the network are more inspired by biology One key difference: SNNs take into account timing in their operation. I will not discuss SNN in this course ! Julien Donini 22  Type of learning Julien Donini 23  Common type of learning Supervised Unsupervised Semi-supervised (labels are known) (no labels) (few labels) labels of class 1 labels of class 2 unknown class decision boundary Julien Donini 24  Unsupervised learning Unsupervised learning = no labels Clustering [bishop] Dimensionality reduction Julien Donini 25  Semi-supervised learning Semi-supervised learning = unlabelled data + few labels Example of the influence of unlabelled data in semi- supervised learning. The unlabelled data (grey dots) influence the separation of the two classes (decision surface) [wikipedia] Julien Donini 26  Reinforcement learning Model for agents that take actions depending on current state Action imply reward and modify state Learn to make best sequences of [kagan] decisions. Nature 529, 484–489 (28 January 2016) Julien Donini 27  Representation learning Representation is how we present the information (data) to solve a problem Example: what is the best number representation to perform this division ? Decimal 121 : 11 Roman CXXI / XI Binary 1111001 : 1011 Hex 79 : B ASCII y : VT Julien Donini 28  Representation learning Representation is how we present the information (data) to solve a problem Example: what are the best coordinates to separate this data ? [deeplearningbook] Julien Donini 29  Deep learning Class of ML algorithms (in general artificial neural networks) that use multiple layer to extract higher level features from raw data [deeplearningbook] Julien Donini 32  Deep learning Adding layers can help uncovering specific data patterns [Montufar, 1402.1869]: The absolute value activation function g(x1,x2) → |x1|,|x2| folds a 2D space twice. Each hidden layer of a deep neural network can be associated to a folding operator. The folding can identify symmetries in the boundaries that the NN can represent. Julien Donini 33  Deep learning Shallow Neural Networks (1 or 2 hidden layers) Deep Neural Network (more layers) Julien Donini 34  Deep Learning https://medium.com/analytics-vidhya/cnns-architectures-lenet-alexnet-vgg-googlenet-resnet-and-more-666091488df5 Julien Donini 35  Number of parameters: it use to be... [deeplearningbook] 1. Perceptron (Rosenblatt, 1958, 1962) 11. GPU-accelerated convolutional network (Chellapilla et al., 2006) 2. Adaptive linear element (Widrow and Hoff, 1960) 12. Deep Boltzmann machine (Salakhutdinov and Hinton, 2009a) 3. Neocognitron (Fukushima, 1980) 13. GPU-accelerated deep belief network (Raina et al., 2009) 4. Early back-propagation network (Rumelhart et al., 1986b) 14. Unsupervised convolutional network (Jarrett et al., 2009) 5. Recurrent neural network for speech recognition (Robinson and Fallside, 1991) 15. GPU-accelerated multilayer perceptron (Ciresan et al., 2010) 6. Multilayer perceptron for speech recognition (Bengio et al., 1991) 16. OMP-1 network (Coates and Ng, 2011) 7. Mean field sigmoid belief network (Saul et al., 1996) 17. Distributed autoencoder (Le et al., 2012) 8. LeNet-5 (LeCun et al., 1998b) 18. Multi-GPU convolutional network (Krizhevsky et al., 2012) 9. Echo state network (Jaeger and Haas, 2004) 19. COTS HPC unsupervised convolutional network (Coates et al., 2013) 10. Deep belief network (Hinton et al., 2006) 20. GoogLeNet (Szegedy et al., 2014a) Julien Donini 36  … before Transformers ! Julien Donini 37  References & Software Julien Donini 43  References (non exhaustive !) Classical Machine Learning textbooks Deep learning book, I. Goodfellow et al, http://www.deeplearningbook.org/ Elements of statistical learning (ESL), Hastie et al., Springer An Introduction to Statistical Learning (ISLR), Hastie et al. Springer Both books available online: http://web.stanford.edu/~hastie/pub.htm Pattern Recognition and Machine Learning, Bishop, Springer A *lot* of courses, lectures and tutorial on the web Online courses: DataCamp, Coursera, Andrew Ng (http://cs229.stanford.edu/) CERN lectures (ex: Kagan https://indico.cern.ch/event/619370) 2 recommended lectures: François Fleuret (EPFL) https://fleuret.org/ee559/ Gilles Louppe (University Liège)https://github.com/glouppe/info8010-deep-learning ML cheatsheet: https://ml-cheatsheet.readthedocs.io/en/latest/index.html Julien Donini 44  Scikit-learn (scikit-learn.org) Julien Donini 45  Deep Learning libraries www.tensorflow.org Keras.io Pytorch.org Julien Donini 46  Notebooks Jupyter notebooks: jupyter.org Install using Anaconda: www.anaconda.com Google colab https://colab.research.google.com/notebooks/intro.ipynb#recent=true Julien Donini 47  Resources Lectures and exercices on ENT https://moodle2024.uca.fr/course/view.php?id=9405 Practice sessions on local Linux computer cluster Setup ML environment: conda activate mlearning Julien Donini 48