Week 15-16 Lecture Notes on AI, Deep Learning, and Machine Learning PDF

Lecture 1: Introduction to AI, deep learning and machine learning  What is artificial intelligence? Artificial intelligence is the ability of a computer to perform tasks commonly associated with intelligent beings.  What is machine learning? Machine learning is the study of algorithms...

Lecture 1: Introduction to AI, deep learning and machine learning  What is artificial intelligence? Artificial intelligence is the ability of a computer to perform tasks commonly associated with intelligent beings.  What is machine learning? Machine learning is the study of algorithms that learn from examples and experience instead of relying on hard-coded rules and make predictions on new data.  What is deep learning? Deep learning is a subfield of machine learning focusing on learning data representations as successive layers of increasingly meaningful representations. Image from https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/ “Traditional” machine learning: handcrafted learned cat features classifier Deep, “end-to-end” learning: learned learned learned learned low-level mid-level high-level cat classifier features features features From: Wang & Raj: On the Origin of Deep Learning (2017) Main types of machine learning Main types of machine learning Supervised learning cat Unsupervised learning dog Self-supervised learning Reinforcement learning Main types of machine learning Supervised learning Unsupervised learning Self-supervised learning Reinforcement learning Main types of machine learning Supervised learning Unsupervised learning Self-supervised learning Reinforcement learning Image from https://arxiv.org/abs/1710.10196 Main types of machine learning Supervised learning Unsupervised learning Self-supervised learning Reinforcement learning Animation from https://yanpanlau.github.io/2016/07/10/FlappyBird-Keras.html Fundamentals of machine learning Data Humans learn by observation and unsupervised learning model of the world / common sense reasoning Machine learning needs lots of (labeled) data to compensate Data Tensors: generalization of matrices to n dimensions (or rank, order, degree) 1D tensor: vector 2D tensor: matrix 3D, 4D, 5D tensors numpy.ndarray(shape, dtype) Training – validation – test split (+ adversarial test) Minibatches small sets of input data used at a time usually processed independently Image from: https://arxiv.org/abs/1707.08945 Optimization Image from: Li et al. “Visualizing the Loss Landscape of Neural Nets”, arXiv:1712.09913 Deep learning Anatomy of a deep neural network Layers Input data and targets Loss function Optimizer Layers Data processing modules Many different kinds exist densely connected convolutional recurrent pooling, flattening, merging, normalization, etc. Input: one or more tensors output: one or more tensors Usually have a state, encoded as weights learned, initially random When combined, form a network or a model Input data and targets The network maps the input data X to predictions Y′ During training, the predictions Y′ are compared to true targets Y using the loss function cat dog Loss function The quantity to be minimized (optimized) during training the only thing the network cares about there might also be other metrics you care about Common tasks have “standard” loss functions: mean squared error for regression binary cross-entropy for two-class classification categorical cross-entropy for multi-class classification etc. https://lossfunctions.tumblr.com/ Optimizer How to update the weights based on the loss function Learning rate (+scheduling) Stochastic gradient descent, momentum, and their variants RMSProp is usually a good first choice more info: http://ruder.io/optimizing- gradient-descent/ Animation from: https://imgur.com/s25RsOr Anatomy of a deep neural network Deep learning frameworks Deep learning frameworks + Actually tools for defining static or dynamic general-purpose computational + graphs Automatic differentiation ✕ ✕ Seamless CPU / GPU usage multi-GPU, distributed x y 5 Python/numpy or R interfaces instead of C, C++, CUDA or HIP Open source Deep learning Lasagne Keras TF Estimator torch.nn Gluon frameworks Theano TensorFlow CNTK PyTorch MXNet Caffe CUDA, cuDNN MKL, MKL-DNN Keras is a high-level HIP, MIOpen neural networks API we will use TensorFlow GPUs CPUs as the compute backend included in TensorFlow 2 as tf.keras https://keras.io/ , https://www.tensorflow.org/guide/keras PyTorch is: a GPU-based tensor library an efficient library for dynamic neural networks https://pytorch.org/