Introduction to Neural Networks

Performance Measure and Training Experience

• Performance measure P: average distance travelled before an error (as judged by human overseer)
• Training experience E: a sequence of images and steering commands recorded while observing a human driver

Machine Learning vs Traditional Programming

• Traditional programming: writing a program to produce the output based on given data
• Machine learning: training a computer to produce the program that solves the problem based on given data and desired output

AI, ML, and DL

• AI: Artificial Intelligence
• ML: Machine Learning
• DL: Deep Learning

Shallow Learning

• Feature extraction is a manual process that requires domain knowledge of the data
• Trainable classifier is often generic (e.g. SVM)
• Features are key to recent progress in recognition

Deep Learning

• Learn underlying features directly from data
• Multiple layers of nonlinear processing units
• Each layer extracts features from the output of previous layer
• Supervised or unsupervised learning of feature hierarchies representations in each layer

Why Now?

• Neural Networks date back decades, but recent dominance is due to advancements in ImageNet Challenge
• ImageNet Large Scale Visual Recognition Challenge (ILSVRC) is a classification task that involves producing a list of object categories present in an image

Neural Network Basics

• Artificial neural network: an information-processing system that mimics the way the human brain operates
• Components: neurons, connection links, weights, and activation functions

Artificial Neuron

• An artificial neuron is the basic building block that constructs complicated neural networks
• It makes a particular computation based on other units it's connected to

Importance of Activation Functions

• Introduce non-linearities into the network
• Activation functions: sigmoid, hyperbolic tangent (tanh), rectified linear (ReLU)

Activation Functions

• Sigmoid activation function:
  • Squashes the neuron's pre-activation between 0 and 1
  • Always positive
  • Bounded
  • Strictly increasing
  • Disadvantages: vanishing gradient problem, computationally expensive
• Hyperbolic tangent (tanh) activation function:
  • Squashes the neuron's pre-activation between -1 and 1
  • Can be positive or negative
  • Bounded
  • Strictly increasing
• Rectified linear (ReLU) activation function:
  • Bounded below by 0 (always non-negative)
  • Not upper bounded
  • Strictly increasing
  • Computationally efficient
  • Disadvantage: the dying ReLU problem

Neural Network Characterization

• A neural network is characterized by its architecture, activation function, and method of determining the weights on the connections (training algorithm)
• Hyperparameters include learning rate, epochs, and batch size