1. Deep Learning and its Variants_Session 1_20240114 - Neural Networks and Machine Learning Concepts

Biological Neural Networks

• A biological neural network consists of 100 billion neurons and 1000 trillion synaptic connections.
• The fundamental units of a biological neural network are neurons, connected by synapses.
• The direction of signal transmission in a biological neural network is along the axon from the nucleus to the synapse.

Biological Inspiration

• The spikes traveling along the axon of the pre-synaptic neuron trigger the release of neurotransmitter substances at the synapse.
• The neurotransmitters cause excitation or inhibition in the dendrite of the post-synaptic neuron.
• The integration of the excitatory and inhibitory signals may produce spikes in the post-synaptic neuron.
• The contribution of the signals depends on the strength of the synaptic connection.

Learning in Biological Neural Networks

• In 1949, Donald Hebb postulated that if a synapse is used more, it gets strengthened, releasing more neurotransmitters.
• This causes the particular path through the network to get stronger, while others, not used, get weaker.
• Each connection has a weight associated with it, which influences the strength of the signal transmission.

Computational Neuron

• An artificial neuron has an input, weights, sum, and threshold.
• The synaptic strengths influence the input to the next neuron, based on the strength of the synaptic connection.
• Dendrites carry signals to the cell body, where they are summed, and if the sum exceeds the threshold, the neuron fires.

Hebbian Learning

• Hebbian learning states that if a synapse is used more, it gets strengthened, causing the particular path through the network to get stronger.
• Machine learning determines the synaptic strength (weights) by finding the optimal weights consistent with the given data.

Perceptron

• The perceptron is the first artificial neuron, and its structure is a model of decision making.
• It makes decisions by weighing up evidence and captures key characteristics: input, weights, sum, and threshold.
• The notation is a dot product of vectors of inputs and weights.
• Perceptrons are brittle, and changing weights can cause a spiky output, rather than a smooth change.

Sigmoid Neurons

• Sigmoid neurons use a different activation function, which produces a smooth output between 0 and 1.
• The output of a sigmoid neuron changes smoothly from 0 to 1 as the input changes.

Classification with Logistic Regression

• Logistic regression is a model that uses a classification task to estimate the probability of a binary outcome.
• The weight coefficients are learned from the data.
• The output of a logistic regression model is a probability between 0 and 1.
• The MT cars dataset is an example of using logistic regression to estimate the probability of a vehicle being fitted with a manual transmission.

Test your knowledge on concepts related to Hebbian Learning, Machine Learning, and the Perceptron model. Explore how synaptic strength impacts network pathways and how optimal weights are determined in machine learning.