Transformers 101: From Zero to Hero

A summary of key concepts from the Data Science Course for the first year in Medicine and Pharmacy programs.

Sequential data is information that occurs in a specific order, like words in a sentence.

Recurrent Neural Networks (RNNs) are a type of artificial neural network that are good at processing sequential data.

Transformers are a type of deep learning model that are better than RNNs for handling sequential data.

Natural Language Processing (NLP) involves computers understanding and processing human language, like text and speech.

Transformers are used in Natural Language Processing (NLP) for tasks like machine translation and text summarization.

The Transformer's encoder is responsible for processing the input sequence and understanding its meaning.

The Transformer's decoder generates the final output sequence based on the encoded information.

A method that helps identify recurring themes or subjects within a text or collection of documents.

A branch of Natural Language Processing (NLP) that aims to decipher the meaning behind sentences, enabling software to understand similar meanings expressed differently or handle words with multiple meanings.

A specialized neural network designed to learn the context of sequential data and generate new data based on that learned understanding. It excels at processing and generating human-like text by analyzing patterns in massive amounts of text data.

A framework commonly used in machine learning for tasks like machine translation, text summarization, and image captioning. It comprises two main components: the encoder and the decoder.

The component in the encoder-decoder architecture that processes the input sequence and attempts to understand its meaning.

The component in the encoder-decoder architecture that generates the final output sequence based on the encoded information.

A mechanism within transformer models that allows the model to assess the importance of different words in a sequence, regardless of their position, leading to more efficient and accurate language understanding.

A sequence is an ordered list of items or events where the order is important. Each item in the sequence is connected to the next. Examples include words in a sentence, notes in a melody, or daily temperatures.

Feed-forward Neural Networks (FNNs) and Convolutional Neural Networks (CNNs) process data without considering the order of inputs. The output for one item is independent of any previous inputs. This is useful for tasks like image classification or health prediction, where input order is less important.

Recurrent Neural Networks (RNNs) were designed to handle sequences by keeping track of information from past inputs. This allows them to analyze data that changes over time and understand relationships between elements.

FNNs and CNNs accept inputs of a fixed size, while RNNs can handle variable-length sequences. This means RNNs can understand sentences of different lengths or time series with different durations.

RNNs use internal 'memory' to keep track of previous information in a sequence. This allows them to understand the context of an item based on its position and what came before.

RNNs are ideal for tasks that involve analyzing patterns, predicting future events based on past data, or understanding the relationships between elements in a sequence.

The memory mechanism in RNNs allows them to process long sequences without losing important information from the beginning of the sequence. This is crucial for tasks like machine translation, where full sentence context is needed

RNNs are used in: natural language processing (NLP), speech recognition, machine translation, time series analysis, and more.

It captures the essence of the input data, transforming it into a compact representation that the decoder can understand.

It utilizes this encoded representation to produce the desired output, like a translated sentence or summary.

This type of neural network effectively handles sequential data, such as sentences or speech, by retaining information over long sequences.

It decides which information from the previous state is irrelevant and should be discarded.

It determines what new information from the current input should be added to the memory.

It controls which information from the memory cell is passed along to the next step in the sequence.

This architecture is commonly used in machine learning to handle tasks such as machine translation, text summarization, and image captioning.

It's a unique type of neural network that handles input sequences by paying attention to the relationships between different parts, unlike traditional RNNs.

In neural networks, the vanishing gradient problem occurs when gradients become increasingly small as they flow backward through the network, leading to slow or ineffective learning. This is often problematic in deep networks.

Residual connections in neural networks allow information to directly flow from earlier layers to later layers, bypassing some layers in the network. This helps overcome the vanishing gradient problem by allowing gradients to flow more easily, even in very deep architectures.

Normalization techniques like Layer Normalization (LN) help ensure that the activations in each layer of a neural network have a consistent distribution, preventing large fluctuations in the network's internal state. This often leads to better training stability and improved performance.

A feed-forward neural network (FFNN) is a type of neural network where information flows unidirectionally from input to output, without loops or feedback connections. This is in contrast to recurrent neural networks (RNNs) which have feedback connections.

Positional encodings in Transformers add information about the relative position of each token in the input sequence. This is crucial because Transformers don't process the input sequentially like RNNs, making it essential to preserve the order of tokens.

Self-attention is a powerful mechanism in Transformers where each word in the input sequence can directly interact with all other words, regardless of their position. This allows the model to understand the relationships between words in a sentence and build context.

The output of the encoder in a Transformer model is a set of vectors, each representing the input sequence with a rich contextual understanding. The decoder then utilizes this encoded output to generate its output, often in the form of a translated sentence or a summarized text.

Similar to encoder layers, they involve multiple attention mechanisms and a feed-forward network, but are designed to generate the output sequence step by step.

A special token that signals the beginning of the output sequence for the decoder.

A special token added at the end of the output sequence, signaling the decoder to stop generating words.

The decoder's self-attention mechanism prevents each position from attending to future positions in the input sequence. This ensures that the decoder only considers preceding words to predict the current word.

This layer in the decoder is responsible for combining encoded information from the encoder with the decoded output, providing context from the original input.

It acts as a classifier, converting the decoded output into probability scores of possible words. The final word probabilities are then calculated using the softmax function.

The decoder uses previously generated words as input to predict the next word, making output generation progressive and context-aware.

The decoder operates sequentially, starting with the start token, and utilizes previously generated words as well as encoder outputs to generate the complete output sequence.