Seq2Seq Models in Machine Learning

Web and Text Analytics 2024-25, Week 8

• The course is about web and text analytics.
• The instructor is Evangelos Kalampokis.
• The website for the course is https://kalampokis/hub.io.
• The information systems lab website is http://islab.uom.gr

Data Preparation - Preprocessing Examples 1.1

• Missing values are removed using df.dropna(inplace=True).
• Ratings equal to 3 (neutral) are removed.
• Ratings greater than 3 are coded as 1 (positive), otherwise as 0 (negative).
• Frequency counts for each rating are calculated and plotted.
• A simplified text preprocessing function is defined.

Data Preparation - Preprocessing Examples 1.2

• Missing values are removed.
• Ratings equal to 3 are removed.
• Ratings greater than 3 are coded as 1 (positive), otherwise as 0 (negative).
• Positive and negative reviews are extracted.
• Data is split into training and test sets.
• TF-IDF is used for feature extraction.
• Class imbalance is handled (e.g., scale_pos_weight).

Data Preparation - Preprocessing Examples 1.3

• Missing values are removed.
• Neutral ratings (equal to 3) are removed.
• Positively rated reviews (4, 5) are coded as 1, and negatively rated reviews (1,2) as 0.
• Negations in text are removed and replaced with "not_" followed by the negated word.
• Text is lemmatized using WordNetLemmatizer.

Feature extraction

• Two features are extracted from the reviews: positive words and negative words counts
• The model is trained using these features.

Train and test a classifier, Naïve Bayes (1)

• Data is split into training and test sets (with 80% training and 20% testing).
• TF-IDF is used to extract features (with specified settings).
• Class imbalance is addressed (using scale_pos_weight).
• A Naïve Bayes Classifier is trained.
• A confusion matrix is used for visualization.

Train and test a classifier, Naïve Bayes (2)

• Data is converted into count-based features using CountVectorizer (with unigrams and bigrams).
• Or tfidfVectorizer (with unigrams and trigrams).
• A Naïve Bayes classifier is trained using count based data.
• Probabilities are predicted to assign True (1) for probabilities greater than 0.8.
• Accuracy, precision, recall, and F1 score are calculated and reported.

Train and test a classifier, XGBoost (1)

• A XGBoost classifier is created and used with RandomizedSearchCV to find the best hyperparameters.
• A parameter grid is defined to search for optimal hyperparameters.
• The best hyperparameter values are printed.
• The best estimator from the RandomizedSearchCV is stored in the variable 'model'.

Train and test a classifier, XGBoost (2)

• The model predicts the sentiment of a given review.

Train and test a classifier, XGBoost (3)

• The model's accuracy, precision, recall, and F1-score are presented, along with a confusion matrix.

Online Courses

• DataCamp course on Recurrent Neural Networks with Keras is linked.
• Coursera course on Natural Language Processing with Sequence Models is listed.

Sequence to Sequence (seq2seq) models

• Seq2Seq models transform one sequence into another.
• They are useful for tasks where both input and output sequences may have varying lengths (e.g., translation, summarization).

Use Cases of the Sequence to Sequence Models

• Seq2Seq models are used for various applications such as machine translation, text summarization, speech recognition, chatbots, image captioning, video captioning, time series prediction, and code generation.

Encoder-Decoder Architecture

• The encoder-decoder architecture is a common way to build Seq2Seq models.
• The encoder transforms the input sequence into a fixed-length representation.
• The decoder takes the representation and produces the output sequence.
• Different neural network architectures (like RNN, LSTM) can be used with the encoder-decoder framework.

Recurrent Neural Networks (RNN)

• RNNs are a type of deep learning model used for sequence data.
• RNNs share weights among their steps and propagate information through the sequence.
• They are effective for NLP tasks, but can struggle with long dependencies (vanishing gradient).

RNN example

• RNNs consider all previous words in a sentence unlike N-grams.

RNN example

• RNNs are able to track dependencies between words in a text corpus.
• The same weights are applied to each word in the sequence.

Training RNN models

• RNNs propagate forward and backward.
• The process is called "back propagation through time".

RNN - Forward Propagation

• Hidden states are important for RNNs to carry prior information during computation.
• A weight matrix (Wa) is shared among steps in the sequence.

RNN - Back Propagation through time

• The gradient shrinks as it propagates backward through time, which leads to the "vanishing gradient" problem.
• RNNs can struggle with long-term dependencies.

Vanishing Gradient Problem

• As information propagates backward further in a sequence, the gradient can approach zero.

RNNs and Vanishing Gradients

• RNNs have advantages such as being able to capture long-range dependencies and taking up less memory than n-gram methods.
• RNNs have disadvantages like struggling to deal with long-range dependencies because of the vanishing gradient issue.

Simple RNN cell

• In the simple RNN cell, the weight matrix Wa is shared across inputs and steps.

GRU cells

• GRU cells use gates to control the flow of information (like input, update, and forget).
• GRU cells address some of the problems with basic RNNs.

GRU and LSTM

• GRU is a variant of LSTM with fewer parameters.
• GRU and LSTM architectures are effective for various sequence learning tasks, yet GRU has fewer processing steps.

LSTM

• LSTMs are designed to better handle long-range dependencies in sequences.
• LSTMs contain cell state and hidden states with gates.

LSTM

• LSTM cells have 3 gates: input, forget, and output.
• The forget gate decides if information from previous steps should be discarded.

RNN - Embeddings

• Embedding layers can be utilized to produce vector representations of tokens.

RNN example

• RNNs can consider previous words in a sentence.

RNN example

• RNNs track dependencies in text and employ the same weights for every word.

RNN example

• LSTMs and GRUs are used to address the RNN's long-term dependency limitations.