Naive Bayes Classifiers in Text Classification

Unit III: Naïve Bayes and Text Classification

• Naïve Bayes and text classification are covered in Unit III.
• The presenter is Dr. S. S. Gharde from the Department of Information Technology/AIML at Government Polytechnic Nagpur.

Contents

• The unit covers Naïve Bayes Classifiers.
• It includes a worked example of training the Naïve Bayes Classifier.
• Other text classification tasks using Naïve Bayes are discussed.
• The use of Naïve Bayes as a language model is explored.
• Evaluation methods, including confusion matrix, accuracy, precision, recall, and F-measure, are explained.
• Test sets and cross-validation are detailed.
• Statistical significance testing is also included.
• The presentation also covers avoiding potential harms in text classification.

Introduction

• Classification is crucial for both human and machine intelligence.
• Examples of classification include deciding what a letter, word, or image is; recognizing faces or voices; sorting mail; and assigning grades.
• Text categorization and sentiment analysis are applications of text classification.

Sentiment Analysis

• Sentiment analysis determines the positive or negative sentiment in a text, such as a movie review.
• Examples of positive and negative movie reviews are provided and labeled.

Why Sentiment Analysis?

• Used to determine a movie review's sentiment.
• Analyze public sentiment about products like the iPhone.
• Assess consumer confidence.
• Gauges political opinions about a candidate or issue.
• Predicts election outcomes or market trends from sentiment.

Scherer Typology of Affective States

• Breaks down emotions into brief, organically synchronized evaluations of major events (e.g., angry, sad, joyful, fearful, ashamed, proud, elated).
• Describes mood as diffuse, non-caused, low-intensity, long-duration changes in feelings (e.g., cheerful, gloomy, irritable).
• Defines interpersonal stances as affective attitudes toward individuals in specific interactions (e.g., friendly, flirtatious).
• Categorizes attitudes as enduring, affectively colored beliefs/dispositions toward objects/people (e.g., liking, loving, hating).
• Describes personality traits as stable dispositions/typical behavior tendencies (e.g., nervous, anxious, reckless).

Basic Sentiment Classification

• Sentiment analysis detects attitudes.
• This unit focuses on classifying text as positive or negative.
• Further classification of emotions and affects will be covered in later chapters.

Summary: Text Classification

• Text classification includes sentiment analysis and spam detection.
• It also covers authorship identification and language detection.
• Categorizing subject matter (topics or genres) is another application.

Text Classification: Definition

• Input: a document and a fixed set of classes.
• Output: a predicted class.

Classification Methods: Supervised Machine Learning

• Input: a document, a set of classes, and a training set of labeled documents.
• Output: a learned classifier that maps documents to classes.
• Specific methods like Naïve Bayes, Logistic Regression, Neural Networks, and k-Nearest Neighbors are included as classification methods.

Naïve Bayes Classifiers

• Naïve Bayes is a simple classification method based on Bayes' Rule.
• Relies on a simple document representation, like the "bag of words."

Naïve Bayes Classifiers: Bag of Words Representation

• The bag-of-words approach simplifies text representation.
• Example of a movie review broken down into words and counts for each.

Naïve Bayes Classifiers: Bag of Words Representation (Table Example)

• A table showing words and associated counts.

Naïve Bayes Classifiers: Bayes' Rule Applied to Documents and Classes

• Provides the formula for calculating posterior probabilities, given by P(c|d) = (P(d|c)P(c))/P(d).

Naïve Bayes Classifier (I)

• Provides the MAP (Maximum A Posteriori) formula for classifying a document using Bayes' Rule: argmax_c∈CP(c|d).

Naïve Bayes Classifier (II)

• The likelihood of features given a class (e.g., x1, x2,...,xn | c) and prior probability P(c) are used to find the class C_Map.

Naïve Bayes Classifier

• The Naïve Bayes assumption assumes conditional independence among probabilities given a class. This enables the probabilities of each word assigned to each class to be multiplied.

Multinomial Naïve Bayes Classifier

• The formula and practical application of the multinomial Naïve Bayes classifier, which uses word counts, are discussed.

Applying Multinomial Naïve Bayes Classifiers to Text Classification

• Procedure for classifying new texts using the trained model.

Example (Dataset)

• Sample data illustrating different aspects (e.g., outlook, temperature, humidity, wind) relevant to a classification problem (likely play tennis or not).

Example (Dataset Breakdown)

• Tables demonstrate how the probabilities are calculated for various inputs from the example data.

Example (Data Table, Movie Reviews)

• Example data showing how Naïve Bayes works with movie reviews for classification.

Training Naïve Bayes Classifier

• Methods for calculating probabilities/likelihoods for class-specific instances.
• Add-one (Laplace) Smoothing is used as a solution when probabilities are zero.

Training Naïve Bayes Classifier (Algorithms/Steps)

• Details on calculating parameters for training the classifier, along with how probabilities are calculated.

Worked example

• Provides a detailed example, including the training data, the test data, and the results of applying the classification.

Naive Bayes for other text classification tasks

• Demonstrates how Naïve Bayes classifiers can be used for more complex tasks, such as spam detection.
• Illustrates ways to use specific, pre-defined phrasing, or words, as features to improve classification accuracy.

Naïve Bayes for other text classification tasks (Additional Details): Spam Detection

• Explains how Naïve Bayes can be applied to detect spam.
• Covers examples of features used in spam detection, such as email subject lines with capital letters, or phrases of urgency.

Naïve Bayes for other text classification tasks (Additional Details): Language ID

• Explains how to use Naïve Bayes to identify languages in text. Example features include different character n-grams (e.g., n=1, n=2, n=3).

Naïve Bayes as a Language Model

• Describes a Naïve Bayes model as a class-specific model of unigrams; a unigram language model for each class.
• Explains assigning probabilities to sentences based on constituent words from each class.

Evaluation: Confusion Matrix

• Describes how to evaluate the performance of a text classifier, focusing on representing algorithm performance using a confusion matrix, which compares gold standards to predicted output.

Evaluation: Accuracy

• Explains accuracy as a measure of overall correctness.

Evaluation: Precision

• Describes precision as a measure of the positive results that the prediction made, out of all positives identified by the classifier.

Evaluation: Recall

• Describes recall as a measure of the positive results that were successfully identified, out of all positives present in the dataset.

Evaluation: F-measure

• Discusses the F-measure as a single metric combining precision and recall, with an emphasis on weighing either precision or recall depending on the specific application needs.

Test sets and Cross-validation

• Discusses using training and development sets, as well as how to assess classifier performance using test sets and cross validation.

Statistical Significance Testing

• Explains how hypothesis testing can evaluate differences between classification system performances.
• Introduces the idea of p-values and how to interpret them to determine if the results from one algorithm are better than another.

Avoiding Harms in Classification

• Highlights the importance of avoiding harms resulting from biased or harmful outputs from classification systems.
• Emphasizes the need to consider representational harms in the classification system's design and output.

End of Unit III