Natural Language Processing (NLP)

Natural Language Processing (NLP) Introduction

• NLP enables computers to understand, interpret, and produce human languages.
• NLP bridges human communication and machine comprehension to help computers understand language and provide feedback.

Key NLP Goals

• Understanding: Extracting meaning from text or speech.
• Interpretation: Recognizing sentiment, intent, or context.
• Interaction: Facilitating natural dialogue between humans and machines.

Challenges in NLP

• Complexity: Human language involves slang, idioms, and dialects with different expressions for the same idea.
• Ambiguity: Words and phrases can be interpreted in multiple ways without sufficient context.
• Context Dependence: Meaning changes based on surrounding text or conversation history.

NLP Applications

• Machine Translation: Automatically converting text/speech between languages.
• Chatbots & Virtual Assistants: Providing automated support and conversation.
• Automated Customer Support: Offering immediate replies to common questions, reducing wait times.
• Sentiment Analysis: Gauging opinions from social media or reviews.
• Content Recommendation: Suggesting content based on language data such as search queries and watch history.
• Automatic Text Summarization: Condensing lengthy documents into concise summaries.
• Educational Tools: Checking grammar, language tutoring, and real-time feedback.
• These applications use language models for context, semantics, and intent detection.

NLP History

• Early NLP used linguistic rules, which are rule-based systems that interpret, parse, or generate language, triggering results when text matches certain criteria.
• Rule-based systems are time-consuming to create and not easily scalable or adaptable to new domains.
• Statistical NLP moved toward data-driven methods.
• Statistical NLP involves feature extraction from text and training statistical or machine learning models (like naive Bayes, SVM), requiring large, annotated datasets.

Neural Networks & Deep Learning in NLP

• Neural networks represent the modern era of NLP.
• Embeddings: word, sentence, or document-level vectors used to capture meaning.
• Transformer architecture: revolutionizes language modeling with attention mechanisms and pre-training on vast amounts of text.

Current Limitations of NLP

• Despite advanced models like GPT-4, NLP is not fully solved.
• Low-resource languages remain challenging due to limited digital resources.
• Contextual nuances such as idioms and sarcasm still cause errors.
• Bias and ethics concerns persist, with models potentially producing harmful or biased content.

Low-Resource NLP

• Most NLP breakthroughs focus on English or other major languages.
• Limited digital resources in some languages make it challenging to create accurate NLP tools.
• Low-resourced languages lead to less accurate translation, morphological analysis, and language modeling.

Contextual Understanding Challenges

• Models may incorrectly label phrases as hate speech or not, depending on the target group.
• The same sentence with different nouns can produce inconsistent classification outcomes.
• Hallucinations: Models sometimes produce misinformation.
• Models may show biased responses if the training data contains stereotypes or harmful content.

Vector Representation of Text

• Text is numerically encoded for processing.
• Vectors are ordered lists or arrays of numbers, with each number being a component or coordinate.
• Vectors represent direction and magnitude in geometry, features in machine learning, and word embeddings in NLP.

Properties of Vectors

• Dimension: The number of components in a vector.
• Magnitude (or Length): The "size" of a vector calculated by the square root of the sum of the squares of its components.
• Direction: Points from the origin to where its components lead in space.
• Zero Vector: A vector with all zero components, having zero magnitude and no direction.

Vector Operations

• Vector Addition: Adding two vectors of the same dimension by adding their corresponding components.
• Scalar Multiplication: Multiplying each component of a vector by a scalar (real number).
• Dot Product (Inner Product): Measures how much two vectors "line up" geometrically
• The dot product can be caluclated as the the product of the magnitudes of the vectors multiplied by the cosign of the angle between them

Cosine Similarity

• It is a normalized dot product ranging from -1 to 1, or 0 to 1 for non-negative vectors.
• Cosine similarity is used to measure similarity between two-word embeddings or two documents.

Distance Measures

• Euclidean Distance: Ordinary distance based on subtracting coordinates.
• Manhattan Distance (or L1 distance): Sum of absolute differences.
• Cosine Distance: 1 – cosine similarity used when magnitudes differ widely.

Transformation or Embeddings

• In NLP, embedding models convert text into vectors, training a model to learn from data rather than manually assigning numbers.
• Operations include adding or subtracting vectors, finding magnitude, and measuring similarity using the dot product or cosine similarity.

Example Calculations of Vectors

• Short examples illustrate vector operations like addition and dot product.
• Magnitude and cosine similarity calculations are also provided.

Summary of Vectors

• Vectors are lists of numbers capturing direction and magnitude.
• They serve as a numeric representation of data in machine learning.
• Vector addition, subtraction, and scaling are common operations.
• Dot product and cosine similarity measure alignment or similarity between vectors.
• Vectors are fundamental in math, physics, engineering, computer science, and machine learning.

Why Vector Representation

• Machines process numeric data efficiently, so text is converted into numeric vectors.
• Once in numeric form, classification, clustering, and searching can be applied.

Words as Atomic Symbols (One-Hot)

• One-hot encoding assigns each word a giant vector with zeros everywhere except for a 1 in the position corresponding to the word's ID in the vocabulary.

Drawbacks of One-Hot Encoding

• Very large (sparse) vectors, especially with huge vocabularies.
• No built-in notion of "similar word," as each vector is equally distant from all others.
• Struggles with new or out-of-vocabulary words.

Distributed Representation

• Instead of using one-hot vectors, each word is assigned a dense vector in Rd, capturing semantic relationships like synonyms and word analogies.

Distributional Similarity & Count-Based Methods

• Words are known by the company they keep (J. R. Firth, 1957).
• Word vectors are created by counting how often a given word co-occurs with other words in a corpus within some context window.
• Term-document matrices or term-term co-occurrence matrices are used.

Disadvantages of Count-Based Methods

• Often produces high-dimensional, sparse representations.
• Very sensitive to how the context window is defined.
• Doesn't directly address antonyms, polysemy, or subtle semantic differences.

Alternatives to Count-Based Methods

• TF-IDF: Weigh terms by frequency and "distinctiveness."
• Latent Semantic Analysis (LSA): Reduces dimensionality of a term-document matrix using Singular Value Decomposition (SVD).
• Pointwise Mutual Information (PMI): Measures how often two words co-occur compared to what's expected by chance.