Probabilities in NLP: Bigram and Unigram

Natural Language Processing (NLP)

• NLP is a branch of artificial intelligence (AI) enabling computers to understand, generate, and manipulate human language.
• It combines computational linguistics, machine learning, and deep learning models to process human language.
• Computational linguistics focuses on understanding and constructing human language models with computers and software tools.
• Natural language is how humans communicate daily, including speech and text.

NLP Components

• Morphological Analysis: Examines word components (prefixes, suffixes, roots).
  • Analyzes word formation and components using machine learning.
• Lexical Analysis: Breaks down text into fundamental units (words, punctuation, whitespace).
  • Also called tokenization.
  • Splits text into sentences and sentences into words.
• Syntactic Analysis (Parsing): Analyzes the grammatical structure of a sentence.
  • Identifies syntactic relationships between words and phrases.
    • Part-of-speech tagging (POS) is a necessary first step.
• Semantic Analysis: Understands the meaning of words and sentences.
  • Includes named entity recognition (NER), word sense disambiguation, and semantic role labeling.
• Discourse Integration: Captures the context and coherence across sentences.
  • Coreference resolution is a common task, identifying when different expressions refer to the same entity.
• Pragmatic Analysis: Understands the intended meaning beyond literal meaning.
  • Interprets idioms, sarcasm, and context-specific implications. 
  • Sentiment analysis tools like VADER assess sentiment.

NLP Applications

• Sentiment Analysis: Determines the sentiment expressed (positive, negative, or neutral).
• Text Classification: Categorizes text into predefined categories.
• Chatbots and Virtual Assistants: Automates customer interactions.
• Text Extraction: Extracts specific pieces of information (names, dates, etc.) from text.
• Machine Translation: Translates text from one language to another.
• Text Summarization: Creates concise summaries of text.
• Market Intelligence: Analyzes numerous text sources for insights.
• Auto-correct: Corrects grammar and spelling errors.
• Intent Classification: Understands the user's intentions behind a query or statement.
• Urgency Detection: Determines urgency in a message or request.
• Speech Recognition: Translates speech to text.

NLP Techniques

• Bag-of-Words (BoW): Represents text as a set of word counts.
• TF-IDF (Term Frequency-Inverse Document Frequency): Determines word importance in a document relative to a corpus.
• Word Embeddings: Represent words in continuous vector space.
• Recurrent Neural Networks (RNNs): Designed for sequential data (useful for language modeling).
• Transformers: Advanced models (like BERT and GPT) are highly effective for many NLP tasks; process entire sequences in parallel.

NLP Challenges

• Ambiguity: Words and sentences can have multiple meanings.
• Context Understanding: Capturing the context of words in different scenarios.
• Data Quality: High-quality, annotated data is essential for training effective models.
• Computational Resources: Training deep learning models can be resource-intensive.
• Model Interpretability: Deep learning models are often difficult to understand internally ("black boxes")

Test your understanding of maximum-likelihood probabilities in natural language processing with this quiz. Questions cover bigram and unigram probabilities, as well as the concepts of counts and perplexity in a corpus. Dive into the fascinating world of language models and their calculations.