Natural Language Processing: Lexical Analysis

• Natural Language Processing (NLP) transforms raw text into meaningful data for analysis and interaction.
• There are five fundamental steps in NLP

Lexical Analysis

• Lexical analysis is the first stage of NLP, breaking text into tokens: words, phrases, or meaningful units that enable machine processing
• Tokenization splits text into individual words or phrases; i.e. "Natural Language Processing is fascinating" becomes "Natural", "Language", "Processing", "is", "fascinating"
• Normalization standardizes text by converting characters to lowercase for consistency; i.e. "Cat," "cat," and "CAT" become "cat"
• Stop Word Removal filters out non-essential words like "and," "the," "is," and "in" to reduce noise
• Lemmatization reduces words to their base or root form; i.e. "running" and "ran" become "run" to standardize similar meanings
• Stemming removes suffixes to reduce words to their root form, but is less precise than lemmatization and can produce non-dictionary words
• Morphological Analysis studies word structure to analyze components like stems, prefixes, and suffixes, understanding word formation rules

Syntactic Analysis

• Syntactic analysis examines sentence structure, determining relationships between words and adhering to language rules for structured text processing
• Parsing breaks down sentences into noun phrases (NP) and verb phrases (VP); i.e. "The cat sat on the mat" parses "The cat" as (NP) and "sat on the mat" as (VP)
• Grammar Checking ensures sentences follow correct grammatical rules, essential for applications like Grammarly
• Dependency Parsing maps relationships between words; i.e. in "She loves coding," "She" is the subject, "loves" is the verb, and "coding" is the object.

Semantic Analysis

• Semantic analysis focuses on understanding the meaning behind words and sentences, resolving ambiguities for accurate comprehension
• Word Sense Disambiguation determines the correct word meaning based on context; i.e. differentiating "bank" as a financial institution vs. a river side
• Named Entity Recognition (NER) identifies specific entities like names, dates, locations, and organizations; i.e. "Apple" as a company vs. "apple" as a fruit
• Semantic Role Labeling identifies the roles words play to extract deeper meaning; i.e. in "John gave Mary a book," "John" is the giver, "Mary" is the recipient, and "book" is the object

Discourse Integration

• Discourse integration analyzes interactions beyond individual sentences, ensuring continuity and coherence in conversation or text
• Anaphora Resolution identifies pronouns and their antecedents to maintain context; i.e. in "John went to the store. He bought milk," "He" refers to John
• Discourse Structure Modeling manages conversation flow by connecting sentences into a narrative
• Context Awareness understands references and implications across multiple sentences, important for machine translation and dialogue systems

Pragmatic Analysis

• Pragmatic analysis interprets intent beyond literal meanings, based on context, tone, and background knowledge, to generate human-like responses
• Implicature Analysis understands indirect suggestions; i.e. "It's cold in here" implies a request to close a window
• Speech Act Theory identifies statements as requests, commands, questions, or expressions of emotion; i.e. "Can you pass the salt?" is a request
• Social Context Awareness considers cultural and contextual nuances; sarcasm detection is used to refine responses for sentiment analysis