NLTK for Data Preprocessing and Tokenization

Installing NLTK

• To install NLTK in Jupyter Notebook or Google Colab, use the following command: !pip install nltk.
• Import the NLTK library: import nltk.
• Download all NLTK resources: nltk.download('all').

Data Preprocessing

• Data Preprocessing is the process of cleaning unstructured text data for analysis, prediction, and information extraction.
• Real-world text data is unstructured and inconsistent; therefore, data preprocessing is crucial.

Tokenization

• Tokenization breaks down text data into individual units called tokens, which can be words, sentences, or characters.
• Tokenization is the first step in text analysis and is implemented using the tokenize class.
• Sentence Tokenization: Splits text into sentences using the sent_tokenize() function from the nltk.tokenize module.
• Word Tokenization: Splits text into individual words using the word_tokenize() function from the nltk.tokenize module.

Frequency Distribution of Words

• Determines how many times each word appears in a given text.
• Generate the frequency distribution using the FreqDist() function in the nltk.probability submodule: from nltk.probability import FreqDist.
• Print the most frequent words using most_common().
• Use Matplotlib's pyplot submodule to create a graph of word frequency.
• The plot() function in pyplot draws a line graph, with the x-axis representing the words and the y-axis representing their frequency.

Filtering Stop Words

• Stop words are common, repetitive words that don't hold significant information, such as "that," "these," "below," "is," "are," "a," "an."
• NLTK provides a list of stop words in the stopwords module.
• Download the stopwords module from the NLTK corpus: nltk.download('stopwords').
• Use the format() method to concatenate output elements with positional formatting using curly braces {}.

Stemming

• Stemming is a text normalization technique that reduces words to their root form (stem).
• Stemming is used by chatbots and search engines to understand the meaning behind search queries.
• Porter Stemmer: A widely used stemming algorithm that removes common suffixes from words.
• Snowball Stemmer: An advanced version of Porter Stemmer, also called Porter2, which addresses some issues with Porter stemming and supports multiple languages.
• Lancaster Stemmer: Uses an aggressive approach, over-stemming some terms, and reduces words to their shortest possible stems.
• Stemming is faster than lemmatization but does not consider the context of words.
• Stemming can lead to inaccurate results due to its aggressive nature.

Lemmatization

• Lemmatization is a text normalization technique that reduces words to their root form (lemma), while preserving their meaning.
• Lemmatization uses vocabulary and morphological analysis to return the most meaningful base form of a word.
• Lemmatization considers context and converts the word to its meaningful base form.
• WordNet Lemmatizer: Provides lemmatization features and is used by major search engines.
• The WordNet lemmatizer defaults to treating words as nouns. Use the part of speech (POS) tags described in the table to provide POS information externally.

Parts of Speech (POS) Tagging

• POS tagging assigns a grammatical category (noun, verb, adjective, adverb) to each word in a sentence.
• POS tagging, also known as word classes or lexical categories, helps analyze the structure of language.
• The collection of tags used for a particular task is called a tagset.
• Rule-Based POS Tagging: Relies on predefined grammatical rules, a dictionary of words, and their POS tags. It is simple but less accurate.
• Statistical POS Tagging: Uses machine learning algorithms to predict POS tags based on the context of words in a sentence. Requires a large amount of training data and resources. It is more accurate.
• Averaged Perceptron Tagger: A statistical POS tagger using the Averaged Perceptron algorithm trained on a large corpus of text. It uses the universal POS tagset.

Name Entity Recognition

• Used to identify names of organizations, people, geographic locations, and other real-world entities within text.
• NLTK provides basic NER functionality.
• Chunking: Groups words into meaningful phrases based on their POS and syntactic structure.
• MaxEnt NE Chunker: Uses the Maximum Entropy classifier to identify and classify named entities, such as people, organizations, or locations.
• Named entities are labeled with tags like "PERSON," "GPE" (Geopolitical Entity), "DATE," etc.

Types Ambiguities

• Ambiguity refers to a word or phrase having multiple interpretations.
• Lexical Ambiguity: A single word has multiple meanings.
  • Homonymy: Two words spelled the same, but with different meanings. Ex: bat, bank.
  • Polysemy: Words with multiple meanings depending on context and tone. Ex: a fine house, a fine situation.
• Syntactic Ambiguity: Ambiguity arises from ambiguous sentence structure or syntax. Ex: I saw the stranger with the telescope.
• Semantic Ambiguity: Ambiguity arises from multiple meanings in a phrase or sentence. Ex: He ate the burnt rice and dal.
• Word Sense Disambiguation (WSD): Determines the correct meaning of a word in a specific context. Ex: I saw a bat.
• Referential Ambiguity: When a pronoun refers to something unclear. Ex: Kiran went to Sunita. She said, "I am hungry."
• Pragmatic Ambiguity: Interpretation depends on context, cultural knowledge, assumptions, and expectations. Ex: Can you pass the salt?
  • Conversational Ambiguity: Ex: "Can you pass the salt?"
  • Assumptions: Sentence carries implicit assumptions not explicitly stated. Ex: After doing submission, you can have your own time.
• Ambiguity due to Noise and Errors:
  • Speech Recognition Errors:
  • Typographical Errors: