Stemming and Language Models Quiz

Topic 4: Stemming and Lemmatization

• Stemming: Process of stripping affixes (prefixes and suffixes) from words to find the root form.
• Porter Algorithm: A lexicon-free FST stemmer using rewrite rules to transform words.
• Omission (FP): Incorrectly removing affixes to find a basic morphological structure.
• Understemming (FN): Word(s) not stemmed to the same root when they should be.
• Overstemming (FP): Stemming two different words to the same root form when they should not be.

Topic 5: Language Models

• Language model: Statistical model assigning probabilities to sequences of words for various applications.
• N-gram models: Assign probabilities based on the history of previous tokens. Unigrams, bigrams, trigrams, n-grams are different types.
• Word prediction: Difficult to predict future words from noisy or ambiguous previous word(s). Analyze previous words to guess the next.
• Unigram Language model: Does not use history. Example sequences: my hometown is in Jordan
• Bigram Language model: Considers one previous word. Example i live in cheras, i am an undergraduate student.
• Trigram Language model: Considers two previous words. Example: i am an <5> student,
• Skip-gram model: Technique with n-grams allowing tokens to be 'skipped'.

Topic 6: Machine Learning and NLP

• Machine learning (ML) in NLP: Applications in personal productivity, language translation, voice assistants, recommendation systems, and self-driving cars.
• Deep learning (DL) in NLP: Google BERT, transformers as NN, word embeddings for recognizing word context.
• Data preprocessing for ML in NLP: Removing stop words, unnecessary punctuations, normalization (lowercasing), tokenization, stemming.
• TF-IDF: Term frequency-inverse document frequency for evaluating the importance of terms in a document.
• Vectorization: Turning text into numerical vectors with TF-IDF.
• Sklearn Tfidfvectorizer: Python library for vectorization.

Topic 7: Part-of-Speech (POS) Tagging

• POS tagging: Assigning grammatical tags (part-of-speech) to words.
• Closed classes: Small, fixed set of grammatical function words (prepositions, articles).
• Open classes: Large classes of words (verbs, nouns, adjectives, adverbs) that can be easily invented.
• POS tags: Nouns (singular, plural, proper: NNP, NNPS), personal pronouns, etc.
• POS tagging approaches: Rule-based and learning-based tagging(unigram, bigram).
• Evaluation measures: Precision, recall, F-measure to assess the accuracy of tagging.

Topic 8: Hidden Markov Models (HMMs)

• Hidden Markov Models (HMMs): Statistical model with hidden states and observable states with transitions.
• Likelihood Computation: Forward algorithm for computing the probability of observations using the probabilities of hidden state paths.
• HMM Decoding: Viterbi algorithm to find the best hidden sequence using the maximum likelihood principle.

Topic 9: Phrase Structure Grammar (PSG)

• PSG: Model for describing the constituent structures of sentences using rewrite rules.
• Parsing: Produce correct syntactic parse trees for sentences based on PSG rules.
• Top-down parsing: Starts with the start symbol, using rewrite (production/rule) rules.
• Bottom-up parsing: Starts with the words of the sentence and constructs phrases using rewrite rules.
• Noun phrase (NP), Verb phrase (VP), Prepositional phrase (PP): Common constituents in sentence structure.

Topic 10: Probabilistic Context Free Grammar (PCFG)

• PCFG: Incorporates probabilities to structure trees for likelihood analysis - used to estimate the most likely parse tree.
• Sentence probability: Sum of probabilities for different derivation choices.
• Viterbi algorithm: Dynamic programming to find the parse tree with the highest probability based on probabilities of different derivations.

Topic 11: Sentiment Analysis (SA)

• Levels of analysis: Document, sentence, entity & aspect - detailed level of granularity.
• Challenges in sentiment analysis: Complex ways of expressing opinions, lexical content alone, negation, sarcasm, intra and inter-sentential reversals.
• Feature extraction: Bag-of-words model.
• Point-wise mutual information (PMI): Info-theoretic approach to find collocations; higher scores don't always mean greater importance to bi-gram.
• Named Entity Recognition (NER): Identifying named entities (persons, organizations, locations, times) and classifying them into predefined categories.

Topic 12: Speech Recognition

• Speech recognition (SR): Converting audio speech to text.
• HMM models for SR: Convert speech signal (analog) into a sequence of numbers for use by the recognizer.
• Spectogram: Visual representation of the frequency domain of a sound (used by speech recognition systems).
• Applications: Voice assistants, speaker recognition.
• SPEECH ANALYSIS: Feature extraction, transformation, & dimensionality reduction for speech processing.

Test your understanding of stemming techniques and various language models. This quiz covers key concepts such as the Porter algorithm, n-gram models, and the challenges of word prediction. Perfect for students diving into natural language processing.