Natural Language Processing Overview

Natural Language Processing (NLP)

• NLP is a field of research in computer science that combines AI and Deep Learning to enable computers to understand and process human language.
• The goal of NLP is to enable computers to interact effectively with humans through natural language.

Natural Language Understanding (NLU) and Natural Language Generation (NLG)

• NLU deals with understanding the meaning of human language, usually expressed as text.
• NLG generates text, allowing computers to "talk" to humans through natural language.

Examples of NLP in Action

• Chatbots use NLU to understand questions and NLG to generate responses.
• Example: a chatbot answering the question "do penguins fly?" requires NLU to understand the question and NLG to generate a response.

Common Tasks in NLP

• Language Translation: translating from one language to another (e.g., Google Translate).
• Speech Recognition: converting voice data into text.
• Text Summarization: summarizing text from literature.
• Sentiment Analysis: extracting subjective qualities like attitudes, emotions, and sarcasm from text.

Challenges in NLP

• Ambiguity: human language can be ambiguous at different levels (word, sentence, etc.).
• Use of common sense: humans do not learn language by observing an endless stream of text, making it difficult for computers to understand language.
• Figurative language: phrases with non-literal meanings pose a challenge for language understanding algorithms.

Text Pre-processing for NLP Applications

• Preprocessing is a crucial step in NLP that cleans, harmonizes, and prepares text for further processing.
• Agenda for text preprocessing includes:
  • Text cleaning/harmonization/reduction of fluctuations
  • Text normalization
  • Segmentation
  • Stop words
  • Stemming & Lemmatization

Tokenization

• Tokenization is the process of breaking down text into smaller chunks, called tokens, such as words, phrases, symbols, or other meaningful elements.