NLP and its capabilities

• AI can analyze, process, and generate speech or text.
• Natural Language Processing (NLP) enables computers to use human languages in written and spoken forms, similar to how humans do, and can extend to non-language applications like coding and music.

NLP Capabilities

• Named Entity Recognition (NER): Identifies and classifies names of persons, places, companies, etc., into predefined labels.
• Topic Modelling: Discovers hidden topics within large document collections.
• Text Categorization: sorts text into taxonomies
• Text Clustering: Groups texts/documents based on content similarities.
• Sentiment Analysis: Identifies, extracts, quantifies, and studies affective states and subjective information.
• Summarization: Generates a short version of an input document, retaining important points.
• Information Extraction: Finds meaningful information in unstructured text.
• Entity Resolution: Identifies records referring to the same real-world entities across internal or public data sources and identifies relationships between records.
• Translation: Converts text from one language to another while retaining meaning.
• Speech Recognition: Converts speech to text.
• Speech Synthesis: Converts text to speech.
• Natural Language Generation (NLG): Transforms data into human language.

Named Entity Recognition (NER) Deconstructed

• NER identifies entities within text and classifies them.
• An online tool allows users to input text, select a language, and label entity types, to see how well NER works.

Sentiment Analysis Deconstructed

• Sentiment analysis detects polarity (positive or negative), emotion (e.g., anger, happiness, sadness), urgency, and intention in text or speech.
• Sentiment analysis allows brands to analyze customer feedback for product improvement.

Summarization Deconstructed

• Extractive models copy and paste relevant phrases from an input document to form a summary.
• Abstractive models generate a summary based on the abstracted content and can use novel words or connectors and can also paraphrase.

Information extraction Deconstructed

• AI NLP extracts information from Big Data which is extensive data sets which are too big to analyse using traditional methods.
• AI can analyze patent data to help investors make more informed decisions

Question Answering Deconstructed

• Question-answering AI generates answers to given questions by querying a knowledge base.
• Closed-domain question answering deals with questions under a specific domain, while open-domain question answering deals with factual questions about nearly everything.

Entity resolution Deconstructed

• Entity resolution identifies when different records refer to the same entity.
• Entity resolution can be used to detect fraud, improve risk assessment, improve investigative outcomes, help ensure compliance, improve customer insights, and reduce false positives and false negatives.

Translation Deconstructed

• Classical machine translation uses dictionaries and grammars needing manual effort.
• Statistics-based translation picks the most likely translation according to sample data.
• Google Translate uses neural networks instead of statistics-based models, which it began in 2016.

Speech recognition Deconstructed

• AI Singapore, NUS, and NTU developed a speech recognition engine (Speech Lab) to recognize conversations in multiple languages, including Singlish.

Speech synthesis Deconstructed

• Google's DeepMind created the WaveNet model which can generate realistic, human-like voices better than its previous systems.
• Speech can be synthesized to imitate a speaker in a language they don't speak, or a person suffering from a voice disorder.

Natural language generation (NLG) Deconstructed

• NLG applications include suggesting email replies, and collating audit findings.
• The complexity, ambiguity, and variety of expressions in human languages make NLG challenging.
• Powerful NLG AIs are massive programs with language information extracted from massive amounts of data, referred to as large language models (LLMs).
• The most popular LLMs are Transformer models.

Further applications

• Chatbots are used to conduct conversations in natural language.
• OpenAI launched ChatGPT which adapts to the style and content of the prompt and can generate realistic and coherent continuations about a topic of their choosing.
• ChatGPT alternatives include Google's Gemini, Meta's LLaMA, Baidu's ERNIE bot, Alibaba's Tongyi Qianwen, and Anthropic's Claude.
• Chatbots can not only generate text, but also analyze sentiment, summarize text, and translate text.
• Besides chatbots, examples of virtual voice agents include Google Assistant, Apple's Siri, and Amazon's Alexa.
• Voice chatbots, like Google Duplex and Amazon Connect, can perform real-world tasks over the phone.
• Computer languages are languages. So language models are applied to them as well and can generate and translate computer code.
• The WaveNet model, can also be used to synthesize other audio signals such as music.

Current challenges

• Voice agents still struggle with voice recognition, especially with multiple languages and dialects.
• ASEAN languages lack sufficient linguistic resources (corpus).
• AI lacks true language understanding, limiting its question-answering abilities and dealing with ambiguous questions.
• Chatbots still have limited scopes.
• Sentiment analysis is affected by many parts of a text. Each affects the different meanings and implications.
• Many NLP systems require additional training.
• LLMs are slow and may produce inaccurate responses (hallucinate) that seem plausible but are factually incorrect.