Introduction to Natural Language Processing

What is Natural Language Processing (NLP)?

• Natural language processing is a field that intersects computer science, artificial intelligence, and linguistics.
• The goal of NLP is for computers to process or "understand" natural language to perform useful tasks.
• Examples of useful tasks that NLP enables are making appointments, buying things, and question answering, like Siri or Google Assistant.
• Fully understanding and representing the meaning of language is a difficult, AI-complete goal.

NLP Levels

• NLP tasks are executed through a series of levels that address the problem from different angles: Phonetic/Phonological Analysis, OCR/Tokenization, Morphological analysis, Syntactic analysis, Semantic Interpretation, and Discourse Processing.

NLP Applications

• Applications range from simple tasks such as spell checking, keyword search, and finding synonyms to complex tasks.
• Some more complex tasks include extracting information from websites, like product prices, dates, locations, and company names.
• NLP can be used to classify the reading level of school texts, and the sentiment of longer documents.
• NLP is also used in machine translation, spoken dialog systems, and complex question answering.

NLP in Industry

• NLP is used in search (written and spoken) and online advertisement matching.
• Automated/assisted translation and sentiment analysis for marketing or finance/trading also use NLP.
• Speech recognition and chatbots/dialog agents use NLP to automate customer support, control devices, and order goods.

What is Special about Human Language?

• Human language is a system built to convey the speaker/writer's meaning, and is not just an environmental signal.
• Human language uses an encoding that children can learn quickly.
• Human language is a discrete/symbolic/categorical signaling system.
• Categorical symbols of a language can be encoded as a signal in several ways, like Sound, Gesture, and Images (writing).
• Symbols in a language are invariant across different encodings.
• Human language is a symbolic/categorical signaling system, with encoding that appears to be a continuous pattern of activation and symbols transmitted via continuous signals of sound/vision.

What is Deep Learning (DL)?

• Deep learning is a subfield of machine learning.
• Most machine learning methods use human-designed representations and input features, such as features for finding named entities.
• Machine learning is optimizing weights to make a final prediction.
• Representation learning attempts to automatically learn good features or representations.
• Deep learning algorithms attempt to automatically learn multiple levels of representation, with raw inputs as a starting point.
• Deep learning focuses on different kinds of neural networks, the dominant model family inside deep learning.
• The models have interesting modeling principles (end-to-end) and connections to neuroscience.

Reasons for Exploring Deep Learning

• Manually designed features are often over-specified, incomplete, and take a long time to design and validate.
• Learned Features are easy to adapt, fast to learn.
• Deep learning is flexible and learnable for representing world, visual, and linguistic information.
• Deep learning can learn unsupervised (from raw text) and supervised (with specific labels like positive/negative).
• Deep learning techniques started outperforming other machine learning techniques around 2010.
• Large amounts of training data and faster machines/multicore CPU/GPUs favor deep learning.
• Deep learning has new models, algorithms, and ideas, as well as better learning of intermediate representations.
• Deep learning provides effective end-to-end joint system learning.
• Deep learning provides effective learning methods for using contexts and transferring between tasks.
• Improved performance for deep learning was first seen in image recognition, then speech recognition, then NLP.

What To Teach

• The goal is to teach an understanding of and ability to use effective modern methods for deep learning.
• Key methods used in NLP, like recurrent networks and attention, will be covered.
• Human languages and the difficulties in understanding and producing them will be thought.
• Systems will be built for some of the major problems in NLP, like word similarities, parsing, machine translation, entity recognition, question answering, and sentence comprehension.

Why is NLP Hard?

• NLP is hard because of the complexity in representing, learning, and using linguistic/situational/world/visual knowledge.
• Human languages are ambiguous, unlike programming and other formal languages.
• Human language interpretation depends on real world, common sense, and contextual knowledge.

Deep NLP

• Deep NLP combines ideas and goals of NLP with using representation learning and deep learning methods to solve them.
• There has been improvement in NLP with speech, words, syntax, and semantics.
• There has been improvement with tools, like parts-of-speech, entities, and parsing.
• Applications have been aided by improvements with machine translation, sentiment analysis, dialogue agents, and question answering.