Text Mining: Definitions and Basics

TM Lecture 1: Definition and Basics of Text Mining

• Text mining is a knowledge-intensive process that extracts useful information from unstructured textual data.
• Text mining aims to uncover hidden patterns, relationships, and insights within large volumes of text.
• Data mining and text mining both aim to discover novel and useful patterns and are semi-automated processes.
• Data mining often uses structured data, e.g., data in databases like SQL tables.
• Text mining typically deals with unstructured data like Word documents, PDFs, XML files, plain text, etc.
• The text mining workflow involves imposing structure on unstructured data and then mining the structured data using algorithms.
• Approximately 85-90% of corporate data is unstructured, including emails, reports, and documents.
• Unstructured data doubles in size approximately every 18 months.
• Tapping into unstructured data is essential for maintaining competitiveness.
• Text mining enables the semi-automated extraction of actionable insights from unstructured sources.

Text Mining Process

• The text mining process includes 5 key stages: text preprocessing, feature extraction, feature selection, pattern discovery, and use of results.
• Text preprocessing transforms unstructured data into a structured format suitable for analysis and includes tokenization, stopword removal, stemming/lemmatization, and noise reduction.
• Tokenization breaks text into words or phrases.
• Stopword removal eliminates common words like "the" and "and".
• Stemming/lemmatization reduces words to their root forms.
• Noise reduction eliminates irrelevant characters like punctuation.
• Feature extraction extracts key elements such as words, terms, and concepts.
• Extracted features should represent documents in a way that captures their semantic meaning.
• Feature selection selects the most important features for analysis, using machine learning/statistical methods like document clustering and classification.
• Pattern discovery analyzes co-occurrence patterns across documents to discover relationships using techniques like classification and clustering.
• The insights derived from text mining are applied to various domains for decision-making, trend analysis, and automation.

Applications & Today's Example Text Mining

• Text mining is particularly valuable in text-rich environments, including law, academic research, finance, medicine, biology, technology, marketing, and email communication.
• Specific uses include analyzing court orders, extracting insights from research articles, summarizing quarterly reports, processing discharge summaries, understanding molecular interactions, analyzing patent files, interpreting customer feedback, and spam filtering.
• Additional email communication applications include email prioritization/categorization and automatic response generation.
• A sample NLP task involves building a workflow to classify free-text documents by reading and preprocessing the text, transforming it into numerical representations using techniques like TF-IDF or word embeddings, and training a classifier to assign labels.
• Additional tasks include sentiment analysis, visualization (e.g., word clouds, scatter plots), and document clustering.
• Text mining transforms unstructured data into structured formats, enabling advanced analysis and discovery of patterns and trends.
• Text mining is a powerful tool for extracting actionable insights from large-scale textual data.

TM Lecture 2: Importance and Overview of Text Preprocessing

• Text mining heavily relies on preprocessing to convert raw, unstructured data into structured formats suitable for analysis.
• Preprocessing techniques for text differ significantly from traditional data mining due to the inherent complexity and variability of textual data.
• Raw text is often messy, with typos, special characters, and inconsistent formatting.
• Text preprocessing ensures uniformity and consistency for machine learning models and improves accuracy, efficiency, and processing speed.
• The text preprocessing pipeline typically includes text cleaning, tokenization, sentence segmentation, normalization, stopword removal, and stemming & lemmatization.
• Text cleaning removes unwanted elements like HTML tags, special characters, numbers, and noise such as URLs, emojis, or irrelevant symbols.
• Tokenization splits text into smaller units, such as words or sentences.
• Sentence segmentation divides text into individual sentences and is useful for tasks requiring sentence-level analysis.
• Normalization converts words to standard forms (e.g., lowercase conversion, removing diacritics).
• Stopword removal filters out common, unimportant words (e.g., "the", "and", "is") to reduce noise and improve focus on meaningful content.
• Stemming & lemmatization reduces words to their root forms; stemming truncates words.

Challenges, Goals and Categories of Text Preprocessing

• Language-specific issues require tailored preprocessing techniques for different languages.
• Noise in text, such as social media texts, OCR outputs, and scanned documents, often contains errors or inconsistencies.
• Variability in sentence structure, punctuation, abbreviations, and slang complicates preprocessing.
• Computers process text as numerical values (binary or Unicode).
• Challenges in how machines read text include encoding mismatches (e.g., UTF-8 vs. ASCII) and character corruption or misinterpretation of special symbols.
• The goals of preprocessing are to reduce redundancy and inconsistency to ensure clean, uniform data for analysis, convert text into a machine-readable format, and improve the efficiency of NLP pipelines.
• Categories of preprocessing techniques include task-oriented approaches and formal frameworks.
• Task-oriented approaches focus on specific tasks such as extracting titles, authors, or metadata from documents and include extracting keywords from research papers or identifying entities in news articles.
• Formal frameworks include classification schemes, probabilistic models, and rule-based systems.

Combining Techniques and Task Orientated Approaches

• Preprocessing techniques are often combined for different applications.
• Algorithms used in preprocessing are general-purpose and can address multiple problems simultaneously.
• Task-oriented approaches include preparatory processing for converting raw input (e.g., PDFs, scanned pages) into a text stream.
• Examples of general-purpose NLP tasks include tokenization, POS tagging, syntactical parsing, and shallow parsing.
• Problem-dependent tasks include examples like text categorization and information extraction (IE).
• Tokenization splits text into smaller units and faces challenges in identifying sentence boundaries and handling punctuation.
• POS tagging assigns grammatical categories to words.
• Syntactical parsing analyzes sentence structure.
• Shallow parsing provides partial analyses of sentences for speed and robustness.
• Parts of speech concepts have historical roots in ancient linguistic traditions and include noun, verb, pronoun, preposition, adverb, conjunction, participle, and article.
• Text categorization tags documents with predefined concepts or keywords.
• Information extraction extracts relevant information and presents it in a structured format, with a distinction from information retrieval.
• IE focuses on pinpointing specific information, while IR retrieves broader sets of documents matching a query.

TM Lecture 3: Core Text Mining Operations

• Text mining involves discovering patterns, relationships, and insights from unstructured textual data using core operations.
• Pattern-discovery algorithms are central to text mining and focus on uncovering meaningful patterns in document collections.
• Distributions identify the proportion of documents labeled with specific concepts.
• Frequent and near-frequent sets find sets of concepts that frequently co-occur in documents.
• Associations discover relationships between sets of concepts.
• Concept selection and proportion calculations help analyze subsets of documents based on their labeling.
• Concept selection refers to selecting subsets of documents labeled with specific concepts.
• Concept proportion calculates the fraction of documents labeled with certain concepts.
• Conditional concept proportion determines the proportion of documents labeled with one concept that is also labeled with another.
• Understanding how concepts are distributed across documents is essential for analysis.
• Concept proportion distribution measures the proportion of documents labeled with each concept in a set.
• Conditional concept proportion distribution measures the proportion of documents labeled with a set of concepts that are also labeled with another concept.
• Identifying frequent and near-frequent sets helps uncover recurring patterns.
• Frequent sets are sets of concepts that appear together in a minimum number of documents.
• Near-frequent sets are sets of concepts that frequently co-occur but not as often as frequent sets.

Associations, Patterns and Knowledge for Text Mining

• Association rules reveal relationships between sets of concepts.
• Association rules indicate that the presence of one set of concepts implies the presence of another set.
• Maximal associations are specialized associations relevant to one concept but not another.
• Interesting patterns highlight significant differences or distances between distributions.
• Concept distribution distance measures the distance between two concept distributions.
• Concept proportion distance measures the difference between two distributions at a particular point.
• Temporal analysis uncovers trends and anomalies in document collections.
• Trend analysis analyzes concept distribution behavior over time.
• Ephemeral associations are temporary relationships between concepts over a fixed time span.
• Deviation detection identifies anomalies that do not fit a defined standard case.
• Background knowledge enhances the accuracy and relevance of text mining results.
• Domain ontologies define a set of all classes of interest and their relationships within a domain and using examples like WordNet for the English language and Gene Ontology for biological processes.
• Domain lexicons include the names of domain-specific concepts and their relations.
• Domain lexicons are used for normalizing concept identifiers and extracting semantic relationships.
• Background knowledge can be integrated using constraints to limit pattern abundance and create meaningful queries and hierarchical representations to create consistent hierarchies for concepts.

Text Mining Query Languages

• Query languages enable users to isolate interesting patterns and perform advanced analyses.
• KDTL (Knowledge Discovery in Text Language) is used for performing queries to isolate interesting patterns.
• KDTL supports pattern-discovery queries, constraints, and auxiliary filtering.
• KDTL query examples include association queries and frequent set queries.
• Association queries are used for finding associations between sets of concepts.
• Frequent set queries are used for identifying frequent sets of concepts.
• Query interface includes user-friendly tools, such as graphical interfaces for building and executing queries, and direct access via command-line access for constructing ad hoc queries.