Data Mining Techniques

• The course aims to equip students with skills in data mining.

Course Objectives

• Define data mining and its role in the Knowledge Discovery in Databases (KDD) process.
• Differentiate between classification, clustering, regression, and association rule mining.
• Perform data preprocessing techniques such as cleaning, transformation, reduction, and integration.
• Apply exploratory data analysis (EDA) methods to understand and visualize datasets.
• Implement supervised learning techniques, like decision trees, SVMs, and ensemble methods.
• Apply unsupervised learning methods, including k-means, hierarchical clustering, and density-based clustering.
• Predict numerical values and identify trends using regression methods.
• Discover frequent patterns and associations in transactional data using algorithms like Apriori and FP-Growth.
• Analyze temporal data utilizing time series analysis and forecasting models.
• Detect anomalies in datasets for fraud detection and outlier identification.
• Conduct text mining and natural language processing (NLP) tasks such as sentiment analysis and topic modeling.
• Explore web mining techniques for analyzing structure, content, and usage of web data.
• Utilize big data mining tools and frameworks such as Hadoop, Spark, and PySpark.
• Apply appropriate evaluation metrics to assess the performance of classification, clustering, and regression models.
• Interpret and validate discovered patterns to ensure meaningful insights.
• Gain hands-on experience with data mining tools such as Python (Scikit-Learn, Pandas, TensorFlow), R, WEKA, and RapidMiner.
• Analyze large-scale datasets using distributed computing platforms.
• Understand the ethical, legal, and privacy issues in data mining.
• Discuss challenges related to large-scale data mining, such as scalability and noise in data.
• Design and execute data mining workflows to solve practical problems in domains like healthcare, finance, marketing, and e-commerce.
• Present findings in a clear and actionable manner to stakeholders.

Chapter 1: Introduction to Data Mining

• Large-scale data is prevalent in commercial and scientific databases due to data generation and collection technologies.
• The modern approach emphasizes gathering all possible data whenever and wherever feasible.
• Collected data is expected to have value either for the initial purpose or for an unforeseen application.

Commercial Viewpoint on Data Mining

• There is a considerable amount of data being collected and warehoused
• Google has several Peta Bytes of web data
• Facebook has billions of active users
• Computers have become more affordable and powerful.
• Strong competitive pressure exists to provide superior, customized services through Customer Relationship Management.

Scientific Viewpoint on Data Mining

• Data is accumulated and stored at extremely high rates
• NASA EOSDIS archives over petabytes of earth science data per year from remote sensors on satellites.
• Data mining aids scientists in automated analysis of extensive datasets and in hypothesis formation.

Opportunities Enabled by Big Data

• Big data presents opportunities to enhance productivity across various aspects of life.
• McKinsey Global Institute reported big data is a frontier for innovation, competition, and productivity.
• Leveraging big data can unlock substantial annual value in the US healthcare sector
• It can also enhance Europe's public sector administration.
• Consumer surplus can be improved through the use of personal location data.
• There is a need for data-savvy managers to take full advantage of big data.

Data Mining Defined

• It involves the non-trivial extraction of previously unknown and potentially valuable information from data.
• Data Mining consists of exploration and analysis, using automatic or semi-automatic methods, to uncover meaningful patterns.
• Key steps include data preprocessing, data mining, and postprocessing.
• Data preprocessing involves feature selection, dimensionality reduction, normalization, and data subsetting.
• Postprocessing includes filtering patterns, visualization, and pattern interpretation.

Origins of Data Mining

• It combines concepts from machine learning/AI, pattern recognition, statistics, and database systems.
• Large-scale, high-dimensional, heterogeneous, complex, and distributed data can make traditional techniques unsuitable.
• Data mining is a key element of the emerging field of data science and data-driven discovery.

Data Mining Tasks

• Key methods include prediction and description.
• Prediction techniques use existing variables to forecast unknown or future values.
• Description methods aim to discover human-interpretable patterns that summarize the data.

Predictive Modeling: Classification

• It is used to find a model for a class attribute as a function of other attribute values.

Examples of Classification tasks

• Assessing the legitimacy of credit card transactions
• Categorizing land cover using satellite data
• Sorting news stories by topic
• Identifying network intruders
• Determining if tumors are benign/malignant
• Predicting secondary protein structures

Classification Application 1: Fraud Detection

• Goal: Predict fraudulent credit card transactions.
• Approach:
  • Utilize credit card transactions and account-holder information.
  • Label past transactions as fraudulent or fair to create a class attribute.
  • Develop a model to classify transactions.
  • Apply the model to detect fraud by monitoring credit card transactions on an account.

Classification Application 2: Churn Prediction

• Goal: Predict telephone customers likely to switch to a competitor.
• Approach:
  • Analyze transaction records of past and present customers to identify attributes
  • Label customers as loyal or disloyal.
  • Build a model to predict customer loyalty.

Classification Application 3: Sky Survey Cataloging

• Goal: Classify sky objects (star or galaxy) from telescopic images from the Palomar Observatory.
• Approach:
  • Segment images.
  • Measure 40 image attributes (features) per object.
  • Model object class based on features.

Sky Survey Cataloging Data

• Data includes 72 million stars, 20 million galaxies, a 9 GB object catalog, and a 150 GB image database.

Regression

• Predict the value of a continuous variable based on other variables, using a linear or nonlinear dependency model.
• Regression is used for:
  • Predicting sales based on advertising expenditure
  • Predicting wind velocities from temperature, humidity, and air pressure
  • Time series prediction of stock market indices

Clustering

• It involves organizing objects into groups where members are similar or related to each other, and distinct from those in other groups.
• Aims: Reduce intra-cluster distances and maximize inter-cluster distances.

Applications of Cluster Analysis

• Used for custom profiling for targeted marketing
• Also used for organizing related documents for browsing
• Also used to categorize groups of genes/proteins with similar functionality
• Clustering is also used to organize stocks with similar price fluctuations
• Helps to summarize the size of large datasets

Clustering Application 1: Market Segmentation

• Goal: Divide a market into subsets of customers for targeted marketing.
• Approach:
  • Collect customer attributes based on geographical and lifestyle information.
  • Locate clusters of similar customers.
  • Evaluate cluster quality by analyzing purchasing patterns within and between clusters.

Clustering Application 2: Document Clustering

• Goal: Group similar documents based on the significant terms they contain.
• Approach:
  • Identify frequently occurring terms in each document.
  • Establish a similarity measure based on term frequencies.
  • Apply the measure for clustering.

Association Rule Discovery

• Given a set of records each of which contain some number of items from a given collection, produce dependency rules which will predict occurrence of an item based on occurrences of other items.
• Example: {Milk} --> {Coke}, {Diaper, Milk} --> {Beer}

Applications of Association Analysis

• Market-basket analysis for sales promotion, shelf management, and inventory management.
• Telecommunication alarm diagnosis to find combinations of alarms occurring together.
• Medical informatics to link patient symptoms and test results with specific diseases.
• Subspace Differential Coexpression Pattern from lung cancer dataset enhanced with the TNF/NFB signaling pathway which is well-known to be related to lung cancer

Deviation/Anomaly/Change Detection

• This involves spotting significant differences from normal behavior.
• It has many uses, including spotting credit card fraud and detecting network intrusions
• It also is used to identify anomalous behavior from sensor networks for monitoring and surveillance
• It is utilized to detect changes in the global forest cover.

Motivating Challenges

• Scalability
• High Dimensionality
• Heterogeneous and Complex Data
• Data Ownership and Distribution
• Non-traditional Analysis