Module 1:Data Mining: Chapter 1 (introduction) Chapter 2 (Getting to Know Your Data)

• DS472 focuses on Data Mining.
• Module 1, Chapter 1 is an introduction to Data Mining.
• Module 1, Chapter 2 addresses getting to know your data.

Learning Outcomes

• A key learning outcome is to define data mining and its applications.
• Students will recognize the kinds of data and patterns that can be mined.
• Another aim is to describe data objects and attribute types

Introduction to Data Mining

• Data mining is important because of the rapid growth of data
• Digital Transformation and Automation have increased data availability and collection.
• Data mining is applicable across all fields including business (e-commerce, transactions, stocks), science (remote sensing, bioinformatics, scientific simulation), and society (news, digital cameras, YouTube).
• Data mining helps finding knowledge within the large amounts of available data.
• Data mining offers automated analysis of massive data sets.
• Data mining has various names dependent on time and field, like KDD and business intelligence.
• Data mining extracts knowledge from data regardless of the intended use.
• In essence, it extracts interesting, non-trivial, implicit, previously unknown, and potentially useful patterns or knowledge.
• Note that overstating data mining by including similar tasks like simple query processing or expert systems usage is easy.

Knowledge Discovery Process

• Data cleaning removes noise and inconsistent data.
• Data integration combines multiple data sources.
• Data selection retrieves relevant data for the analysis task from the database.
• Data transformation consolidates data into appropriate forms for mining.
• Data mining extracts data patterns or knowledge using intelligent methods.
• Pattern evaluation identifies interesting patterns representing knowledge.
• Knowledge presentation uses mined knowledge visualizations.

Types of Data

• Data can be mined.
• This includes Relational, warehouse, and transactional databases
• Also included are Data streams, sensor data, time-series, temporal data, sequence data (including bio-sequences), structure data, graphs, social networks, multi-linked data, object-relational and heterogeneous databases
• Spatial, spatiotemporal, multimedia, text databases and The World-Wide Web are mineable data.

Patterns that can be mined

• The data mining techniques used in generalization help summarizing and contrasting data characteristics, for example, dry vs. wet regions
• Association and Correlation Analysis identifies frequent patterns (or frequent itemsets), such as items frequently co-purchased at a supermarket
• An association rule: Bread -> Milk [0.5%, 75%] offers quantifiable support and confidence metrics
• There are relationships of association, correlation vs. causality.
• Not all strongly correlated items are correlated.
• Building a model able to classify on object characteristics is possible via Classification (supervised learning).
• Model learning uses training data.
• Diagnoses and credit card fraud can be detected.

More on Patterns that can be Mined

• In Cluster Analysis (unsupervised learning) data objects can by grouped when similar to each other
• Targeting ads to specific groups is possible, like news readers.
• Outlier Analysis detects data objects non-compliant with the general behavior of the data
• The identification of shared and disparate object characteristics helps defining outliers.
• Noise is different to an outlier
• Analysis can use classification or clustering techniques.
• Rare event analysis and transaction fraud detection are examples of usage.

Relevant Technologies

• Machine Learning, pattern recognition and Statistics are relevant technologies for data mining.
• Visualization algorithms, Databases and high-performance computing all enable data mining.

Multiple Disciplines in Data Mining

• Having scalable algorithms enables handling tera-bytes of data.
• Micro-arrays can have tens of thousands of dimensions.
• Highly complex data includes Data streams and sensor data, time-series and temporal data, sequence data, structured data, graphs, social networks, multi-linked data, heterogeneous databases, legacy databases, spatial, spatiotemporal, multimedia, text and Web data and programs.

Data Mining Applications

• Applications include from web page classification, clustering to PageRank & HITS algorithms.
• Web page analysis, analysis & recommender systems, basket data analysis to targeted marketing are also examples.
• Biological and medical data analysis, software engineering is possible
• There are many existing data mining tools (e.g., SAS, Oracle Data Mining Tools).

Issues in Data Mining

• Data mining algorithms must be efficient and scalable.
• Mining methods must be parallel, distributed, stream-oriented, and incremental.
• Data must be diverse and complex.
• Mining must be able to handle dynamic, networked, global data repositories.
• Social impacts, privacy-preserving techniques, and invisible mining are all important.

Data Objects and Attributes

• Data sets are made up of data objects.
• An entity is represented by a sales database, the objects may be customers, store items, and sales
• Patients in medicine, students, and professors at university.
• Data objects are described or represented by attributes.
• These data objects are referred to as samples, examples, instances, data points, or objects.

Important definitions

• An attribute is a data field that represents a characteristic or feature of a data object.
• interchangeable nouns attribute/dimension/feature/variable
• Dimension is most used in warehousing
• Machine learning literature tends to use the term feature
• Statisticians most likely use variable
• Data mining and database professionals commonly use the term attribute.

Attribute Types

• An attribute type is set by the type of possible values.
• The four types: Nominal, Binary, Ordinal and Numeric (interval/ratio scaled)
• Nominal attributes are categories, codes, or states without meaningful order like hair color, marital status
• Binary attributes only have two categories/states, like 0/1 (absent/present) and Boolean
• Ordinal attributes have ordered values but unknown magnitude between these, e.g Size or Grades
• Numeric Attributes are quantities represented by integer/real values.

Categorizing numeric attributes

• Interval-Scaled attributes are scales of equal size units without true zero-point like temperature in C˚or F˚
• Ratio-Scaled Attributes are numeric with zero-point.
• Ratio between two data object's attribute can be calculated from area, weight, height, any count and monetary quantities.
• Discrete Attribute (Nominal are generally these
• They have a countable set of values, like zip codes or profession.

Continuous Variables

• Attribute (Ratio and interval) have real numbers as values.
• They can be measured practically using number of digits
• They are typically represented as floating-point variables
• Statistical Descriptors help understand central tendency and distribution.

Basic Statistical Descriptions of Data

• This process measures data's central tendency, dispersion, spread and variation
• To measure a tendency one needs to know Mean, Median, Mode and Midrange.
• To measure dispersion one needs know Range, max, min, variance standard deviation, quantiles, outliers.

Working Out The Tendency

• Mean works by dividing all values by the sample size.
• Weighed average uses weight to reflect significance.
• Trimmed means values can be calculated from cleaned data.
• Sorting to get the median sorts the data
• Then the middle value is found if the data size is odd or the two middle numbers are averaged otherwise
• This computation method is expensive for big data so approximate values are necessary.

Describing the median in detail

• Median assumptions are that data are grouped into x-values and that frequencies are known.
• First, find the median interval.
• By interpolation compute

median = L1 + (n / 2 − (∑ freq )l/ freqmedian) width where L1 = lower boundary of interval N = total number of sample values, (∑ freq) sum of frequencies lower than interval. freqmedian = frequency of interval width = width of the median interval. Mode Value is the most frequent.

More tendency data characteristics

• Unimodal sets have the same hightest values and most frequency i.e bimodal and trimodal as well
• One value with highest frequency = Unimodel, two are bimodal and three are trimodal
• An approximate unimodel calculation involves using mean − mode = 3 × (mean − median)
• Midrange uses minimum and maximum values
• Compute this easily like any SQL using aggregate functions i.e max() and min().
• If symmetric distribution, the tendencies will be centred

Dispersion basics

• World application data tends to have asymmetric
• Data distributions are skewed
• Quartiles are 25th (Q1), and 75th (Q3).
• Inter-quartile range equals Q3 – Q1.
• Five number summary equals min, Q1, median, Q3, max.
• Boxplot illustrates ends of the box at the quartiles.
• Whisker plot illustrates outliers plot outliers individually higher or lower values than 1.5 x IQR
• Standard deviation includes sample: s, population: σ.
• Scalable algebraic computations measure variance.

Common formulas

• Variance: σ ̂2 = (1/N) (Σ__(i=1..n) (x_i − μ)^2) = (1/N) Σ__(i=1..n) x_i^2 − μ^2
• The root represents standard deviation
• Popular visualization plots help visualize such distributions.
• The five number summary creates boxplots
• With Histograms the x-axis values while y-axis shows the frquency.

More on graphic visualization

• Quantile plots pair values with a percentage to show data amount i.e x,, f.
• Quantile graphs univariate distribution against the corresponding ones i.e q-q with points/ co-ordinates against the place with visualization.
• Data visualization has graphical representations help communicate it clearly and effectively.

Data visualization can

• This technique is used extensively across many applications like business operations, the tracking of progress of tasks, and used to discover data relationships that are otherwise not easily observable.
• There is visual proofs of computer representations.
• Pixel visualization and geometric projection are available techniques for graphics Pixel representation is proportional to the value. A feature can represent one attribute. Space filling also saves space Other techniques such as as geometric transformations and projections can enhance the view. Additionaly, direct and scattered visualizations can be used.

Icon-based visualization

• Matrices, pursuit techniques and hyperslices can also be leveraged
• General techniques include shape/color codes and small representation

Hierarchical Vizualizaton

• Can partition dimensions in sub-data set to easier visualize simultaneously for large and high dimensionality data
• Dimensional Stacking, Worlds-within-Worlds, Tree-Map, Cone Trees and InfoCube are relevant methods

Hierarchical visualization techniques

• Dimensional Stacking involves partition n-dimensional attribute space in 2-D subspaces which are 'stacked' into each other
• Adequate data can be ordinal - low dimensionality to show
• High quantity of data with over nine dimensions is complicated.
• Mapping dimensions appropriately makes the display more effective.

Key Readings

• Data Mining: Concepts and Techniques, (introduction)
• Knowing Your Data
• Data Mining: Practical Machine Learning Tools and Techniques Chapter concept