Untitled

• Module 1 is an introduction to data mining and how to get to know your data.
• The learning outcomes are to define data mining and its applications, recognize data types and patterns, and describe data objects and attribute types.

Chapter 1: Introduction (1.1 - 1.5)

• The topics covered will be: why data mining, data mining definition, data types that can be mined, pattern types that can be mined, technologies used, applications, and major issues.

Chapter 2: Getting to Know Your Data (2.1 - 2.3)

• The topics covered will be: data objects and attribute types, basic statistical descriptions of data, and data visualization.

Why Data Mining

• Data is growing at a rapid rate with increased collection and availability due to digital transformation and automation across all fields.
• Data mining helps find knowledge within this data because the amount of data is overwhelming.
• Data mining is the automated analysis of massive data sets, fulfilling the necessity driven by data overload.

What is Data Mining?

• Data mining has changed names based on its time and field, these alternative names are: knowledge discovery, mining in databases (KDD), knowledge extraction, data/pattern analysis, data archeology, data dredging, information harvesting, business intelligence, etc.
• Data mining extracts knowledge from data no matter its purpose.
• In other words, the extraction of interesting (non-trivial, implicit, previously unknown and potentially useful) patterns or knowledge from data.
• Do not mistake data mining for simple search, query processing or expert systems that do not rely on data.

Knowledge Discovery in Databases (KDD) process

• This includes data cleaning, data integration, data selection, data transformation, data mining, pattern evaluation, and knowledge presentation.
• Data cleaning is to remove noise and inconsistent data.
• Data integration is where multiple data sources may be combined.
• Data selection is where data relevant to the analysis task are retrieved from the database.
• Data transformation is where data are transformed and consolidated into forms appropriate for mining by performing summary or aggregation operations.
• Data mining uses intelligent methods to extract data patterns or knowledge.
• Pattern evaluation identifies the truly interesting patterns representing knowledge based on interestingness measures.
• Knowledge presentation uses visualization and knowledge representation techniques to present mined knowledge to users.

Data Types

• Data types that can be mined include relational databases, data warehouses, and transactional databases.
• Advanced data that can be mined include data streams, sensor data, time-series data, temporal data, sequence data, structure data, graphs, social networks, multi-linked data, object-relational databases, heterogeneous databases, legacy databases, spatial/spatiotemporal data, multimedia databases, text databases, and the World-Wide Web.

What kinds of patterns can be mined?

• Generalization is when you summarize and contrast data characteristics e.g., dry vs. wet region.
• Association and correlation analysis identifies frequent patterns or itemsets, e.g., items frequently purchased together in a supermarket.
  • Association involves correlation vs. causality.
  • A typical association rule is: Bread -> Milk [0.5%, 75%] (support, confidence).
• Classification is supervised learning. A model that classifies objects based on characteristics and learns from examples (training data). Applications: credit card fraud detection and patient diagnoses.
• Cluster analysis is unsupervised learning, where data objects are grouped by similarity. Used in segmenting online shoppers for targeted ads.
• Outlier analysis identifies data objects that do not comply with general behavior and can utilize classification/clustering techniques. Applications include transaction fraud detection and rare event analysis.

Technologies Used for Data Mining

• Machine learning
• Pattern recognition
• Statistics
• Applications
• Database technology
• Visualization
• Algorithms
• High-performance computing

Why Confluence of Multiple Disciplines

• It is due to the tremendous amount of data that require algorithms to be highly scalable.
• It is due to the high-dimensionality of data, like Micro-arrays potentially having tens of thousands of dimensions.
• It is due to the high complexity of data found in datastreams, sensor data, time-series data, social networks, multimedia, software program simulations etc.

Data Mining Applications

• Web page analysis for classification/clustering or PageRank and HITS algorithms.
• Collaborative analysis & recommender systems for targeted marketing and biological/medical analysis
• It classification, cluster analysis (microarray data analysis), biological sequence analysis, and biological network analysis.
• Data mining can be used in software engineering as seen in the IEEE Computer, Aug. 2009 issue.
• Data mining can be performed using SAS, MS SQL-Server Analysis Manager, Oracle Data Mining Tools and invisible data mining systems or tools.

Major Issues in Data Mining

• Mining methodology is one issue. Mining various and new kinds of knowledge, Mining knowledge in multi-dimensional space, Data mining: An interdisciplinary effort, Boosting the power of discovery in a networked environment, Handling noise, uncertainty, and incompleteness of data, Pattern evaluation and pattern- or constraint-guided mining.
• User interaction is an issue. Interactive mining,Incorporation of background knowledge and Presentation and visualization of data mining results Efficiency and Scalability is another. Efficiency and scalability of data mining algorithms, Parallel, distributed, stream, and incremental mining methods are major issues.
• Diversity of data types poses an issue. Handling complex types of data and Mining dynamic, networked, and global data repositories is another major consideration.
• Data mining and society are major issues. Social impacts of data mining, Privacy-preserving data mining and Invisible data mining demand consideration.

Data Objects and Attributes Types

• Datasets are made up of Data objects that represent an entity.
  • Examples of entities: in a sales database - customers, store items, and sales; in a medical database - patients; in a university database - students, professors, and courses.
• Data objects are described/represented by attributes.
• Data objects can also be referred to as samples, examples, instances, data points, or objects.

Attributes

• An attribute is a data field that represents a characteristic/feature of a data object.
• The words attribute, dimension, feature, and variable are often used interchangeably. But:
  • "Dimension" is commonly used in data warehousing.
  • "Feature" is often used in Machine learning literature.
  • Statisticians prefer "variable."
  • Data mining and database professionals commonly use the term attribute, and "attribute" is used in this course.

Attribute Types

• An attribute type is based on the set of possible values it can have. The four types are: Nominal, Binary, Ordinal, and Numeric.
• Nominal Attributes: Each value represents some kind of category, code, or state. Values do not have meaningful order. E.g. Hair color, marital status, occupation, ID numbers, zip codes
• Binary Attributes: A nominal type with two categories/states: 0 or 1. Where 0 means absent and 1 means present. If the states correspond to true and false, binary attributes are referred to as Boolean. E.g. Medical test result or gender
• Ordinal Attributes: Attributes with meaningful order/ranking but the magnitude between successive values is unknown. E.g. Size = {small, medium, large}, grades = {A, B, C, D, F} and army rankings.
• Numeric Attributes: Quantitative, measurable values and can be integer or real values. Numeric attributes can be interval-scaled or ratio-scaled.
  • Interval-Scaled Attributes: Measured on a scale of equal-size units. There is no true zero-point. E.g. temperature in C˚ or F°, calendar dates.
  • Ratio-Scaled Attributes: Numeric with an inherent zero-point. E.g., area, weight, height, length, counts, monetary quantities. Ratio between two data object's attribute can be calculated.

Discrete vs. Continuous Attributes

• Discrete Attribute (Nominal, Binary and Ordinal): Only a finite countable set of E.g., zip codes, profession, or the set of words in a collection of documents and is represented as an integer variable
• Continuous or Numeric Attribute (Ratio and Interval): Real numbers as attribute values. E.g., temperature, height, or weight. Used practically, real values measured/represented using a finite number of digits and is a floating-point variable.

Basic Statistical Descriptions of Data

• Statistical descriptions are used to better understand the data by measuring its central tendency and distribution (variation and spread).
• This includes measuring characteristics like mean, median, mode, midrange, range, max, min, quantiles, outliers, variance, and standard deviation.

Measuring Central Tendency Characteristics

• The Mean is the average and center, calculated as sum of data divided by sample size.
• The trimmed mean is the mean of a dataset when trimming extreme values.
• The Weighted average or Weighted arithmetic mean differs from the regular mean by weighting data based on significance or importance.
• The Median is the middle value after sorting the dataset. The middle value or the sum of the two middle numbers divided by 2, otherwise.
• The Mode is the value that occurs most frequently in the data. There can exist multiple, which is known as Unimodal(1), Bimodal(2), Trimodal(3).
• Midrange= average of the min and max data values.
• Data in real world application tend to have asymmetric data distribution or positively (negatively) skewed data.

Measuring Distribution Characteristics

• This includes quartiles/outliers using boxplots. Quartiles: Q1 (25th percentile) & Q3 (75th percentile)
• Inter-quartile range: IQR = Q3 - Q1, Five number summary: min, Q1, median, Q3, max
  • Boxplot: ends of the box are the quartiles; median is marked; add whiskers, and plot outliers individually
  • Outlier: usually, a value higher/lower than 1.5 x IQR
• Standard Deviation: the square root of variance s2 (or σ2)

Distribution of Data

• It should be visualized.
• Boxplot is a visual graphic display of the five-number summary.
• Histograms: x-axis are values, y-axis represent frequencies
• Quantile plots pair each data value (xi) with the indication, fi. (100 fi) % of data are xi or less. Quantile-quantile (q-q) plots: graphs quantiles of different univariate distributions against one another.
• Scatter plots: pairs of values serve as coordinates for points, plotted on a plane

Data Visualization

• Data visualization aims to make data clear via graphical representation. Helps discover data relationships and it provides visual proof of the data.
• This can be done through pixel-oriented, geometric projection, icon-based, or hierarchical visualization techniques and visualizing complex data/relations.

Data Visualization - Pixel-oriented

• Each window represents an attribute, and each data object is is represented as a pixel on each window. The density/color is proportional to the value.
• Pixels cab be laid out in Circle Segments. This saves space, and it presents connections.

Data Visualization - Geometric projection

• Geometric Visualization consists of geometric transformations and projections.
• This includes:
  • Direct visualization:
  • Scatterplot/scatterplot matrices
  • Landscapes, Projection pursuit
  • Techniqueprojection: Help users find meaningful projections of multidimensional data
  • Prosection views, Hyperslice
  • Parallel coordinates

Data Visualization - Icon-based

• Involves icons, and the visualization of the data values as features of the icons.
• Common methods are Chernoff faces or stick figures.
• Using shape coding, shapes represent certain information.
• Color icons use color to encode information.
• Tile bars use small icons to portray vectors in retrieval.

Data Visualization - Hierarchical

• For data with high dimensionality, it is difficult to visualize dimensions simultaneously.
• Therefore, the visualization partitions all dimensions into subsets in a hierarchical manner.
• This includes Dimensional Stacking, Worlds-within-Worlds, Tree-Map, Cone Trees
• Partitioning of attribute values into stacked subspaces.
• Important attributes have priority to the outer levels if you partition the attribute values into ranges.

Complex data relations

• Can be found through visualization, and most notably tag clouds.
• These present non-numerical data such as text and social networks, where tag sized is representative of its importance.