Data Preprocessing

• Data preprocessing is important to explore the appropriate technique to prepare data for data analytics

Why Data Preprocessing?

• Real-world data is often dirty, meaning it is incomplete, noisy, or inconsistent
• Incomplete data lacks attribute values or contains aggregated data
• Noisy data contains errors or outliers
• Inconsistent data contains discrepancies in codes or names
• Data can be "dirty" due to issues during collection, human/computer error, or transmission errors
• Data preprocessing is important because quality data leads to quality data mining results

Why is Data Preprocessing Important?

• Decisions must be based on the quality of data
• Duplicate or missing data can cause incorrect statistics
• Data warehouses need consistent integration of quality data
• Data extraction, cleaning, and transformation are the majority of the work in building a data warehouse

Multi-Dimensional measure of Data Quality

• Data quality can be measured by accuracy, completeness, consistency, timeliness, believability, value added, interpretability, and accessibility.
• Broad categories include intrinsic, contextual, representational, and accessibility

Major Tasks in Data Preprocessing

• Data cleaning involves filling missing values, smoothing noisy data, identifying/removing outliers, and resolving inconsistencies
• Data integration involves integrating multiple databases, data cubes, or files
• Data transformation includes normalization and aggregation
• Data reduction obtains a reduced representation in volume while producing similar analytical results
• Data discretization is a part of data reduction with particular importance for numerical data

Mining Data Descriptive Characteristics

• Understanding data better involves understanding central tendency, variation, and spread
• Data Dispersion includes median, max, min, quantiles, outliers, variance
• Numerical dimensions correspond to sorted intervals
• Data dispersion is analyzed with multiple granularities of precision using boxplots or quantile analysis
• Dispersion analysis on computed measures involves folding measures into numerical dimensions, using boxplots or quantile analysis

Measures of Central Tendency

• Central tendency measures the location of the middle or center of a distribution
• Common measures include mean, median and mode

Measures of Central Tendency - Mean

• The mean, also known as the average, is calculated by summing all the scores and dividing by the number of scores
• Each observation is equally significant
• The Sample Mean is the sum of all x values divided by n
• The Population Mean is the sum of all x values divided by N
• The advantage of mean is that the value is sensitive to any change within the observations
• The disadvantage of mean is that the value is very sensitive to outliers

Weighted Mean

• A weighted mean can be calculated using a weighting factor
• Population can be the weighting factor:
  • Population A $23,000 100,000
  • Population B $20,000 50,000
  • Population C $25,000 150,000

Measures of Central Tendency - Median

• The median is the value that divides the distribution into two equal-sized groups
• If the number of observations is odd, the median is the middle value
• If the number of observations is even, the median is the average of the two middle values
• Median = Not affected by extreme values at the end of distribution of outliers

Measures of Central Tendency - Mode

• The mode is the most frequently occurring value in the distribution
• This can only be used with nominal data
• The mode can be located quickly

Characteristics of Data

• Not all data is the same
• There are limitations as to what you can and can’t do with a data set, depending on the characteristics of the data

Continuous vs. Discrete Data

• Continuous data has a value (i.e. real numbers)
  • EG: 1, 1.43, and 3.1415926
  • Distance, tree height, amount of precipitation, etc
• Discrete data consist of discrete values, and the numbers defined must be whole numbers
  • EG: 1, 2, 3

  • of vegetation types

Grouped vs. Individual Data

• Individual data concerns the effects of grouping the data
• EG: Family income value vs classes or groupings
• EG: Elevation 1000m vs classes or groupings
• In grouped data, the raw individual data is categorized into several classes, and then analyzed
• Grouping may introduce significant distortion
• Grouping reduces the amount of information

Scales of Measurement

• Data is the plural of a datum that is generated by recording measurements
• Measurements involve the categorization of an item i.e assigning an item to a set of types, when the measure is qualitative
• To give a quantitative measurement use numbers

Type of Data

• Data used in statistical analyses can be divided into four types: nominal, ordinal, interval, and ratio
• Nominal scale data are broken down into categories of names or types.
• Dichotomous nominal data has just two types
  • Yes/No, Female/Male etc
• Multichotomous data has more than two types
  • Vegetation types, soil types, counties, eye color, etc.
• Nominal scale is not a scale because categories cannot be ranked, or ordered in any way

Ordinal Scale

• Data can be categorized and placed in order.
• Assigned a relative importance value
• Star system restaurant rankings - 5 stars > 4 stars; 4 stars > 3 stars etc

Interval Scale

• Takes the notion of ranking items in order one step further, the distance between adjacent points on the scale are equal
• EG: Fahrenheit Scale - degrees are equal, but there is no absolute zero point) subtract degrees e.g 10 to 90, you cannot create ratios of multiply values

Ratio Scale

• This is similar to the interval scale, but with the addition of having a meaningful zero value, which allows us to compare values using multiplication and division operations
• precipitation, weights, heights, etc
• EG - Rain - 2 inches is twice as much as 1 inch
• EG - Age - 100 years is twice as old as 50 years

Which is Better: Mean, Median, or Mode?

• Mean is selected by default, and the key advantage of the mean is that it is sensitive to any change.
• The key disadvantage is that observations can be outliers.
• The mean should be interval data
• Median can be determined for ordinal data
• Mode can be used with nominal, ordinal, interval, and ratio data
• The mode can b used with nominal data

Data Cleaning

• Data cleaning is a big problem in data warehousing
• Importance the number one problem in data warehousing
• Data cleaning tasks Fill:
  • Identify
  • Correct
  • Resolve inconsistencies, redundancy caused by data integrations

Handling Missing Data

• Ignore the tuple, when classifying not effective because of the missing values per attribute varies considerably
• Fill in manually, too time consuming and infeasible
• Fill in automatically with a global constant
• Imputation:
  • Delete observations
  • Hot deck,
  • Cold deck,
  • Predictive imputation

Imputing the Missing Data

• Delete missing observations, can be ok if the data is small
• Cold Deck Imputation Fill:
  • Measure of central tendency
• Hot Deck Imputation:
  • Can identify more than one case so can average the values out that it identifies
• Distributed- Based Imputation:
  • Tries to capture the observed of the distribution of the data
• Statistical Imputation
  • Considers the missing value
• Predictive Imputation:
  • Classifier model for the missing mechanism

Noisy Data

• Noise: random error or variance in a measured variable
• Requires data cleaning
  • Duplicate records
  • Incomplete data

Handling Noisy Data

-Binning partition data and then smooth by bin means, smooth boundaries, etc.

• Smoothing by fitting the data into regression functions
• Clustering and then detect and remove outliers deal with potentially outliers

Simple Discretization Methods: Binning

• Equal width with uniform grid
• Outliers my dominate presentation
• Skewed data is handled well
• Managing categorical data can be tricky