Business Intelligence: Data Mining Fundamentals

• Study notes covering various aspects of business intelligence, data mining, and statistical analysis for a midterm review

Business Intelligence & Data Mining Fundamentals

• Data mining discovers correlations, patterns, and trends from large data sets.
• It consolidates machine learning, pattern recognition, statistics, databases, and visualization techniques.
• The goal is to discover actionable patterns and rules.

Evolution and Importance of Data Mining

• Data mining is now essential due to competitive pressures, increased data production/storage, affordable computing power, and user-friendly software.
• Easy to misuse if you don't apply it carefully
• Cross-Industry Standard Process for Data Mining (CRISP-DM) offers a standard process.
• CRISP-DM was developed by Daimler AG, SPSS, and NCR, and is a leading methodology in the industry.

CRISP-DM Process Stages

• Business Understanding: Define objectives and translate them into data mining problems.
• Data Understanding: Collect data, assess quality, and perform exploratory data analysis (EDA).
• Data Preparation: Cleanse, prepare, and transform the dataset for modeling.
• Modeling: Apply and calibrate modeling techniques to optimize results, additional data preparation may be required.
• Evaluation: Evaluate models for effectiveness.
• Deployment: Use models, can be simple report generation or complex implementation.

The Importance of Human Oversight in Data Mining

• The automation is no substitute for human judgment.
• Humans are required in every stage
• Black box software algorithms can be dangerous if not handled carefully.

Common Fallacies in Data Mining

• Fallacy 1: Data mining tools can automatically solve all business problems.
  • Reality: It's a process integrating business objectives.
• Fallacy 2: Data mining processes are autonomous.
  • Reality: Requires significant intervention and continuous evaluation.
• Fallacy 3: Data mining quickly pays for itself.
  • Reality: Return rates vary based on factors like startup costs and data preparation.
• Fallacy 4: Data mining software is easy to use.
  • Reality: Requires subject matter knowledge.
• Fallacy 5: Data mining identifies the causes of business problems.
  • Reality: It uncovers patterns, but humans identify causes.
• Fallacy 6: Data mining automatically cleans data in databases.
  • Reality: Often uses legacy systems with data needing preprocessing.
• Fallacy 7: Data mining will always yield positive results.
  • Reality: Not guaranteed, but can sometimes provide actionable insights.

Data Mining Tasks

• Description provides patterns and trends.
• Estimation uses data to estimate changes in numerical target variables.
• Prediction forecasts future outcomes.
• Classification categorizes data.
• Clustering groups similar data points.
• Association identifies data attributes that go together.

Supervised vs. Unsupervised Learning

• Supervised: Has a target variable and known categories (e.g., estimation).
• Unsupervised: Exploratory, no target variable (e.g., clustering).

Types of Data Mining Tasks

• Supervised (Directed): Includes description, estimation, classification, and prediction.
• Unsupervised (Undirected): Includes clustering and association.

Data Processing & Cleaning

• Data must be processed by cleaning because it is often incomplete, noisy, and from legacy databases.
• The values are old, not relevant, missing etc
• Data needs cleaning and transformation for data mining purposes

Handling Missing Data

• May contain missing or erroneous values, data in a unsuitable form for data mining, obsolete or redundant fields, outliers, values not aligned with policy or common sense.
• Data preparation often takes 60% of the effort.
• Deleting records is not always best as it can create bias and eliminate valuable information.
• Use method of replace with mode.
• Using method replace with mean.
• Handling missing data by consulting domain experts.
• Use Data imputation where a statistical model is used to derive the most realistic value for missing entry

Identifying Outliers

• Look for outliers near extreme data limits.
• Outliers can represent errors in data entry.
• Neural networks benefit from normalized data.
• Histograms can examine numeric field values.
• Scatter plots may determine outliers as well

Data Transformation

• Variables should have same ranges
• Important to normalize data
• Min-max normalization values fall between 0 and 1.
• Z-score standardization values range from -4 to 4,

Normality & Data Transformation

• Skewness is when data is not symmetric to the mean.
• Kurtosis is a measure of whether the data is heavy-tailed or light-tailed relative to a normal distribution.
• Transformations ensure data distribution is closer to normal.
• Z-score standardization can identify outliers.

Using Interquartile Range (IQR)

• Determine method for identifying outliers
• Data is divided into four quartiles.
• Defined location for outlier
• Use of flag/dummy variables and transforming data

Considerations for ID Fields

• Create an ID field if one doesn't exist.
• Do not use as a variable in the analysis

Confidence in Estimates

• Accuracy varies
• Sampling error exists between sample estimate versus variable
• Aoint estimates lack confidence measures.
• Analogies with darts highlight lack of precision.

Confidence Interval Estimation

• Margin of error is how much room there is for error.
• Margin of error relies on sample size and sample variability and level.
• Reducing margin of error relies on sample size and sample variability and level.
• With a larger sample, the more preciise the sample will be.

Hypothesis Testing

• Method for checking claims based on data.
• Null hypothesis is the status quo number.
• Alternative hypothesis is an alternative claim.
• Reject the null hypotheis id sufficient.
• You do not reject the null hypthesis, there is not sufficient evidence to conclude it is correct.

Type I and Type II Errors

• Type 1: Reject H0 when it is actually true (false positive).
• Type 2: Failing to reject H0 when it is actually false (false negative).

Types of Hypothesis Tests

• Left-tailed test determines if H0:μ ≥ μ0 vs. Ha:μ < μ0.
• Right tailed if H0:µ ≤ μ0 vs. Ha:μ > μ0.
• Two tailed if H0:μ = μ0 vs. Ha:μ ≠ μο.
• Test statistic measures deviation from hypothesized mean.

P-Value Interpretation

• Probability of observing a test statistic as extreme as the one calculated, assuming H0 is true.
• A decision is made based off the P score.
• Structure of parameters can be determined with sufficient evidence.

Hypothesis Testing vs. Exploratory Data Analysis (EDA)

• Hypothesis testing is confirmatory.
• Uses statistical models to confirm or reject a hypothesis.
• EDA delves into data and examines relationships.
• It then develops initial associations.

Statistical Analysis Steps

• Select population parameter.
• Set and form Ho and Ha.
• Decide the a.
• Get sample and information.
• Cacalute t(data).
• Chose correct method for finding P.

Sample Analysis

• Test if number of variables differ, with significance level a=0.05
• If there's correlation, null hypothesis has to equal zero, and is rejected if value <0.05

Regression Analysis and Multicollinearity

• Split is helpful or not
• Follow linearity and distribution
• Check error terms
• Multicollinearity determines independence

Variable Selection

• Regression means to address included test scores
• There should be a consistent addiiton

Model Direction

• Direction depends on if variable can be accepted.

Essential Steps for Testing Regression

• Population parameter needs to be obtained.
• Form Ho and Ha, it is then correlated to value and sample.

Overall Statistical Analysis

• Samples and variables
• Check the range
• Make analysis of the model by checking the p score, for accuracy

Classification Task

• Most common data mining
• Several examples exist like determining bank application, what class with education,
• In classification there is is a class of data