Data Science Process Chapter 2

Fundamentals of Data Science

• The methodical discovery of useful relationships and patterns in data is enabled by a set of iterative activities known as the data science process.
• The standard data science process includes:
  • Understanding the problem
  • Preparing data samples
  • Developing the model
  • Applying the model to a dataset to see how it works in the real world
  • Deploying and maintaining the models

Reference Books

• Data Science: Concepts and Practice, by Vijay Kotu and Bala Deshpande (2019)
• DATA SCIENCE: FOUNDATION & FUNDAMENTALS, by B. S. V. Vatika, L. C. Dabra (2023)

Lecture 2

• Covers the data science process.

Chapter 2: Data Science Process

• The data science process is a generic set of steps.
• The fundamental objective is to address the analysis question.
• Algorithms used to solve business questions can include decision trees, artificial neural networks, or scatterplots.
• Software tools range from custom coding to RapidMiner, R, Weka, SAS, Oracle Data Miner, and Python.

Data Science Process

• A process model with six phases that naturally describes the data science life cycle.
• Includes phases like:
  • Business Understanding
  • Data Understanding
  • Preparing the Data
  • Modeling
  • Evaluation
  • Deployment

Prior Knowledge

• Refers to information already known about a subject.
• Helps define the problem, business context, and necessary data. Key parts include:
  • Objective of the problem
  • Subject area of the problem
  • Data needed to solve the problem.

Prior Knowledge: Objective of the Problem

• The data science process starts with a need for analysis, a question, or a business objective.
• It is the most important step; without a well-defined problem, finding the right dataset and algorithm is impossible.
• Revisions to assumptions, approach, and tactics are common during the process.

Prior Knowledge: Subject Area of the Problem

• The data science process uncovers hidden patterns and relationships between attributes.
• Identifying false or spurious signals (patterns) is essential.
• Knowing the subject matter, context, and business process generating the data is crucial.

Prior Knowledge: Data

• Understanding the data collection, storage, transformation, reporting, and usage is essential.
• Surveying existing data helps to narrow down the need for new data. Specific data quality factors include
  • Quality
  • Quantity
  • Availability
  • Gaps
  • Business questions

Data Terminology

• Dataset: A collection of data with a defined structure.
• Data frame: A table structure with rows and columns (headers)
• Data Point (Record, Object, Example): A single instance within a dataset (a single row).

Data Preparation

• Data preparation is the most time-consuming step in data science process.
• Data is rarely in the suitable format, so transformation is required.
• Tabular format with records in rows and attributes in columns is typical for most data science algorithms.

Data Preparation steps

• Data Exploration
• Data Quality
• Handling missing value
• Data type conversion
• Transformation
• Outliers
• Feature selection
• Sampling

Data Exploration

• Provides basic understanding of data.
• Involves computing descriptive statistics and visualization.
• Exposes data structure, value distribution, extreme values, and inter-relationships.
• Use of statistics such as mean, median, mode, standard deviation, and range to describe data. A scatterplot can help visualize data.

Data Quality

• A continual concern in data collection, processing, and storage.
• Data accuracy and quality is essential.
• Data warehouses are used to store and maintain the data quality. Common quality techniques include:
  • Removing duplicates
  • Identifying and handling outliers
  • Standardizing attribute values
  • Handling missing values.

Handling Missing Values

• A common data quality issue is missing attribute values.
• Methods exist for dealing missing values:
  • Replacing with derived values—e.g. mean, minimum, or maximum
  • Ignoring the records with missing values in the data

Data Type Conversion

• Data attributes can be numeric (interest rate), integer numeric (credit score), or categorical.
• Categorical data may need to be converted to numeric for model applications, including linear regression models.
• A technique called binning converts numeric ranges to categorical values based on bins.

Transformation

• Some data science algorithms (e.g., k-nearest neighbor) require numeric and normalized attributes.
• Normalization converts values to a consistent scale (often 0 to 1) to prevent attributes with larger values to dominate comparisons.

Outliers

• Outliers are anomalies in a dataset; they need to be understood and addressed.
• They can arise from data errors (incorrect entry) or valid data captures (very high income for example)
• Outliers require special treatment depending on the data science application.

Feature Selection

• A large number of attributes complicates models and can significantly degrade performance
• Not all attributes are important for prediction of interest
• Feature selection reduces the model complexity, boosts performance, and avoids "curse of dimensionality".

Sampling

• A subset of records (representative samples) from the original data is selected.
• Sampling reduces the amount of data needing processing, speeding up data science tasks.
• The use of representative samples allows for data insight gathering.
• The risk of sampling is that it could impact the relevance of the model, but benefits often outweigh the risk.