Data Science Process - Chapter 2

Fundamentals of Data Science

• The methodical discovery of useful relationships and patterns in data is enabled by a set of iterative activities collectively known as the data science process.
• The standard data science process includes: Understanding the problem, Preparing the data samples, Developing the model, Applying the model to a dataset, Deploying and maintaining the models
• Examples of reference books relevant to this subject are "Data Science: Concepts and Practice" (Vijay Kotu and Bala Deshpande, 2019) and "DATA SCIENCE: FOUNDATION & FUNDAMENTALS" (B. S. V. Vatika, L. C. Dabra, Gwalior, 2023).

Lecture 2

• This lecture focuses on the data science process.

Chapter 2: Data Science Process

• Data science is an iterative process.
• The objective is to address specific analysis questions.

Data Science Process

• The methodical discovery of useful relationships and patterns in data is enabled by a set of iterative activities.
• The process centers around understanding problems, preparing data, developing models, testing them, and then implementing and maintaining the solutions.

Prior Knowledge

• Involves understanding the problem and context before data collection.
• Gaining prior knowledge.
  • Objective of the problem.
  • Subject area of the problem.
  • Data

Data Preparation

• Preparing the dataset for a data science task (e.g. data exploration approaches, data quality, missing values, data type conversion, transformation, outliers, sampling).
• Requires structured (tabular) data for most algorithms – so if the data is not suitable it needs to be transformed or modified.
  • Data exploration is a critical part of this process.
  • Data quality issues are to be identified.

Data Exploration

• Data exploration methods involve descriptive statistics and visualizations to understand data structure, distributions of values, extreme values and interrelationships within the dataset.

Data Quality

• Ensuring data quality includes data alerts, cleansing, and transformation.
• Data that is collected or stored in well-maintained data warehouses has higher quality than data sourced elsewhere.

Handling Missing Values

• Missing attribute values is a data quality issue that needs to be addressed.
• Methods to deal with missing values, including replacing with mean, minimum, or maximum values.
• Alternatively, records with problematic data can be ignored to create a smaller dataset.

Data Type Conversion

• Input data must be converted to a specific data type suited to the data science algorithm.
• Non-numerical data needs to be converted. This can involve binning, and creating categorical data.

Transformation

• Some data science algorithms require specific data types.
• Normalization is a method used to convert variables into a uniform scale (e.g. from 0–1).

Outliers

• Outliers are anomalies in the data and require special treatment.
• These can be an issue if the data includes incorrect or unusual values.

Feature Selection

• A large number of features in the dataset can negatively impact the performance of a model.
• All attributes need to be evaluated for their relevance to the analysis question

Sampling

• A subset of representative data can support effective data analysis and modeling procedures.
• Sampling reduces processing complexity and improves model build times.

This quiz explores the iterative data science process as outlined in Chapter 2. It covers essential activities such as understanding the problem, preparing data, and model development. Delve into the structured approach that defines data science and its applications.