Data Science Process - Lecture 3

Fundamentals of Data Science

• Course: DS302
• Instructor: Dr. Nermeen Ghazy
• Reference Books:
  • Data Science: Concepts and Practice, Vijay Kotu and Bala Deshpande, 2019
  • DATA SCIENCE: FOUNDATION & FUNDAMENTALS, B. S. V. Vatika, L. C. Dabra, Gwalior, 2023

Lecture 3

Chapter 2 - Data Science Process

Modeling

• Model: Abstract representation of data and relationships within a dataset.

• Simple Rule of Thumb: Example: mortgage interest rate reduces with increasing credit score. This is directional, not fully quantitative.

• Modeling steps in predictive data science:

  • Training data
  • Build model
  • Test data
  • Evaluation
  • Final model

• Association analysis and clustering: Descriptive data science techniques.

• No target variable to predict, hence no test dataset.

• Both predictive and descriptive models have an evaluation step.

• Splitting training and test data sets: Modeling step creates a representative model. The dataset used to create the model is called the training dataset.

• Validity check: The created model must be validated with another known dataset (test dataset or validation dataset).

• Standard rule: Two-thirds of the overall dataset can be used for training, and the remaining one-third for the test dataset.

Application

• Deployment: Stage where models become production ready.
• Model Deployment Stage Deals With:
  • Product readiness
  • Technical integration
  • Model response time
  • Remodeling
  • Assimilation

Knowledge

• Framework for extracting non-trivial information from data.
• Advanced approaches, like data science algorithms, are needed to extract knowledge.
• Data science process begins with prior knowledge and ends with posterior knowledge (incremental insights).
• Spurious irrelevant patterns can arise during the data science process.

Chapter 3 - Data Exploration

• Objectives: Understanding dataset structure, identifying patterns, assessing data quality before in-depth analysis.
• Process involves:
  • Data understanding
  • Data preparation
  • Data science tasks
  • Interpreting the results

1- Data Understanding

• Broad overview of each attribute/variable in the dataset.
• Interactions between attributes, typical values, data point variations, extreme values.

2- Data Preparation

• Anomalies addressed before applying data science algorithms (outliers, missing values, highly correlated attributes.)
• Identifying and removing correlated attributes as certain algorithms perform poorly with them.

3- Data Science Tasks

• Basic exploration can replace the entire data science process.
• Scatterplots reveal clusters in low-dimensional data.
• Simple, visually-based rules can assist predictive models.

4- Interpreting the Results

• Data exploration aids in understanding prediction, classification, and clustering outcomes.
• Histograms are tools for data visualization, assessing numeric predictions, error rate estimations, and more.

Datasets

• Iris dataset: Popular in learning data science.

• 150 observations from three species of Iris.

• Iris Setosa, Virginica, Versicolor. (50 observations each).

• Four attributes (sepal length, sepal width, petal length, petal width).

• Iris dataset readily available in standard data science tools and from the UCI Machine Learning dataset repository.

• All four aspects of the Iris dataset are continuous numeric values measured in centimeters.

• Iris setosa classification is simple (petal length < 2.5 cm)

• Dataset is widely used in data science education for its simplicity, clarity, and representing a common problem that common data science algorithms approach

• Used to represent and illustrate straightforward classification

Types of Data

• Data formats and types vary.
• Data properties determine which operations can be applied (numeric values, ordered labels, threshold-based data).

Descriptive Statistics

• Overall dataset summaries for understanding its characteristics (e.g., average age, median rental price).
• Analyzing key characteristics: central tendency (mean/median), spread (range/variance), overall distributions.
• Two main types:
  • 1-Univariate Explorations: Focuses on a single attribute to summarize its characteristics.
  • 2-Multivariate Exploration: Exploring relationship and interactions between multiple attributes.

Descriptive Statistics - Univariate

• Describes characteristics of single attributes.

Measures of Central Tendency

• Mean: Arithmetic average.
• Median: Middle value in a sorted dataset, less sensitive to outliers.
• Mode: Most frequent value.

Measures of Spread

• Range: Difference between max and min values (simple but sensitive to outliers).
• Deviation, Variance and Standard Deviation: Measures spread encompassing all values, less sensitive to outliers and considering distributions.

Example A (Data Exploration Exercise)

• Example dataset with student exam scores (Mathematics, Science, English, History).
• Steps for analysis:
  • Organizing data in tabular format
  • Calculating measures of central tendency (mean for each subject).
  • Calculating measures of spread (range, standard deviation).

Multivariate Exploration

• Examining relationships and interactions among multiple attributes within the dataset.
• Data points as coordinate points in multi-dimensional space.
• Correlation: Statistically measures relationships between attributes.
• Correlation Coefficient (r): Measures the strength of linear dependence (-1 < r< 1)