Data Science: Overview, Roles and Tools

Data Science Overview

• The primary goal of Data Science is to extract insights from data
• Data science uses statistics, computer science, and domain knowledge to extract insights from data to support data-driven decisions

Fields Related to Data Science

• Data science integrates statistics, programming, machine learning, and domain expertise.
• Astrology is not a core component of Data Science

Roles in Data Science

• A Data Scientist typically does not write fiction

Data Science Life Cycle

• Problem formulation is the first step

Collaboration Tools

• Google Colab facilitates real-time collaboration in notebooks

Digital Skills Essentials (DSE)

• The passing score is 60%.

Python Libraries for Data Science

• Seaborn is a Python library suited for statistical data visualization
• Pandas helps with data manipulation
• Matplotlib performs visualization
• Scikit-learn assists with machine learning

Machine Learning Frameworks in Python

• PyTorch qualifies as a Machine Learning framework

Jupyter Notebooks

• Jupyter Notebooks do not support static HTML pages

Data Types in Data Science

• Data Science deals with both structured and unstructured data

Data Analysis

• Data analysis aims to interpret and understand data

IDEs

• Visual Studio Code is a text-based IDE

Pivot Tables

• Rows are used to group data in a pivot table
• Pivot tables summarize large datasets quickly using rows, columns, values, and filters
• Refreshing a pivot table ensures it reflects changes in the source data
• Right-click the pivot table and select "Refresh" to update it with the latest information
• Aggregation functions in pivot tables summarize data using functions like Sum, Count, Average, Min, and Max

Exploratory Data Analysis (EDA)

• EDA is a process to understand data patterns and characteristics before modeling, using statistics and visualizations

Open Data

• Open Data gives freely accessible datasets for research, testing models, and creating public tools

Data Science vs Business Intelligence

• Business Intelligence (BI) reports past data, while Data Science predicts future trends using advanced analytics

Applications of Data Science

• Netflix uses Data Science to enhance user experience through show recommendations based on viewing history
• Data Science can predict crop yields in agriculture using satellite data and weather patterns

Importance of Domain Expertise

• Domain expertise helps correctly interpret results and ensures solutions are practical for the target industry

Python Fundamentals

• Value_2 is a valid Python variable name
• The output of 5 ** 2 is 25
• "=" is the assignment operator in Python
• "and" is a logical operator
• "is" checks for object identity
• Dictionaries are used to store key-value pairs
• Lists are ordered and changeable data types

Python Conditionals and Functions

• The "if" statement checks a condition in Python
• def myFunc(): is the correct syntax for a Python function definition

Python Data Types and File Handling

• print(type(3.14)) outputs <class 'float'>
• file.read() reads a file in Python
• list[1:3] is the correct slicing syntax for a list
• Tuples are immutable data types

Python Loops

• "for" is the keyword used to begin a loop
• print(10 not in [5, 10, 15]) returns False

Variables in Python

• A variable is a container that stores data values, e.g., student_age = 20
• The == operator checks if two values are equal
• The "and" operator checks if both conditions are True
• "==" checks equality, "is" checks object identity

Data Types

• Lists are mutable; tuples are immutable

Dictionaries

• Dictionaries are collections of key-value pairs

Tuples

• Tuples are written in round brackets, e.g., (1, 2, 3)

Conditional Statements

• The "if" statement executes a block of code if a condition is true

Loops

• For loops iterate over a sequence (list or range)

Functions

• The "def" keyword defines a function

File Reading

• Use open('filename.txt', 'r') to open a file for reading

List Slicing

• List slicing extracts a part of a list using index ranges

Logical Operators

• if age > 18 and is_student: is an example of using a logical operator

Data Preprocessing

• Data collection is the first step
• The pandas function drop_duplicates() removes duplicate rows
• Replacing "-" with 0 is part of data cleaning

Pandas Methods

• The astype() method in pandas converts data types
• df[df.column == value] keeps rows where a column equals a value
• df.dtypes returns data types of each column

Handling Missing Data

• Handling missing data ensures accurate analysis
• Drop or fill missing values after identification
• To ensure accurate analysis is the purpose

Parsing XML Data

• The xml.etree.ElementTree helps parse XML data

Combining DataFrames

• The pandas method concat() combines multiple DataFrames

Removing Data

• Remove rows where ‘State’ is not ‘Finished’ from the dataset

Counting Null Values

• The isnull().sum() from Pandas counts null values

Adding New Columns

• The new columns added in the question pool section are level and q_type

Data Cleaning

• Data cleaning corrects or removes inaccurate, incomplete, or irrelevant data
• df.rename(columns={'old': 'new'}) changes a column name
• df.drop_duplicates() removes duplicate rows
• Replace "-" with 0 in a DataFrame using df.replace('-', 0)
• df['col'] = df['col'].astype(int) turns a column to int type

Label Column Purpose

• The ‘Label’ column classifies rows as 'Passed' or 'Failed' based on a score

Outlier Detection

• Box plots visually detect outliers
• To ensure the data only includes completed entries remove rows where 'State' is not 'Finished'

Null Value Function

• df.isnull().sum() detects null values in each column

XML parsing

• xml.etree.ElementTree can parse XML data

State Filtering

• df[df['State'] == 'Finished'] is a filtered DataFrame with only 'Finished' rows

Exploratory Data Analysis (EDA)

• The main goal is to understand the data's structure and patterns

EDA Tasks

• Modeltuning is not a typical EDA task

Outlier Detection

• Box plots are best for visual outlier detection

Center Statistics

• The mean represents the data's center

Interquartile Range (IQR)

• IQR is defined as Q3 - Q1

Skewed Distribution

• A right-skewed distribution has a long tail to the right

Descriptive Statistics

• df.describe() provides descriptive statistics

Missing Value Identification

• .isnull().sum() helps find missing values

Data Visualization Library

• Seaborn is used for data visualization

Categorical Variable Frequencies

• Bar charts compare categorical variable frequencies

Dispersion Measure

• Range is a measure of dispersion

Symmetrical Data Distribution

• A symmetrical data distribution occurs when the mean, median, and mode are approximately equal

Missing Values and Pandas

• The dropna() pandas function drops rows with missing values

Continuous Variables

• The scatter plot analyzes relationships between two continuous variables

Data Spread Measure

• Variance identifies the spread of data from the mean

EDA Purpose

• EDA helps understand data structure, discover patterns, detect outliers, and prepare data for modeling

EDA Libraries

• Pandas and Seaborn are two Python libraries useful for EDA, also Matplotlib

DataFrame Missing values

• df.isnull().sum() is used to check for missing values in a DataFrame

Box Plot Use

• Box plots are useful for identifying outliers and visualizing data distribution

IQR Definition

• The Interquartile Range (IQR) is the difference between the third quartile (Q3) and the first quartile (Q1)

Pandas Describe

• The describe() method in pandas displays summary statistics: count, mean, std, min, max, and quartiles

Outlier Importance

• Outlier detection is important because outliers can distort statistical analyses and affect model accuracy

Common Correlation Plot

• Scatter plots or heatmaps are commonly used to identify correlation

Skewness

• Skewness indicates data distribution asymmetry, positive skew indicates a longer tail on the right

Handling Missing Data Techniques

• Use dropna() to drop rows/columns or fillna() to fill missing values

mean v median

• Mean is the average, median is the middle value when data is sorted

Histogram Data

• Histograms are suitable for numerical, continuous data

Standard Deviation

• A large standard deviation implies data points are spread widely from the mean

Median and Mean vs Data

• The median is preferred over the mean when data contains outliers or is skewed

Visualizing Numerical Variables

• Histograms or box plots can visualize one numerical variable's distribution

Data Analysis Process

• Understanding the problem is the initial step

Data Type

• Nominal is a type of data

Parts of A Whole Charts

• Pie charts

Data Set Center

• The mean, is a statistical measure to describe the center of a data set

Outliers Plot

• Box plots detect outliers

Data Vis Python Library

• Matplotlib is a Python library for visualization

Data Manipulation PyLibrary

• Pandas is best library

Variable w/ labels

• Categorical

Means Test

• T-test

Histogram Usage

• Visualizes frequency distribution

P-values Determine

• Statistical significance

Trends for Time

• Line Chart

Boxplot Graph

• Sns.boxplot()

Data Analysis Purpose

• To inspect, clean, transform, and model data

Common Data Chart Tool

• Matplotlib, is used for analysis

Standard Deviation Use

• Measures dataset data dispersion

outlier Detection Method

• Box Plot helps

Age Variable

• Numerical

P-Values Indicate

• Strong anti null

Read File

• pd.read_csv(), reads a file.csv into pandas

Histogram Use

• Distrubution Visualization

t-test

• Compare 2 means

Pandas Describe

• Gives summary statistics

Correlation Measurement

• Linear direction/strength
• Shows correlations

EDA Roles

• Understand data, find patterns, look for oddities, and data preparation

Libraries for EDA

• Seaborn and Pandas

Check values

• Look for is null with sum

Useful Plots

• Box plot

Def IQR

• Q1-Q3

Week 6

• ML - Ai that learns
• SL - Data models make predictions
• Unsupervised Learning - to see hidden groupings
• Examples of reinforcement learning - rewards and penalties
• Over fitting - The model is poor on unseen samples but fine on existing
• Reducing Overfitting - By gaining more data
• Normalization - data transforms
• Label Ecoding - Conveerts numbers from text
• Accuracy : correct predictions
• F-1 Score - Precision and recall
• Cross Validation - Multiple splits

• ML - learning from data
• SL - label data
• Unsupervised Learning - is helpful
• RL - Action based
• Overfitting - Noise In new data
• Cross Values - Prevents overfitting
• FS ensures fairness
• Normalization 0 - 1
• Text becomes numeric
• Correct prediction
• PE - Positive
• Rev - captured actives

f Balance - PR

V - multiple splits

NP Model - growing data

Supervised Learning

• Supervised learning requires labeled data

Unsupervised Learning

• Market segmentation

Reinforcement Learning

• Actions directly influence future outcomes

Overfitting

• Overfitting happens with incorrect data

ML Prevention

• Cross Value prevents this

Features Scaling

• FS ensures fairness

Normalization

• Normalization goes 0 to 1

Transforming to Numeric

• Text should become numeric

Accurate

• PE or Positive

Recall Captured

• captures actuals

Precision Balance

• Precison and recall must be balances

split

• Mutliple splits

NP Model

• Is growing data

Classifcations

• Used to classify

Examples

• Diagnosing diseases is a good example

Require Label

Supervised Learning REQUIres A LABEL

Features Vectors

• Refers to imput features

Not classification

Means is not a classification algorithm

Evaluation

• Evalutes Model accurately

matrixes

• gives data and helps

Tp In matrix world

• TRUE POSSITIVE

Precision

• Tp div Tp + fp

Rememberal

• TP div Tp + FN

Rememberal2

• f score

Evaluation help

To help with evaluations do cross validations

High Quality models

• Overfitting bad

means

• Cant get it is the model bad then

deaf Khmer community,

• To connect deaf peoples faces and see reactions

Random forest,

• Uses accuracy

Letters

• 33 are recognizes

Interface.

G Radio uses web and gestures

Data processing?

• Mediapie

Technique

• Scaling

data training split

• 80/20

good for systems?

• High acuraccy + efficy

Extraction of keypoints

• 21 keypoint are extracted

Display on Interface

• Instantly

Labels.

• The letters ko , ka etc are the labels

importatnt,

• Helps bridge gaps

loaded models and Interfaces

• Loaded in jobs and models### Gradios Interface
• Real time Deployed
• They are

Aim

• To use a prototypes
• To Class models + radio### The 3 step training for radiio

1. Training
2. Saving
3. Testing + deployment ### libraries

• Scikit - learn### types

• Use decisons tree### How saved

• With the libs

How get set up

• must load radio!### keys

• input to show!### testing
• Colab or space faces### reporting

• model accuarcy and interfaces

sets prep?

• What topic for classification### datasets?

• data cleaning### splits and data Splitting the train set!

assesing performance

• model by using test data!

what to install in radio??

pip install radio### What Is main goal here? To use or make classification based model### required step

• creating the faces

Commmon use

• SCikit learn ### Classification Decision Tree!### HOW TO Store or use Joblib Or Pickle!### get radio with a command? , pip install Gradio!

essential components,

Input AND Out### Where we Deploy? - faces, colab or local