Introduction to Data Science Unit 1

Introduction to Data Science

• Data science is a multi-disciplinary science that aims to generate knowledge from data for decision making.
• Data science involves collecting data from multiple sources, cleaning, integrating, and processing it to produce useful information.

Definition of Data Science

• Data science is a way to extract knowledge from data to support decision making.
• It involves processing data to generate patterns, models, and insights that can be used for decision making.

Types of Data

• Structured Data: follows a specific schema, can be associated with a schema, and is typically stored in relational databases. Examples: customer data, account data, transaction data.
• Semi-Structured Data: has some structure, but not a fixed schema, and is often stored in XML, JSON, or other formats. Examples: XML data, JSON objects, server logs.
• Unstructured Data: does not follow a specific schema, and can be in the form of text, images, audio, or video. Examples: social media data, email data, image data.
• Data Streams: a sequence of data generated over time, can be structured, semi-structured, or unstructured, and is often processed in real-time. Examples: IoT sensor data, social media feeds.

Statistical Data Types

• Categorical Data: used to define categories, can be nominal (no relationship between categories) or ordinal (categories have a specific relationship).
• Quantitative Data: numeric data, can be discrete (distinct numbers) or continuous (continuous values).

Measurement Scales

• Nominal Scale: categorical data with no relationship between categories, examples: gender, occupation.
• Ordinal Scale: categorical data with a specific relationship between categories, examples: age categories, income categories.
• Interval Scale: quantitative data with equal intervals between values, but no absolute zero, examples: IQ, temperature in Celsius.
• Ratio Scale: quantitative data with equal intervals between values and an absolute zero, examples: temperature in Kelvin, age.

Sampling

• Population: the entire set of data being studied.
• Sample: a representative subset of the population.
• Statistic: a value computed from the sample data.
• Parameter: a value predicted from the sample data about the population.

Basic Methods of Data Analysis

• Descriptive Analysis: summary statistics and data visualization to understand the data.
• Exploratory Analysis: identifying patterns and relationships in the data.
• Inferential Analysis: using sample data to make inferences about the population.
• Predictive Analysis: using data to predict future outcomes or trends.### Data Analysis Methods
• Three basic methods used for analyzing data:
  • Descriptive analysis
  • Exploratory data analysis
  • Inferential data analysis

Descriptive Analysis

• Used to present basic summaries about data without interpreting it
• Includes different statistical values and graphs
• Different types of data are described in different ways
• Examples of descriptive analysis:
  • Frequency table of various categories for categorical data
  • Measures of central tendency (mean, median) and spread (range, interquartile range) for quantitative data

Categorical Data

• Gender is a categorical variable
• Summary of categorical data is presented in a frequency table
• Frequency table includes:
  • Frequency of each category
  • Proportion of each category
  • Percentage of each category
• Graphs for categorical data:
  • Bar chart or pie chart

Quantitative Data

• Height is a quantitative variable
• Descriptive statistics for quantitative data:
  • Measures of central tendency (mean, median)
  • Measures of spread (range, interquartile range)
• Mean and median are two basic measures of central tendency
• Mean is sensitive to outliers, while median is not
• Mode is another measure of central tendency, but it's not commonly used

Sampling Distribution and Central Limit Theorem

• Sampling distribution is a probability distribution of means of random samples from a population
• Sampling distribution is used to determine if sample statistics are close to population parameters
• Central Limit Theorem states that with the increase in sample size, the sampling distribution approaches a normal distribution
• Conditions for Central Limit Theorem:
  • Independent random samples
  • Sufficiently large sample size (but less than 10% of population)
• Sampling distribution follows a normal distribution with mean = population mean and standard deviation = population standard deviation / sqrt(sample size)

Standard Normal Distribution

• Standard normal distribution is a standardized form of normal distribution
• Mean of standard normal distribution is 0, and standard deviation is 1
• Z-score for standard normal distribution is x / √n
• 95% of the area under the standard normal curve lies between -2 and 2

Inferential Data Analysis

• Used to make inferences about a population based on a sample
• Includes hypothesis testing and confidence intervals### Estimation of Parameters of the Population
• A point estimate is a single value that is used to estimate a population parameter.
• A good point estimate should be unbiased and have a small standard deviation.
• An example of a point estimate is the proportion of students who play some sport (40.5%).
• A confidence interval is a range of values within which the population parameter is likely to lie.
• The confidence interval is calculated using the sample proportion and the standard error.
• The probability that the confidence interval contains the population parameter is called the confidence level (e.g. 95%).
• The confidence level is determined by the z-score, which is the number of standard deviations from the mean.

Confidence Interval to Estimate Mean

• The confidence interval can be used to estimate the mean of a population.
• The standard error in the estimated mean is calculated using the sample standard deviation.
• The formula for the confidence interval is: (sample mean - z * standard error) to (sample mean + z * standard error).
• The confidence interval is used to estimate the average height of students in a sample.

Significance Testing of Statistical Hypothesis

• Significance testing involves testing a hypothesis to determine if the data supports it.
• The process of significance testing involves:
  1. Testing pre-conditions on the data.
  2. Making a statistical hypothesis (null and alternative hypotheses).
  3. Performing the desired statistical analysis.
  4. Analysing the results.
• The null hypothesis defines a particular value for the parameter or specifies that there is no difference or change.
• The alternative hypothesis specifies the values or difference in parameter values.

Correlation and Regression

• Correlation is used to determine the strength of linear association between two quantitative variables.
• The correlation coefficient (r) measures the strength and direction of the linear relationship.
• The value of r lies between -1 and 1, where a positive value indicates a positive relationship and a negative value indicates a negative relationship.
• A correlation coefficient close to 1 or -1 indicates a strong linear association.
• Correlation does not imply causation.
• Simple linear regression predicts a response variable using one explanatory variable.
• The equation for simple linear regression is: ypredicted = a + bx.
• The method of least squares is used to find the regression line by minimizing the sum of squares of the residuals.
• The predictive power of the model is determined by the coefficient of determination (r2).