Data Science Fundamentals

Introduction to Data Science

• Data Science is considered the most revolutionary technology of the era.
• The primary function is to derive useful insights from data.
• These insights are used to solve real-world complex problems.
• Mastery of data science requires understanding its basic fundamentals.
• Statistics and probability are essential for understanding the math behind data science and machine learning algorithms.
• Understanding machine learning involves learning about its different types and algorithms.
• Supervised learning algorithms, starting with linear regression, are a key component.
• Logistic regression is useful for solving classification problems.
• Decision trees can solve complex data-driven problems.
• Random forests can solve both classification and regression problems through use cases and examples.
• K-nearest neighbor (KNN) can be used for complex classification problems.
• Naive bias is a significant algorithm, notably used in Gmail spam detection.
• Support Vector Machines (SVMs) are used to draw hyperplanes between different classes of data.
• Unsupervised learning involves using k-means for clustering.
• Association rule mining facilitates Market Basket analysis.
• Reinforcement learning includes understanding concepts and seeing demonstrations.
• Deep learning involves grasping neural networks and their various types.
• Understanding data science concepts and interview tips are crucial for acing interviews.

The Growing Importance of Data Science

• Data science is in high demand due to the increasing rate of data generation.
• Processing and making sense of vast amounts of data are key challenges.
• Understanding the sources of data and how technology's evolution has increased the need for data science is key.
• IOT and social media are major contributors to data generation.
• Data science helps businesses like Walmart use data patterns to increase potential.
• Data science is about extracting, processing, and using data to create solutions.
• Understanding machine learning and its types is essential.
• The k-means algorithm and its use cases are important in data science.
• Clustering movies based on social media popularity using k-means is a practical application.
• A data science certification can be beneficial for career advancement.

Evolution of Data and Technology

• Early technology involved telephones and limited data generation.
• Data was memorized, not stored digitally.
• Smartphones allow for everything about us to be stored on them.
• PCs processed very little data initially.
• Floppy disks stored small amounts of data.
• Hard disks later stored gigabytes of data.
• Data is now stored everywhere, including the cloud and various appliances.
• Smart cars generate a lot of data through internet and mobile phone connections.
• Initially, there was little structured data.
• Simple BI tools were sufficient to process data.
• Current data volumes are too large for simple tools, so we need data science.

Impact of the Internet of Things (IoT)

• 2.5 quintillion bytes of data are produced each day.
• The growth of IoT is accelerating this data production.
• IoT refers to networks of devices that communicate and transfer data through the internet.
• IoT devices include vehicles, TVs, coffee machines, refrigerators, and washing machines.
• Data is measured in zettabytes, with one zettabyte equal to a trillion gigabytes.
• By the end of 2019, IoT was estimated to generate over 500 zettabytes of data per year.
• Traditional BI tools are insufficient to handle this volume of data.
• Data science provides a solution for managing and processing IoT data.

Role of Social Media in Data Generation

• The fact that we are all in love with social media generates a lot of data for us
• Large amounts of data are generated every minute on social media platforms like Instagram and Twitter.
• Processing and analyzing this much data with traditional methods is hard.
• Data science is a simple process that extracts useful information from data to address this is what needed.

Additional Factors in Data Generation:

• Online transactions are very popular like paying bills, shopping, and buying homes.
• Streaming music and videos on platforms like YouTube generate significant data.
• Healthcare has integrated with the internet, with devices like Fitbits tracking health data.
• Education is increasingly online.
• Nearly all activities are carried out online.
• Data science extracts useful insights from data.
• It is used to grow your business.

Data Science in Business: Walmart's Example

• Walmart is the world's largest retailer with over 20,000 stores in 28 countries.
• It is building a cloud capable of processing 2.5 petabytes of data every hour.
• Walmart uses customer data to gain insights into shopping patterns to increase potential of the business.
• Analysts know customer details, such as correlations between buying Pop-Tarts and cookies.
• During Halloween, analysis of cookie sales identified stocking issues, prevented loss of sales
• Through Association Rule mining, Walmart found strawberry Pop-Tart sales increased sevenfold before hurricanes.
• Strawberry Pop-Tarts were then placed near checkouts before hurricanes to increase sales.
• Walmart analyzes social media data to identify trending products.
• They found Facebook users liked cake pops, leading to their introduction in Walmart stores.
• Effective data processing and analysis enable Walmart to find hidden patterns and improve business.
• They invest time and money in data analysis to find useful insights.
• Walmart capitalizes on identified associations between products through promotions and discounts.

Core Concept of Data Science

• About uncovering findings from data.
• Data science surfaces hidden insights to help companies make smart business decisions.
• Netflix analyzes movie viewing patterns to understand and cater to user interests.
• Data has a lot of power if you know how to process it.
• Data science is all about extracting the useful information from your business data.

Data Scientists and Data Exploration

• When faced with challenging situations, data scientists become detectives.
• They seek patterns and characteristics in the data.
• The information is used for the betterment of the organization.

Who is a Data Scientist?

• Data scientists view data through a quantitative lens.
• Math is a critical skill for building predictive models.
• Understanding the underlying mechanics of these models is essential since these predictive models are based on hard math.
• Math is important, but it is not the only type that is utilized.
• Many machine-learning algorithms are based on linear algebra.

Essential Skills for Data Scientists

• They need to be good with technology.
• They analyze large data sets and work with complex algorithms.
• Data scientists must be efficient with coding languages like SQL, Python, R, and SAS.
• They need to be tactical business consultants.
• Since they work closely with data, they know the business intricately to the point where they know each and every aspect of your business.
• Business acumen is as important as skills in algorithms, math, and technology.

Essential Data Scientist Skills

• Statistics provides the numbers from the data and is therefore very important.
• Familiarity with statistical tests, distributions, and maximum likelihood estimators is needed
• Probability theory and descriptive statistics help in making better business decisions.
• Expectation to use the tools of the trade.
• This means knowing: statistical programming languages like R or Python with database quering language like SQL
• People generally prefer: R and Python because of amount of packages they have.
• At a minimum you should know a language or or python and a database query language.

Data Extraction and Processing

• Data needs to be extracted from multiple sources like MySQL and MongoDB databases.
• It needs to be stored in a proper format or structure for analysis and querying.
• Finally, the data can be loaded into a data warehouse for analysis.

Data Wrangling and Exploration

• Data wrangling cleans data, addressing missing or null values and inconsistent formats.
• One of the most difficult tasks in data science
• Data wrangling is a time-consuming task
• The goal is to understand what to do with these outliers.
• After cleaning, the data is analyzed to make sense of it.

Skills for Data Scientists

• Data scientists need to identify trends, outliers, and unexpected results in data.
• Machine learning is crucial for processing large datasets, especially in data-driven companies like Netflix or Google Maps.
• Key machine learning algorithms to be familiar with include: K-Nearest Neighbor, Random Forest, K-Means, and Support Vector Machines.
• These algorithms can often be implemented using Python or R libraries.
• Understanding machine learning is crucial due to the vast amounts of data being generated.
• Interview processes for data scientist positions often involve questions about machine learning algorithms and implementation skills.
• Big data processing frameworks like Hadoop and Spark are necessary for handling large volumes of structured and unstructured data.
• Data visualization is essential for presenting data in an understandable and visually appealing format.
• Tools such as Tableau and Power BI are popular for data visualization.
• Besides technical skills, a data scientist needs a data-driven problem-solving approach and creativity with data.

Job Roles in Data Science

• Data scientists understand business challenges and offer data analysis and processing solutions.
• They perform predictive analysis and identify trends to aid better decision-making.
• Expertise in R, Matlab, SQL, and Python is essential.
• Higher education in mathematics or computer engineering is advantageous.
• Data analysts visualize and process large datasets, performing queries on databases.
• Optimization skills are crucial for creating algorithms to extract information from large databases without corrupting data.
• Data analysts must know SQL, R, SAS, and Python.
• Certifications in these technologies can enhance job applications.
• Good problem-solving skills are essential.
• Data architects create blueprints for data management, ensuring integration, centralization, and protection with security measures.
• Data architects ensure that data engineers have the best tools and systems.
• Expertise in data warehousing, data modeling, extraction, transformation, and load (ETL) is required.
• Proficiency in Hive, Pig, and Spark is necessary.
• Data engineers build and test scalable big data ecosystems.
• They update existing systems with newer versions and improve database efficiency.
• Technologies requiring hands-on experience include Hive, NoSQL, R, Ruby, Java, C++, and Matlab.
• Experience with popular data APIs and ETL tools is helpful.
• Statisticians need a sound understanding of statistical theories and data organization.
• They extract insights and create new methodologies for engineers.
• Statisticians need a passion for logic and knowledge of database systems like SQL and machine learning concepts.
• Database administrators ensure databases function properly and manage permissions.
• They are responsible for database backups and recoveries.
• Skills needed include database backup and recovery, data security, and data modeling/design.
• Business analysts link data-oriented technologies with actionable business insights.
• They focus on business growth and act as a link between data engineers and management.
• Understanding business finances, business intelligence, data modeling, and visualization tools is essential.
• Data and analytics managers oversee data science operations and assign duties based on skills and expertise.
• Strengths include technologies like SAS, R, SQL, good management skills, social skills, leadership, and innovative thinking.
• Proficiency in Python, SAS, R, Java, etc. is needed.

Data Lifecycle

• The data lifecycle consists of six steps: business requirement, data acquisition, data processing, data exploration, modeling, and deployment.
• Understanding the problem is crucial before starting a data science project.
• This involves identifying the central objectives and variables to be predicted.
• Data acquisition involves gathering data from different sources.
• Key questions include: What data is needed? Where does it live? How can it be obtained? How to store and access it efficiently?
• Data processing involves formatting, structuring, and cleaning the data, removing missing, inconsistent, or corrupted values.
• Data exploration involves brainstorming data analysis using histograms and interactive visualizations to understand patterns.
• Data modeling involves carrying out model training.
• Model training finds a model that accurately answers questions.
• It involves splitting data into training and testing datasets.
• A model is built using the training data.
• The training and testing data is evaluated using machine learning algorithms.
• The most suitable model for business requirements is identified.
• Deployment involves setting up the model in a production or production-like environment for user acceptance and validation.
• Any issues with the model or algorithm must be fixed at this stage.

Statistics and Probability

• Statistics and probability are foundational for machine learning, deep learning, AI, and data science.
• Mathematics and probability are embedded in everyday life, from shapes and patterns to the petal count of a flower.

Agenda Overview

• Session will begin with understanding what data is.
• It will move on to quantitative and qualitative data categories.
• Statistics, basic terminologies, and sampling techniques will be discussed.
• Descriptive and inferential statistics will be covered.
• The session will focus on descriptive statistics, including measures of center, spread, Information Gain, and entropy.
• A use case to understand these measures and will be reviewed.
• A confusion Matrix will be explained.
• The probability module will include basic terminologies and the different probability distributions.
• Types of probability, including marginal, joint, and conditional probability, will be discussed with a use case.
• Bayes' theorem will be explained using an example.
• A demonstration of the R language will be provided for the descriptive statistics module.
• The Inferential statistics module will discuss point estimation, confidence interval, and margin of error with a use case.
• Hypothesis testing and a demo explaining inferential statistics will conclude the session.

What is Data?

• Data is facts and statistics collected for reference or analysis.
• It can be collected, measured, analyzed, and visualized using statistical models and graphs.
• Data is divided into qualitative and quantitative subcategories.
• Qualitative data deals with characteristics and descriptors that cannot be easily measured but can be observed subjectively.
• Qualitative data is further divided into nominal and ordinal data.
• Nominal data does not have any order or ranking (e.g., gender, race).
• Ordinal data has an ordered series of information (e.g., customer ratings of a restaurant's service).
• Quantitative data deals with numbers and measurable quantities.
• There are two types of quantitative data: discrete and continuous.
• Discrete data, also known as categorical data, can hold a finite number of possible values (e.g., number of students in a class).
• Continuous data can hold an infinite number of possible values (e.g., weight of a person).
• A discrete variable is also known as a categorical variable and can hold values of different categories (e.g., "spam" or "not spam" for a message variable).
• Continuous variables can store an infinite number of values (e.g., weight).
• Dependent variables have values that depend on independent variables.

Definition of Statistics

• Statistics is an area of applied mathematics.
• It concerns data collection, analysis, interpretation, and presentation.
• Statistical methods can be used to visualize data, collect data, and interpret data.
• Mathematics deals with data to solve complex problems.

Examples of Problems Solved by Statistics

• Determining a new drug's effectiveness in curing cancer, requiring a test to confirm its effectiveness.
• Assessing the probability of winning a bet on whether a home run will be hit in a baseball game.
• Analyzing sales data to identify areas for business improvement by understanding relationships between different variables.

Basic Terminologies in Statistics

• Population refers to a collection or set of individuals, objects, or events whose properties are analyzed.
• Sample is a subset of the population, chosen to represent the entire population.

Sampling

• Sampling is a statistical method that selects individual observations within a population.
• Sampling infers statistical knowledge about a population.
• It helps finding statistics of a population like the mean, median, mode, standard deviation or variance.

Reasons for Sampling

• Sampling is used because it's often impractical to study an entire population.
• Surveying the habits of teenagers in the U.S. would be too time consuming, therefore a sample is sufficient.
• A sample of the population is studied to draw inferences about the entire population.
• The goal is to analyze the sample and have it represent the entire population.

Types of Sampling Techniques

• Probability sampling
• Non-probability sampling
• Probability sampling involves samples from a large population chosen using probability.

Types of Probability Sampling

• Random sampling: Each population member has an equal chance of being selected.
• Systematic sampling: Every nth record is chosen from the population.
• Stratified sampling: Strata are created to form samples
• A stratum is a subset of the population that shares a common characteristic.
• Random sampling is then used on these strata to choose the final sample.

Types of Non-Probability Sampling

• Non-Probability sampling types include Quota, Judgment, and Convenience sampling.

Types of Statistics

• Descriptive statistics: Used to describe and understand the features of a specific data set by giving a summary of the data.
• Inferential statistics: Makes inferences and predictions about a population based on a sample.
• Inferential statistics generalizes a large data set and applies probability to draw a conclusion, inferring data parameters based on a statistical model by using sample data.

Descriptive Statistics

• Descriptive statistics is used to describe data sets via summaries about samples and measures of the data.
• Two measures in descriptive statistics:
  • Measure of central tendency (measure of center).
  • Measures of variability (measures of spread).

Measures of Center

• Measures of center are statistical measures that represent the summary of a data set.
• The measure of central tendency include: mean, median, mode
  • Mean: The average of all the values in a sample.
  • Median: The central value of the sample set.
  • Mode: The value that is most recurrent in the sample set.

Measures of Spread

• A measure of spread, also called a measure of dispersion, is used to describe the variability in a sample or population.
• Measures of variability: range, interquartile range, variance, and standard deviation.
  • Range: Measures how spread apart the values in a data set are (max value - min value).
  • Interquartile range: How data set is divided into quartiles
    • Quartiles: Tell us about the spread of a data set by breaking the data set into different quarters. IQR = (Q3 - Q1)
  • Variance: Measures how much a random variable differs from its expected value.
  • Deviation: The difference between each element from the mean.

Sample and Population Variance

• Standard deviation measures the dispersion of data from its mean.
• Daenerys has 20 dragons with numbers 9, 2, 5, 4, etc.

Calculating Standard Deviation

• First, find the mean of the sample set by adding all numbers and dividing by the total samples.
• For the example, the mean is calculated as 7.
• Subtract the mean from each data point and square the result.
• Find the mean of the squared differences.
• Take the square root to find the standard deviation.
• The standard deviation for the example is 2.983.

Information Gain and Entropy

• Information Gain and entropy are important in machine learning algorithms like decision trees and random forests.
• Entropy measures the uncertainty present in data.
• S represents the set of all instances in the data set.
• N represents the different types of classes in the data set.
• Pi represents the event probability.
• Information Gain indicates how much information a feature gives about the final outcome.
• H(s) is the entropy of the whole data set.
• SJ is the number of instances with the J value of an attribute A.
• s is the total number of instances in the data set.
• V is the set of distinct values of an attribute A.
• h of s j is the entropy of subsets of instances.
• hedge of a comma s is the entropy of an attribute a

Use Case: Predicting Whether a Match Can Be Played

• The goal is to predict whether a match can be played by studying weather conditions.
• Predictor variables: outlook, humidity, wind, and temperature.
• The target variable is "play," with values "yes" or "no.".
• A decision tree is used to solve this problem.

Decision Trees

• Each branch of the tree denotes a decision.
• Out of 14 observations, 9 result in "yes" for playing.
• Data is clustered based on the outlook (sunny, overcast, rain).
• When the outlook is sunny, we had two yeses and three nos.
• When the outlook is overcast, all four observations are yes.
• When the outlook is rain, we have three yeses and two nos.
• The decision is made by choosing the Outlook variable as the root node.
• The root node is the topmost node in a decision tree.
• The Outlook node has three branches: sunny, overcast, and rain.
• Overcast results in a 100% pure subset.
• Entropy measures the impurity or uncertainty.
• Lesser uncertainty or entropy of a variable means it is more significant.
• The root node is assigned the best attribute for the most precise outcome.

Using Information Gain and Entropy for Decision Trees

• Information Gain and entropy help understand which variable best splits the data.
• From 14 instances, 9 said yes and 5 said no.
• Entropy is calculated as 0.940.
• The goal is to find the information gain for each attribute (Outlook, windy, humidity, temperature).
• The variable with the highest Information Gain is chosen.
• The information gain for the windy attribute is 0.048.
• The information gain of the Outlook variable is 0.247.
• The information gain for the humidity variable is 0.151.
• The information gain of attribute temperature is 0.029.
• The Outlook variable has the maximum gain (0.247).

Confusion Matrix

• The confusion matrix describes the performance of a model.
• It is used for classification models.
• It calculates the accuracy of your classifier by comparing your actual results and Your predicted results.

Confusion Matrix Example

• Given data from 165 patients, 105 have a disease, and 60 do not.
• The classifier predicted "yes" 110 times and "no" 55 times.
• In reality, 105 patients have the disease, and 60 do not.
• The actual value is no and the predicted value is no for 50 of the cases.
• Classifier correctly classified 50 cases as "no.".
• 10 cases were incorrectly classified (actual value is "no," but the classifier predicted "yes").
• The classifier wrongly predicted that five patients do not have diseases (whereas they actually do have diseases).
• The system correctly predicted the outcome for onehundred patients who in fact, have the disease.
• True positives are the cases in which we predicted a yes when in reality the patient did have the condition.
• False positive = be predicted Yes when they in fact should have been predicted as No.
• False negative are be predicted No but in reality the actual result should have been a Yes.
• True negatives are the instances your classifier predicted No and they in fact were negative (did not have the condition).

R Demo

• Demonstrates how to calculate mean, median, mode, variance, and standard deviation in R.
• It also includes how to study variables by plotting a histogram.
• The demo uses randomly generated numbers and stores them in a variable called "data."
• The mean is computed using the mean() function and assigned to the "mean" variable.
• The median is calculated using the median() function.