Data Science: Information Gain and Entropy

Data Science Course Information

• Course: Data Science
• Program: Software Engineering
• Department: Computer Science
• Term: 7th Term, Final Year
• Teacher: Engr. Mehran M. Memon

Information Gain and Entropy

• Information Gain and Information Entropy are used in Decision Trees.
• Information Gain is a metric for evaluating the quality of a split in a dataset.
• Entropy is a measure of the uncertainty or randomness in a dataset (high entropy = more randomness, low entropy = less randomness).

Example Data and Split

• Example dataset is given with x and y values.
• A split is made at x = 1.5.
• The split divides the data into two branches (left and right), each with a different mix of blue and green points.

Entropy Calculation

• Entropy measures the impurity of a dataset.
• A dataset of only one color has zero entropy (e.g. all blue points).
• A dataset of mixed colors (e.g., blue, green, and red) has higher entropy.
• Entropy is calculated using a formula ( Σ pi * log2(pi) where pi is the proportion of each class in the dataset).

Information Gain Calculation

• Information Gain is calculated by finding the difference between the entropy before a split (initial entropy) and the weighted average of the entropy after the split.
• The formula takes into account the size of each branch after the split, (e.g., 4 elements in left branch and 6 in right branch).

Probability

• Probability is the ratio of desired outcomes to total outcomes (desired outcomes/total outcomes).
• Probabilities always add up to 1.

Types of Events

• Disjoint Events: Events that cannot occur at the same time (e.g., drawing a king and a queen from a deck).
• Non-Disjoint Events: Events that can occur at the same time (e.g., a student getting 100 in statistics and 100 in probability).

Probability Distribution

• Probability Density Function (PDF): The equation describing a continuous probability distribution.
• Properties of PDF:
  • Graph is continuous.
  • Area under the curve is equal to 1.
  • Probability for a range of values is the area under the curve within that range.

Normal Distribution

• A type of probability distribution that is bell-shaped.
• Describes how a random variable will likely be distributed.
• Important parameters are mean (μ) and standard deviation (σ).
• Formula: Y = [ 1/ (σ * sqrt(2π)) ] * e ^[-(x - μ)^2 / (2 * σ^2) ]

Standard Deviation and Curve

• Standard Deviation affects the shape of the normal curve (wide vs. narrow).

Central Limit Theorem

• The sampling distribution of the mean becomes approximately normal as the sample size increases. This applies to any independent random variable.

Types of Probability

• Marginal Probability: Probability of a single event.
• Joint Probability: Probability of two or more events happening at the same time.
• Conditional Probability: Probability of an event given that another event has already occurred.

Bayes' Theorem

• Shows the relationship between conditional probability and its inverse.

Point Estimation

• Estimation of a single population value based on sample data.

Methods for Finding Estimates

• Method of Moments: Equating sample moments with population moments.
• Maximum Likelihood: Maximizing the likelihood function.
• Bayes' Estimators: Minimizing average risk.
• Best Unbiased Estimators: Unbiased and good estimators for a parameter.

Interval Estimate

• An interval (or range of values) used to estimate a population parameter.

Confidence Interval

• Measure of confidence that an interval estimate contains the population mean.
• A range of values with a specified probability of containing the true population parameter.

Margin of Error

• Difference between the point estimate and the true population parameter.
• Maximum possible distance between the point estimate and the parameter being estimated.

Estimating Level of Confidence

• Probability that the interval estimate contains the population parameter.
• Calculated using the standard normal table and critical values to get the Z-score.

Data Set for Case Study

• A case study is presented about training salary and package for candidates.
• The data shows how salary packages are obtained by candidates who did and did not attend training.
• Data is in a table format that compares salary package of candidates with and without training.