CS4061D Data Analytics - Descriptive Statistics

Data Analytics (CS4061D) - Descriptive Statistics

• Data Summarization:
  • Used to identify typical data characteristics for an overview.
  • Used to identify data points that should be treated as noise or outliers.
  • Techniques categorized into measures of location and measures of dispersion.

Measurement of Location

• Also called measuring central tendency.
• Summarizes location information into a single number.
• Popular measures: mean, median, mode, midrange.
• Measured in three ways: distributive, algebraic, and holistic.

Distributive Measure

• Measures computed on subsets of a data set, then merged to arrive at a measure for the entire data.

Algebraic Measure

• Measures computed by applying algebraic functions to one or more distributive measures.
• Example: average = sum()/count()

Holistic Measure

• Measures computed on the entire data set as a whole.
• Example: calculating median.

Mean of a Sample

• Represented as X̄.
• Types: simple mean, weighted mean, trimmed mean.
• Assumed sample values of X₁, X₂, X₃..., X_n

Simple Mean

• Arithmetic mean or average (AM).
• Defined as the sum of all sample values divided by the number of sample values.
  • X̄ = (X₁ + X₂ + ... + X_n) / n

Weighted Mean

• Involves weights associated with each sample value.
  • X̄ = (ΣW_iX_i) / Σ W_i

Trimmed Mean

• Used to reduce the effect of extreme values in a data set.
• Obtained by removing a percentage of the highest and lowest values and calculating the mean from the remaining data.

Properties of Mean

• Lemma 3.1: Mean of combined samples.
• Lemma 3.2: Adding a new observation to a sample.
• Lemma 3.3: Removing an observation from a sample.
• Lemma 3.4: Adding/removing multiple observations to/from a sample.
• Lemma 3.5: Constant addition/subtraction from each value.
• Lemma 3.6: Multiplying/dividing each observation by a constant.

Mean with Grouped Data

• Data organized into classes with frequencies.
• Methods for calculating mean: direct method, assumed mean method, step deviation method.

Direct Method

• Calculating mean involves each value x_i for each frequency f_i added together. Formula: X̄ = Σ(f_ix_i) / Σf_i

Assumed Mean Method

• Select an assumed value A as a reference point to simplify calculations.

Step Deviation Method

• Reduces calculation complexity when the class intervals are large, by dividing each deviation from the assumed mean (A) by a common value (h), the step difference.

Ogive: Graphical Method

• Used for finding the mean of grouped data
  • Plot cumulative frequencies against the class limits
  • "Less-than" and "More-than" ogives

Other Measures of Mean

• Arithmetic Mean (AM)
• Geometric Mean (GM)
• Harmonic Mean (HM)
• Relationship between AM, GM, and HM:
  • AM ≥ GM ≥ HM

Median of a Sample

• The middle value when data is arranged in order.
• Formula for odd and even number of observations exists to calculate the median

Median of Grouped Data

• The median is the middle value from the data set
• Involves classes and frequency
• A class is called median class when its cumulative frequency is greater than N/2

Mode of a Sample

• The most frequently occurring observation in a data set.

Mode of Grouped Data

• Identifies the modal class (highest frequency). Formula exists to calculate the class where the mode will lie.

Relation Between Mean, Median, and Mode

• Symmetric data: Mean, median, and mode are the same.
• Positively skewed data: Mode < Median < Mean
• Negatively skewed data: Mean < Median < Mode

Empirical Relation

• Relation between mean, mode, and median for moderately skewed data.

Midrange

• It is the average of the largest and smallest values in a data set.

Measures of Dispersion

• Measures how spread the data is from the mean
• Examples:
  • Range
  • Variance and Standard Deviation
  • Mean Absolute Deviation (MAD)
  • Absolute Average Deviation (AAD)
  • Interquartile Range (IQR)

Range of a Sample

• Represents the difference between maximum and minimum values in the sample. - A measure of the spread of data.

Variance and Standard Deviation

• Measures how spread the data is from the average.
  • Variance: σ² = Σ (xi – X̄)² / (n – 1)
  • Standard Deviation: the square root of variance (σ).
  • The sample mean and denominator are adjusted in formula to account for limited spread of sample

Coefficient of Variation (CV)

• Represents the ratio of standard deviation to the mean, expressed as a percentage.
• CV = (σ/X̄) * 100
• Used to compare dispersion of data sets with different means.

Mean Absolute Deviation (MAD)

• Robust alternative to variance • Median of absolute differences between each data point and the data's mean

Interquartile Range (IQR)

Box Plot

• Graphical representation of five number summary
  • Minimum, Q1, Median (Q2), Q3, Maximum