Statistics and Data Analysis

Handling Missing Values

• A set of fields with missing values can be handled using default values, mean values, or random values.
• There is no mode in the given list of numbers: 13, 15, 12, 17, 22, 11, 19.
• When handling missing values using means and modes, the mean is used for numeric fields and the mode is used for categorical fields.
• If the mode doesn't exist, a default value or a random value can be used.
• For numeric fields, the mean is calculated and approximated if necessary.
• For categorical fields, the mode is calculated from the existing values.

Handling Missing Values (Using Means and Modes)

• Field1 mean = 17.44
• Field3 mean = 334.44
• Field4 mean = 81.78
• Field2 is categorical, and its mode is A.

Handling Missing Values (Using Random Values)

• No additional information provided.

Handling Outliers

• Outliers are data values that deviate from expected values of the rest of the data set.
• Outliers are extreme values that lie near the limits of the data range or go against the trend of the remaining data.
• Outliers need more investigation to make sure they don't contain errors.

Handling Outliers Using Inter-quartile Range

• The inter-quartile range (IQR) is used to detect outliers.
• Q1, Q2, and Q3 are calculated using the following steps:
  • Order the data set in ascending order.
  • Use the median to divide the ordered data set into two halves.
  • The median is the second quartile (Q2).
  • The first quartile (Q1) is the median of the lower half of the data.
  • The third quartile (Q3) is the median of the upper half of the data.

Computing Q1, Q2, and Q3

• Example #1: Q1 = 15, Q2 = 40, Q3 = 43
• Example #2: Q1 = 17, Q2 = 37.5, Q3 = 40

Detecting Outliers using Inter-quartile Range

• IQR is calculated as Q3 - Q1.
• A data value is an outlier if it is less than (Q1 - 1.5IQR) or greater than (Q3 + 1.5IQR).
• Example: Data set 75000, -40000, 10000000, 50000, 99999 does not contain outliers.
• Example: Data set 75000, 40000, 10000000, 50000, 99999, 75000 contains an outlier, 10000000.

Noisy Data

• Noisy data are data that have incorrect values.
• Reasons for noisy data include:
  • Faulty data collection instruments
  • Human or computer errors during data entry
  • Transmission errors
  • Technology limitations

Smoothing Noisy Data

• Smoothing noisy data corrects errors using:
  • Validation and correction
  • Standardization

Validation and Correction of Noisy Data

• This step examines the data for data-entry errors and tries to correct them automatically as far as possible using:
  • Spell checking based on dictionary lookup for identifying and correcting misspellings.

This quiz covers concepts related to statistical analysis, including data occurrence and handling missing values.