Exploratory Data Analysis Basics

Exploratory Data Analysis (EDA)

• EDA uses visual and numerical methods to summarize data and identify patterns, anomalies, and relationships.
• It's crucial for understanding datasets before more complex analyses.

Descriptive Statistics

• Tables and Graphs: Frequency tables count data categories; graphs (bar graphs, pie charts) show frequency distributions for categorical and quantitative data.
• Bar Graphs: Each bar represents a category, height corresponds to frequency or relative frequency.
• Histograms: Used for quantitative variables, bars show intervals of values; heights correspond to frequencies. Visualize distributions of quantitative data, like the Human Development Index (HDI).
• Box Plots: Summarize five key statistics (min, Q1, median, Q3, max); display outliers as individual points.

Measures of Central Tendency

• Mean (ȳ): Calculated as Σyi / n; sensitive to outliers. Useful for data with roughly symmetrical distribution, like GDP growth rates but can be misleading if data has extreme values.
• Median: Middle value in ordered data; resistant to outliers. Useful if a dataset has extreme values or skewed distribution, like in income distributions where a few individuals earn very high incomes.
• Mode: Most frequent value; applicable to both categorical and quantitative data. Useful for finding the most common value, like in household sizes.

Measures of Variability

• Range: Difference between the largest and smallest observations. Shows the overall spread of the data.
• Standard Deviation (SD): Measures spread around the mean; larger SD indicates more variability (e.g., number of computer crashes or income variability between nations).
• Variance: Square of the standard deviation, shows the variability in earnings data.
• Interquartile Range (IQR): Difference between the 75th and 25th percentiles; less sensitive to outliers.

Graphical Representations of Data

• Frequency Distributions: Summarizes data into intervals for quantitative data.
• Relative Frequencies: Highlight proportions, for example, gender representation in Parliament.

Practical Data Analysis Tools

• Z-Scores: Measures distance from the mean in standard deviation units.
• Empirical Rule: For bell-shaped distributions, approximately 68%, 95%, and 99.7% of data fall within one, two, and three standard deviations from the mean, respectively.
• Percentiles and Quartiles: Indicate the percentage of data below a certain value. Quartiles are specific percentile values (25th percentile = Q1, 75th percentile = Q3).

Association Between Variables

• Association: Exists if values of one variable tend to occur with specific values of another variable.
• Contingency Tables: Display joint frequencies of variables, with rows and columns corresponding to explanatory and response variables.
• Marginal Totals: Row and column totals; summarize variable distributions.
• Conditional Distributions: Proportions showing how response variables vary by explanatory variables.

Chi-Square Test of Independence

• Purpose: Test whether two categorical variables are independent in a population.
• Key Steps: Calculate expected frequencies, compute test statistic, compare test statistic to critical value or compute p-value to determine independence.

Scatterplots and Relationships

• Scatterplots: Visual representation of two variables' relationship; useful for identifying linear or non-linear patterns, outliers and clusters.
• Covariance: Measures the direction of a linear relationship. Positive–values indicate that the variables tend to move in the same direction, negative values indicate that they tend to move in opposite directions.
• Correlation Coefficient (r): Standardized measure of the strength and direction of a linear relationship. Ranges between -1 and 1.

Regression Analysis

• Simple Linear Regression: Model relationship between a response variable (y) and an explanatory variable (x).
• Model: y = a + bx + €.
• Residual Analysis: Important for evaluating model fit and assumptions; residuals represent the difference between observed and predicted values. Key checks include mean zero, random distribution, and consistent spread (homoscedasticity).
• Multiple Regression: Extends simple linear regression to include multiple predictor variables. It uses metrics, like R-squared, to assess the goodness of fit.

Hypothesis Testing

• Global Test (Multiple Regression): Tests whether all predictor coefficients are zero.
• Individual Tests (Multiple Regression): Tests whether individual predictor coefficients are zero.

Multicollinearity

• Definition: High correlation between predictor variables, inflates variance of regression estimates.
• Detection: Variance Inflation Factor (VIF).
• Solution: Remove redundant predictors or use regularization.

Dummy Variables

• Incorporating Categorical Predictors: Dummy variables represent categorical predictors in regression.

Cluster Analysis

• Definition: Groups data based on similarity.
• Types: Hierarchical (agglomerative or divisive), non-hierarchical (e.g., K-means).
• Distance Measures: Euclidean, Manhattan (used for calculating distances between observations).

Validating Clusters

• Optimal Number of Clusters: Elbow method, gap statistic.
• Cluster Validation Statistics: Measures like silhouette scores.

Visualization Techniques

• Chi-Square Distribution: Right-skewed shape; critical for hypothesis testing.
• Scatterplots: Visualize relationships between variables.
• Box Plots: Summarize data variability.
• Residual Plots: Assess model fit, detecting non-linear relationships.