Exploratory Data Analysis (EDA)

Exploratory Data Analysis (EDA) is the initial investigation of data using various methods to gain insights without formal statistical modeling.

EDA helps in identifying errors in data collection, verifying assumptions, and suggesting appropriate statistical models for further analysis.

Non-graphical EDA techniques typically involve calculating summary statistics (like mean, median, variance) to understand the data's characteristics.

Graphical EDA uses visual representations like histograms, boxplots, and scatterplots to visualize data patterns and relationships.

Univariate EDA focuses on analyzing a single variable at a time to understand its distribution and characteristics like central tendency (mean, median), spread (variance, standard deviation), and shape (skewness, kurtosis).

Multivariate EDA explores the relationships between two or more variables. It helps to understand how variables influence each other.

Categorical data represents categories or groups, like gender (male, female) or college (H&SS, MCS, SCS).

Quantitative data represents numerical measurements, like age (20, 30, 40) or temperature (25°C, 30°C).

A histogram is a bar graph that visually represents the frequency distribution of a quantitative variable.

Mean is the average of all data values in a set, calculated by summing all values and dividing by the number of values.

Median is the middle value in a sorted dataset. If there are an even number of values, the average of the two middle values is taken.

Mode is the most frequent value in a dataset. Distributions can be unimodal (one peak), bimodal (two peaks), or multimodal (multiple peaks).

Variance is a measure of spread, indicating how far data points typically deviate from the mean. Calculated by averaging the squared deviations from the mean.

Standard Deviation is the square root of the variance. It has the same units as the original data, making it more interpretable than variance.

Interquartile Range (IQR) is a robust measure of spread, calculated as the difference between the third quartile (Q3) and the first quartile (Q1). The middle half of the data falls within this range.

Skewness is a measure of asymmetry in a distribution. A positive skew indicates a longer tail towards higher values, while a negative skew indicates a longer tail towards lower values.

Kurtosis measures the peakedness of a distribution compared to a Gaussian (normal) distribution. A positive kurtosis indicates a higher peak and fatter tails, while a negative kurtosis indicates a flatter peak and thinner tails.

A boxplot is a graphical representation of a quantitative variable, showing the median, quartiles, and outliers. It provides information about central tendency, spread, and outliers.

A quantile-normal plot (QN plot) is a graphical tool used to assess the normality of a distribution by comparing the quantiles of a dataset with the expected quantiles of a standard normal distribution.

Cross-tabulation is a technique used for examining the relationship between two categorical variables by displaying the counts of observations for each combination of categories.

Correlation, denoted by 'r', measures the strength and direction of linear relationship between two quantitative variables. It ranges from -1 to +1, indicating a perfect negative, no, or perfect positive correlation respectively.

Covariance, denoted by Cov(X,Y), measures the degree to which two quantitative variables vary together. A positive covariance indicates a tendency for both variables to increase or decrease together, while a negative covariance indicates an inverse relationship.

A scatterplot is a graphical representation of the relationship between two quantitative variables, where each data point represents a pair of values, with one variable on the x-axis and the other on the y-axis.

Degrees of freedom (df) represent the number of independent pieces of information available for calculating a statistic. In general, a variance or standard deviation calculated from 'n' data values has 'n-1' degrees of freedom.

Side-by-side boxplots are a powerful graphical EDA technique used to compare the distributions of a quantitative variable across different levels of a categorical variable. They provide a visual overview of central tendency, spread, and outliers for each group.

EDA is not a strict set of rules but an iterative process of exploring data to gain insights, check assumptions, identify errors, and suggest appropriate models for further analysis.

EDA is a crucial step in any data analysis workflow, providing valuable insights that guide further analysis and help make more informed decisions.

EDA involves a combination of non-graphical and graphical techniques to understand the characteristics of data, identify patterns and relationships, and guide further statistical analysis.

EDA is a powerful and versatile tool for exploratory data analysis, providing a foundation for deeper understanding and more effective statistical modeling.