Untitled

• Exploratory Data Analysis(EDA) is a data analytics process to understand data in depth and learn its characteristics, often using visual means.

Importance of EDA

• Understanding the dataset, features, data types, and data spread to choose analysis methods
• Spot hidden patterns, relationships between data points for model building
• Spot errors or outliers that affect results
• EDA insights help decide which features are most important for building models and to improve performance
• Understanding the data aids in choosing the best modelling techniques and adjust them for better results

Types of EDA

• Univariate Analysis
• Bivariate Analysis
• Multivariate Analysis

Univariate Analysis

• It focuses on studying one variable to understand their characteristics
• It describes data and finds patterns within a single feature.
• Summary statistics like mean, median, mode, variance, and standard deviation help describe the central tendency and data spread
• Commonly uses histograms to show data distribution, box plots to detect outliers and understand data spread, and bar charts for categorical data

ANOVA

• ANOVA (Analysis of Variance) is a statistical method to test whether there are significant differences between the means of two or more groups
• ANOVA returns two parameters F-test score and P-value
• F-test Score assumes the means of all groups are the same, calculates how much the actual means deviate from the assumption, and reports it as F-test score; a larger score means there is a larger difference between the means
• P-value tells how statistically significant the calculated score value is, and if it is below a predefined threshold, at least one group has a significantly different mean

Bivariate Analysis

• Bivariate analysis explores the relationship between two variables to find connections, correlations, and dependencies
• Scatter plots visualize the relationship between two continuous variables
• Cross-tabulation, or contingency tables, shows the frequency distribution of two categorical variables for understanding their relationship
• Correlation coefficient measures how strongly two variables are related, often using Pearson's correlation for linear relationships

Regression Plots

• Regression plots create a regression line between 2 parameters and visualize their linear relationships
• A regression line represents the best-fit line that predicts the dependent variable based on the independent variable

Correlation vs Causation

• Correlation is a measure of the extent of interdependence between variables.
• Causation is the relationship between cause and effect between two variables
• It is important to understand the difference between Correlation and Causation
• Correlation does not imply causation

Pearson’s Correlation

• Measures the linear dependence between two variables X and Y
• The resulting coefficient is a value between -1 and 1 inclusive.
  • 1: Total positive linear correlation
  • 0: No linear correlation, variables most likely do not affect each other
  • -1: Total negative linear correlation

P-Value

• The P-value is the probability value that the correlation between two variables is statistically significant
• A significance level of 0.05 is typically chosen, meaning 95% confidence that the correlation between the correlation is significant
• By convention, when
  • the p-value is < 0.001: strong evidence the correlation is significant
  • the p-value is < 0.05: moderate evidence the correlation is significant
  • the p-value is < 0.1: weak evidence the correlation is significant
  • the p-value is > 0.1: there is no evidence the correlation is significant

Multivariate Analysis

• Examines the relationships between two or more variables in the dataset to understand how variables interact.
• It includes techniques like pair plots, which show the relationships between multiple variables at once, helping to see how they interact
• It includes Principal Component Analysis (PCA), which reduces the complexity of large datasets by simplifying them, while keeping the most important information

Specialized EDA

• Specialized EDA techniques are tailored for data and analysis needs.
  • Spatial Analysis is for geographical data, using maps and spatial plotting to understand the geographical distribution of variables
  • Text Analysis uses techniques like word clouds, frequency distributions, and sentiment analysis to explore text data
  • Time Series Analysis is applied to statistics sets that have a temporal component, and entails inspecting and modeling styles, traits, and seasonality
    • Techniques like line plots, autocorrelation analysis, transferring averages, and ARIMA are generally utilized in time series analysis

Data Exploration

• Exploring data characteristics involves examining distribution, central tendency, and any outliers or anomalies
• It selects appropriate analysis methods and to spot potential data issues
• Should calculate summary statistics (mean, median, mode, standard deviation, skewness, kurtosis) for numerical variables
• These provide an overview of distribution and help identify any irregular patterns or issues

Descriptive Statistics

• Refers to a branch of statistics that involves summarizing, organizing, and presenting data meaningfully and concisely
• Focuses on describing and analyzing a dataset's main features and characteristics without generalizations or inferences
• Primary goal is to provide a clear and concise summary of the data, enabling researchers or analysts to gain insights and understand patterns, trends, and distributions
• Typically includes measures like central tendency, dispersion (range, variance, standard deviation), and shape of the distribution (skewness, kurtosis)
• Involves a graphical representation of data through charts, graphs, and tables, which aids data visualization and interpretation

Visualizing Data Relationships

• Visualization is a powerful EDA tool that uncovers relationships
• It also identifies patterns or trends that may not be obvious from summary statistics alone
• For categorical variables, create frequency tables, bar plots, and pie charts to understand the distribution of categories, identify imbalances or unusual patterns
• For numerical variables, generate histograms, box plots, violin plots, and density plots to visualize distribution, shape, spread, and potential outliers
• To explore relationships between variables, use scatter plots, correlation matrices, or statistical tests like Pearson's correlation coefficient or Spearman's rank correlation.

Handling Outliers

• Outliers are data points that significantly differ from the rest of the data, often caused by errors in measurement or data entry
• Detecting and handling outliers is important because they can skew your analysis and affect model performance
• Outliers can be identified using methods like interquartile range (IQR), Z-scores, or domain-specific rules
• Once identified, outliers can be removed or adjusted depending on the context
• Properly manage outliers ensures analysis is accurate and reliable