Principal Component Analysis Overview

Study Notes

Introduction

Principal Component Analysis (PCA) is a statistical procedure transforming multiple possibly correlated variables into fewer uncorrelated variables called principal components.
It simplifies data by reducing the number of variables needed to explain most data variability.
PCA finds the directions of maximum variance in data, projecting data onto these directions.
This projection retains maximum information while decreasing dimensionality.

Key Concepts

Correlation: PCA handles variables whose values tend to change together.
Variance: Maximising variance explained by each principal component is crucial. High variance indicates more data information and stronger component descriptor strength.
Uncorrelated Variables: Principal components are orthogonal; no linear relationship exists between them.

Steps Involved in PCA

Standardization: Data is standardized (often using z-scores) to have zero mean and unit variance, preventing variables with larger scales from dominating analysis.
Covariance Matrix: The matrix showing pairwise relationships between variables is calculated. A covariance matrix entry at (i, j) represents covariance between variables i and j.
Eigenvalue Decomposition: The covariance matrix is decomposed to find its eigenvalues and eigenvectors. Eigenvalues represent variance explained by each principal component, and eigenvectors represent the principal components themselves (directions).
Eigenvector Sorting: Eigenvectors are ordered by descending eigenvalues. Larger eigenvalues’ eigenvectors capture more data variance.
Principal Components: Highest variance-capturing eigenvectors are the principal components, representing data in reduced dimensions.
Score Calculation: Data points are projected onto principal component axes to obtain scores, representing data in the new, reduced dimensional space.

Applications of PCA

Dimensionality Reduction: PCA reduces variables in machine learning tasks, aiding visualization and model building when dealing with massive data.
Data Visualization: PCA creates 2D or 3D plots to visualize high-dimensional data structures.
Feature Extraction: New features are created from existing correlated variables, revealing important patterns.
Noise Reduction: Noise not aligned with major variances is potentially filtered.
Image Compression: PCA reduces image storage needs.

Limitations

Interpretability: Principal Component meanings can be less clear than original variables, especially in complex datasets.
Information Loss: Reducing variables results in lost information, although PCA generally retains much variance. PCA only considers the variance direction, not other statistical measures (median, mode).
Sensitivity to Scaling: PCA is sensitive to variable scaling and requires prior standardization.
Assumption of Linearity: PCA assumes primarily linear relationships in data.
Non-linear Relationship Handling: PCA struggles with complex non-linear relationships or non-Gaussian data distributions.

Description

Explore the fundamentals of Principal Component Analysis (PCA), a key statistical technique for dimensionality reduction. This quiz covers important concepts such as correlation, variance, and the transformation of variables into uncorrelated components. Test your understanding of PCA and its applications in data analysis.