Factor Analysis and PCA Overview

Factor Analysis

• Factor Analysis (FA) is a statistical method that specifies a joint density model for data using fewer parameters.
• It infers the latent factors representing unobserved variables that influence the observed variables.
• There are challenges with unidentifiability in factor analysis, leading to multiple possible solutions.
• FA employs latent variables to express complex marginal distributions over observed variables using tractable joint distributions.
• The Expectation-Maximization (EM) algorithm is used to estimate the maximum likelihood estimator in this context.

Principal Components Analysis (PCA)

• PCA is a dimensionality reduction technique that aims to identify the principal components of a dataset, which capture maximum variance.
• It provides a linear transformation that projects the original data onto a lower-dimensional space.
• The principal components are orthogonal and are ordered in descending order of variance explained.
• The classical PCA theorem states that finding principal components is equivalent to maximizing the variance of the projected data.
• PCA involves finding the eigenvectors and eigenvalues of the covariance matrix of the data.

Singular Value Decomposition (SVD)

• SVD is used to perform PCA.
• It decomposes a matrix into three matrices: U, Σ, and V where U and V are orthogonal matrices, and Σ is a diagonal matrix containing singular values.
• The singular values represent the variance of the data along each principal component direction.

Choosing the Number of Latent Dimensions

• Choosing the number of latent dimensions is a critical aspect of factor analysis and PCA, impacting model complexity and interpretability.
• Common methods for choosing the number of latent dimensions include:
  • Scree plot: Visualizes the variance explained by each component.
  • Elbow method: Identifies a sharp decrease in variance explained, indicating an optimal cutoff.
  • Information criteria: Use metrics like AIC or BIC to trade off model complexity and fit.

Clustering

• Clustering is a technique for grouping data points such that points within a group are similar and dissimilar from points in other groups.
• It falls under unsupervised learning, where no target variable is available.
• The goal of clustering is to identify natural groupings or patterns within the data.
• Cluster shapes can be arbitrary, not necessarily circular.
• There are many clustering algorithms designed to detect arbitrary shapes.
• Clustering evaluates similarity using metrics like Euclidean distance, Cosine similarity, and Manhattan distance.

Types of Clustering

• Partitional clustering: Each data point belongs to only one cluster.
• Hierarchical clustering: Creates a nested set of clusters organized in a tree structure.

K-Means Clustering

• K-means clustering is a partitional clustering algorithm that aims to partition data into K clusters.
• It uses an iterative process to assign data points to clusters based on their proximity to cluster centroids.
• The goal is to minimize the sum of squared distances between data points and their assigned centroids.
• It involves a two-step process:
  • Assignment step: Assigning each data point to the nearest centroid.
  • Update step: Updating the centroids based on the assigned data points.
• The K-means algorithm works well for data with distinct clusters and can be accelerated using the Robbins-Monro procedure.
• Potential challenges with K-means include:
  • Depending on the initial centroid locations, the algorithm can converge to local optima.
  • Sensitivity to outliers.

Applications of K-Means

• Image segmentation: K-means clustering can be used to partition images into regions based on color or other pixel characteristics.
• Data compression: Applications in vector quantization, where a smaller number of K centroids represent the entire dataset, leading to reduced storage.

Mixtures of Gaussians

• Mixture of Gaussians is a probabilistic model used for clustering data.
• It assumes that the data is generated from a mixture of Gaussian distributions, each representing a different cluster.
• It is a flexible model than can handle clusters with different shapes and sizes.
• The model involves latent variables, which represent the cluster membership probabilities for each data point.
• The goal of the model is to estimate the parameters of the Gaussian distributions (mean, variance) and the mixture weights.

Maximum Likelihood in Mixtures of Gaussians

• The maximum likelihood approach is used to estimate the model parameters by maximizing the likelihood of the observed data given the model.
• The Expectation-Maximization (EM) algorithm is commonly used to find the maximum likelihood estimates.

EM for Gaussian Mixtures

• The EM algorithm alternates between two steps:
  • Expectation (E) step: Calculates the expected values of the latent variables given the current parameter estimates.
  • Maximization (M) step: Updates the parameter estimates by maximizing the expected complete-data log likelihood.

Alternative View of EM

• EM algorithm can be viewed as a way to maximize the incomplete-data log likelihood by iteratively improving the lower bound on the likelihood.
• This perspective highlights the key steps:
  • Finding a lower bound on the log likelihood using Jensen's inequality.
  • Maximizing the lower bound in the M-step, which improves the lower bound and the log likelihood.

Relation to K-Means

• K-means algorithm corresponds to a special case of EM for Gaussian mixtures.
• Key differences:
  • K-means uses hard assignment of data points to clusters, whereas EM uses a soft assignment based on posterior cluster membership probabilities.
  • K-means assumes same variance for all clusters.
• K-means is a faster but less flexible approach compared to EM for Gaussian mixtures.

Factor Analysis (Continued)

• Factor analysis extends beyond mixture models by allowing for continuous and correlated latent variables.
• It aims to explain a set of observed variables using a smaller number of underlying factors.
• Correlation among observed variables arises from their dependence on the common underlying factors.
• Similar to mixtures of Gaussians, factor analysis utilizes latent variables to model complex dependencies and simplify the observed data generation process.

Evaluating Clustering Output

• Evaluating the quality of a clustering solution is essential for assessing its performance.
• Common evaluation metrics include:
  • Purity: Measures how well clusters correspond to known class labels.
  • Rand index: Measures the agreement between the clustering and a known ground truth.
  • Mutual information: Quantifies the shared information between the clustering and the ground truth.

Hierarchical Clustering

• Hierarchical clustering creates a hierarchy of clusters, representing nested structures.
• Two main approaches:
  • Agglomerative clustering: Starts with each data point as a separate cluster and iteratively merges clusters until a single large cluster is formed.
  • Divisive clustering: Starts with a single cluster containing all data points and iteratively divides the clusters until each data point is in its own cluster.

Agglomerative Clustering

• Involves the following steps:
  • Begin with each data point in a separate cluster.
  • Iteratively merge the two closest clusters until a desired number of clusters is obtained.
  • The distance between clusters can be defined using various linkage criteria:
    • Single link: Uses the minimum distance between points in two clusters.
    • Complete link: Uses the maximum distance between points in two clusters.
    • Average link: Uses the average distance between all pairs of points in the two clusters.

Divisive Clustering

• Involves the following steps:
  • Begin with a single cluster containing all data points.
  • Iteratively split the cluster into two most dissimilar sub-clusters.
  • This process continues until each data point is in its own cluster.
  • The key challenge is to determine how to split the cluster based on some measure of dissimilarity.

Dissimilarity Analysis Steps

• 1. Compute a dissimilarity matrix between all pairs of objects.
• 2. Create a graph G with vertices representing objects and edges representing object relationships
• 3. Construct a minimum spanning tree whose edges are the most dissimilar ones.
• 4. Iteratively move objects between the clusters until a stopping criterion is met.
• 5. Stop the process when the difference in dissimilarity between the two clusters, G and H, becomes negative.