Principle Component Analysis (PCA)

Principle Component Analysis

• Goal is to reduce data dimension while preserving information.
• Applications: visualization, feature extraction, data compression.
• An unsupervised learning algorithm.
• A linear dimensionality reduction technique.

Math Behind PCA

• Center the data by subtracting the mean from each data point: $\tilde{x}i = x_i - \mu$, where $\mu = \frac{1}{n}\sum{i=1}^n x_i$
• Compute the covariance matrix: $\Sigma = \frac{1}{n}\sum_{i=1}^n \tilde{x}_i \tilde{x}_i^T$
• Calculate eigenvectors and eigenvalues of Σ: $\Sigma v_i = \lambda_i v_i$, where $v_i$ is the $i$-th eigenvector and $\lambda_i$ is the $i$-th eigenvalue.
• Sort eigenvectors by their eigenvalues: $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_D$
• Select the first K eigenvectors: $V = [v_1, \dots, v_K]$
• Project data onto the new subspace: $y_i = V^T \tilde{x}_i$, where $y_i \in \mathbb{R}^K$
• Variance explained is calculated as: $\frac{\sum_{i=1}^K \lambda_i}{\sum_{i=1}^D \lambda_i}$

PCA Example Calculation

• Given $X = \begin{bmatrix} 1 & 2 \ 3 & 4 \ 5 & 6 \end{bmatrix}$
• $\mu = \begin{bmatrix} 3 \ 4 \end{bmatrix}$
• $\tilde{X} = \begin{bmatrix} -2 & -2 \ 0 & 0 \ 2 & 2 \end{bmatrix}$
• $\Sigma = \frac{1}{3} \begin{bmatrix} -2 & -2 \ 0 & 0 \ 2 & 2 \end{bmatrix} \begin{bmatrix} -2 & 0 & 2 \ -2 & 0 & 2 \end{bmatrix} = \frac{1}{3} \begin{bmatrix} 8 & 8 \ 8 & 8 \end{bmatrix}$
• $\Sigma v_i = \lambda_i v_i \Rightarrow \begin{bmatrix} 8 & 8 \ 8 & 8 \end{bmatrix} \begin{bmatrix} v_{11} \ v_{21} \end{bmatrix} = \lambda_i \begin{bmatrix} v_{11} \ v_{21} \end{bmatrix}$
• $\lambda_1 = \frac{16}{3}$, $v_1 = \begin{bmatrix} \frac{\sqrt{2}}{2} \ \frac{\sqrt{2}}{2} \end{bmatrix}$ and $\lambda_2 = 0$, $v_2 = \begin{bmatrix} -\frac{\sqrt{2}}{2} \ \frac{\sqrt{2}}{2} \end{bmatrix}$
• $V = \begin{bmatrix} \frac{\sqrt{2}}{2} \ \frac{\sqrt{2}}{2} \end{bmatrix}$
• $Y = V^T \tilde{X} = \begin{bmatrix} \frac{\sqrt{2}}{2} & \frac{\sqrt{2}}{2} \end{bmatrix} \begin{bmatrix} -2 & -2 \ 0 & 0 \ 2 & 2 \end{bmatrix} = \begin{bmatrix} -2\sqrt{2} & 0 & 2\sqrt{2} \end{bmatrix}$

PCA in Python

• Use sklearn.decomposition and numpy.
• Example implementation:

from sklearn.decomposition import PCA
import numpy as np

X = np.array([[1, 2], [3, 4], [5, 6]])

pca = PCA(n_components=1)
pca.fit(X)

print(pca.explained_variance_ratio_)
print(pca.singular_values_)

PCA Limitations

• It's a linear dimensionality reduction technique.
• Sensitive to outliers.
• Assumes data is normally distributed.
• Not scale-invariant.

Independent Component Analysis

• Goal is to separate a multivariate signal into statistically independent additive subcomponents.
• Applications include blind source separation and feature extraction.
• ICA is an unsupervised learning algorithm.
• It uses linear transformation techniques.

Math Behind ICA

• Given data $X = {x_1, \dots, x_n}$, $x_i \in \mathbb{R}^D$ where $x_i = As_i$, with $s_i$ being the independent components and $A$ being the mixing matrix.
• Center the data: $\tilde{x}i = x_i - \mu$, where $\mu = \frac{1}{n}\sum{i=1}^n x_i$
• Whiten the data: $\hat{x}_i = W\tilde{x}_i$, where $W = (\mathbb{E}[\tilde{x}\tilde{x}^T])^{-\frac{1}{2}}$
• Estimate the unmixing matrix: $B = (W^T)^{-1}$
• Recover the independent components: $s_i = Bx_i$

ICA in Python

• Use sklearn.decomposition and numpy.
• Example implementation:

from sklearn.decomposition import FastICA
import numpy as np

X = np.array([[1, 2], [3, 4], [5, 6]])

ica = FastICA(n_components=2)
ica.fit(X)

print(ica.components_)
print(ica.mixing_)

Understanding R&D

• Research and Development (R&D) activities drive innovation and introduce new products/services by companies or governments.
• Initial phase in the development of a novel service or product.

R&D Categories

• Basic Research: Advances scientific knowledge without commercial goals.
• Applied Research: Focuses on practical application of basic research findings.
• Development: Uses research findings to create new products or processes.

Benefits of R&D

• Economic Growth: New products and services stimulate economic activity.
• Improved Productivity: Innovation often leads to more efficient processes.
• Competitive Advantage: Companies gain an edge by being first to market.
• Societal Benefits: Advances improve lives in healthcare, technology, and other fields.

Challenges of R&D

• Uncertainty: Outcomes are not guaranteed; projects may fail.
• High Costs: Requires significant investment in equipment and skilled personnel.
• Long Timeframes: Can take years to bring a new product/service to market.
• Market Acceptance: Even successful projects may not be commercially viable.

R&D Spending

• Varies across industries and countries, reflects competitiveness.

Examples of R&D

• Pharmaceutical companies researching new drugs.
• Engineering firms developing energy-efficient engines.
• Software companies creating new mobile applications.

Tax Incentives for R&D

• Many countries offer them to encourage investment in R&D, such as tax credits or deductions.

R&D Process

• Idea Generation
• Research and Experimentation
• Prototyping
• Testing
• Analysis
• Product Development
• Market Launch

R&D in Start-Ups

• Crucial but poses unique challenges (limited resources).
• Often relies on external funding or partnerships to conduct R&D.