Independent Component Analysis PDF

Independent Component Analysis Adrian Mai, M.Sc. Neural Signal Analysis and Modeling Lecture Slides Winter 2022  Independent Component Analysis (ICA) Independent Component Analysis (ICA) – Multivariate analysis approach – Linear transformat...

Independent Component Analysis Adrian Mai, M.Sc. Neural Signal Analysis and Modeling Lecture Slides Winter 2022  Independent Component Analysis (ICA) Independent Component Analysis (ICA) – Multivariate analysis approach – Linear transformation – Applications are related to blind source separation Idea – Given signal may be a mixture of various different (independent) signals – Features of the individual signals may be obscured Hidden features may be revealed by unmixing the individual signals Any idea in which cases this might be helpful?  Graphical Representation Signals – 4 original source signals 𝑠𝑖 𝑡 – 4 recorded signals 𝑥𝑖 𝑡 – 4 estimations of the source signals 𝑠Ƹ𝑖 𝑡 – 𝑖 = 1, 2, 3, 4 James & Hesse 2015 Goal is the extraction of the individual source signals 𝑠𝑖 (𝑡) from the recorded signals 𝑥𝑖 (𝑡) without any prior knowledge about the original signals, in a way that the mutual information between – the original signal 𝑠𝑖 𝑡 and the estimation 𝑠Ƹ𝑖 𝑡 is maximized. – the different estimations 𝑠Ƹ𝑖 𝑡 is minimized. Estimations 𝑠Ƹ𝑖 (𝑡) are the independent components (IC) ICs are unknown and we want to recover the original sources Blind source separation  Assumptions Number of recorded signals is equal to the number of original source signals Source signals are linearly mixed – i.e., the recorded signals are linear combinations of the original source signals Source signals are statistically independent Source signals have a non-gaussian distribution – Central limit theorem states that the distribution of the sum of statistically independent random variables approaches a normal (gaussian) distribution – Inverse perspective is that the individual distrubutions are non-gaussian  Mathematical Formulation I 1 Signals – 4 original source signals 𝑠𝑖 𝑡 – 4 recorded signals 𝑥𝑖 𝑡 – 4 estimations of the source signals 𝑠Ƹ𝑖 𝑡 – 𝑖 = 1, 2, 3, 4 Step 1 James & Hesse 2015 – Source signals 𝑠𝑖 𝑡 are projected by the sources and linearly mixed on the way to the sensors – Each sensor receives a linear combination 𝑥𝑖 𝑡 of the source signals 𝑠𝑖 𝑡 𝑥 =𝑨 ⋅𝑠 𝑥 = sensor signals known 𝐴 = linear mixing matrix unknown 𝑠 = source signals unknown  Mathematical Formulation II 2 Step 2 – Recovery of the source signals 𝑠𝑖 𝑡 from the recorded sensor signals 𝑥𝑖 𝑡 may be desirable for certain applications – ICA unmixes the linear combinations 𝑥𝑖 𝑡 without any prior knowledge about the source signals based on a multivariate statistical procedure into estimations 𝑠Ƹ𝑖 𝑡 of the signals 𝑠𝑖 𝑡 James & Hesse 2015 – Rearranging our initital equation (𝑥 = 𝑨 ⋅ 𝑠) results in 𝑠 = 𝐴−1 ⋅ 𝑥 = 𝑊 ⋅ 𝑥 𝑠 = source signals unknown 𝑊 = 𝐴−1 = linear unmixing Matrix unknown 𝑥 = sensor signals known  Mathematical Formulation III Step 3 – Substitution of the source signals 𝑠𝑖 (𝑡) in the equation from step 2 by the estimations 𝑠Ƹ𝑖 (𝑡), as the unmixing matrix is unknown and the problem can‘t be solved perfectly but only via estimations James & Hesse 2015 𝑠Ƹ = 𝐴−1 ⋅ 𝑥 = 𝑊 ⋅ 𝑥 𝑠Ƹ = source signal estimations unknown 𝑊 = 𝐴−1 = linear unmixing Matrix unknown 𝑥 = sensor signals known Equation from step 3 represents the final problem of ICA – Identification of the unknown unmixing matrix 𝑊 for separation of the known linear combinations 𝑥 at the sensors into estimations 𝑠Ƹ of the original source signals 𝑠 – unmixing via transform matrix 𝑊 is a linear transform  Factorization of the Unmixing Matrix Factorization of the unmixing matrix 𝑊 via singular value decomposition and subsequent orthogonal diagonalization results in 1 − 𝑊 =𝑉⋅𝐷 2 ⋅ 𝐸𝑇 𝐷 and 𝐸 kann be computed via an eigendecomposition of the covariance matrix of our sensor data – 𝐷 = matrix in which the columns are the eigenvectors of the covariance matrix – 𝐸 = diagonal matrix in which the entries are the corresponding eigenvalues of the eigenvectors 𝑉 is a rotation matrix and has to be solved  Whitening Insertion of the factorized unmixing matrix into the initial equation results in 1 1 −2 −2 𝑠Ƹ = 𝑉 ⋅ 𝐷 ⋅ 𝐸𝑇 ⋅𝑥 =𝑉⋅ 𝐷 ⋅ 𝐸 𝑇 ⋅ 𝑥 = 𝑉 ⋅ 𝑥𝑤ℎ𝑖𝑡𝑒𝑛𝑒𝑑 Estimations 𝑠Ƹ of the original source signals correspond to a rotated version (rotated by 𝑉) of the whitened sensor signals 𝑥𝑤ℎ𝑖𝑡𝑒𝑛𝑒𝑑 Whitening – decorrelation in all dimensions by 𝐸 𝑇 1 − – Normalization of variance to 1 in all dimensions by 𝐷 2 (sphering) Shlens 2014  Solution of the Rotation Matrix I Final equation requires the solution of the rotation matrix 𝑉 𝑠Ƹ = 𝑉 ⋅ 𝑥𝑤ℎ𝑖𝑡𝑒𝑛𝑒𝑑 Solution of 𝑉 is an optimization problem to maximize the statistical independence of the individual components between each other – Corresponds to a maximization of the non-gaussianity of their distributions high gaussianity low gaussianity https://sccn.ucsd.edu/~arno/jsindexica.html  Solution of the Rotation Matrix II ICA assumes that the sources are statistically independent, i.e., 𝑁 𝑃 𝑠 = ෑ 𝑃(𝑠𝑖 ) 𝑖=1 Mutual information 𝐼 can be used to quantify the statistical independence of random variables 𝑃(𝑠) 𝐼 𝑠 = න 𝑃(𝑠) ⋅ 𝑙𝑜𝑔2 𝑑𝑠 ς𝑁 𝑖=1 𝑃(𝑠𝑖 ) Goal is to minimize the mutual information 𝐼 – Minimization is optimized in case of statistical independence – Logarithm and therefore the product reduce to zero  Solution of the Rotation Matrix III Entropy 𝐻 is a measure of the uncertainty of a distribution 𝐻 𝑠 = −∫ 𝑃 𝑠 ⋅ 𝑙𝑜𝑔2 𝑃 𝑠 𝑑𝑠 Mutual information 𝐼 can be expressed by the entropy 𝐻 𝑁 𝐼 𝑠 = ෍ 𝐻(𝑠𝑖 ) − 𝐻(𝑠) 𝑖=1  Solution of the Rotation Matrix IV Substitution of original signals 𝑠 by their estimations 𝑠Ƹ results in 𝑁 𝐼 𝑠Ƹ = ෍ 𝐻(𝑠Ƹ𝑖 ) − 𝐻(𝑠)Ƹ 𝑖=1 Setting 𝑠Ƹ = 𝑉 ⋅ 𝑥𝑤ℎ𝑖𝑡𝑒𝑛𝑒𝑑 and subsequent rearrangement results in 𝑁 𝐼 𝑠Ƹ = ෍ 𝐻((𝑉 ⋅ 𝑥𝑤ℎ𝑖𝑡𝑒𝑛𝑒𝑑 )𝑖 ) − 𝐻 𝑥𝑤ℎ𝑖𝑡𝑒𝑛𝑒𝑑 − 𝑙𝑜𝑔2 ( 𝑉 ) 𝑖=1 2nd and 3rd term can be ignored – 2nd term is independent of the rotation matrix we want to solve – 3rd term reduces to zero as the determinant of a rotation matrix is 1  Solution of the Rotation Matrix V Final equation 𝑁 𝐼 𝑠Ƹ = ෍ 𝐻((𝑉 ⋅ 𝑥𝑤ℎ𝑖𝑡𝑒𝑛𝑒𝑑 )𝑖 ) 𝑖=1 Final optimization problem 𝑁 𝑎𝑟𝑔𝑚𝑖𝑛 𝑉= = 𝐼 𝑠Ƹ = ෍ 𝐻((𝑉 ⋅ 𝑥𝑤ℎ𝑖𝑡𝑒𝑛𝑒𝑑 )𝑖 ) 𝑉 𝑖=1 Rotation matrix 𝑉 has to be chosen to minimize the sum of the marginal entropies – Implies the maximization of statistical independence as well as the maximization of the non-gaussianity of the distributions  Procedure Summary 1. Compute the covariance matrix of the source signals. 2. Perform an eigendecomposition on the covariance matrix. 3. Whiten the source signals using the obtained eigendecomposition. 4. Optimize the rotation matrix so that the statistical independence between the estimations is maximized.  Considerations Order of the ICs can be randomly permuted – i.e., the indices ICs are not necessarily related to the indices of the source signals ICs can be randomly flipped across the origin – i.e., the sign of the IC may be correct or inverted ICs can be rescaled with an arbitrary length – i.e., the scaling may differ from the scaling of the source signals Rotation matrix does not have an analytical form and must be approximated numerically via an optimization procedure  PCA vs. ICA PCA and ICA both identify new basis vectors for a data set PCA ICA Assumption on Orthogonal Statistically independent basis vectors (mathematically independent) (no orthogonality required) Assumption on No assumptions Non-gaussian data distribution Maximization of Statistical Maximization of variance absolute kurtosis optimization problem (i.e., non-gaussianity) Main application Dimensionality reduction Blind source separation  References C. J. James and C. W. Hesse. Independent Component Analysis for biomedical signals. Physiological Measurement, 26(1):R15-39, 2005. J. Shlens. A Tutorial on Independent Component Analysis. arXiv, 1404.2986v1, 2014.