Factor Analysis PDF

Factor Analysis – Introduction and Types (PCA, EFA) Factor Analysis is a statistical technique used primarily to reduce a large set of observed variables into a smaller set of underlying factors. It helps in identifying the structure of relationships among variables, revealing latent constructs or...

Factor Analysis – Introduction and Types (PCA, EFA) Factor Analysis is a statistical technique used primarily to reduce a large set of observed variables into a smaller set of underlying factors. It helps in identifying the structure of relationships among variables, revealing latent constructs or dimensions within the data. This is particularly useful in fields like psychology, marketing, social sciences, and education, where we deal with complex, multi-faceted constructs that cannot be measured directly (e.g., intelligence, customer satisfaction, or brand loyalty). There are two primary types of factor analysis: Principal Component Analysis (PCA) and Exploratory Factor Analysis (EFA). Each serves a unique purpose within the context of data reduction and interpretation. 1. Principal Component Analysis (PCA) Purpose of PCA PCA is a data reduction technique that aims to transform a set of correlated variables into a set of uncorrelated components. These components capture as much of the data's variance as possible, making it easier to interpret and visualize. Unlike other types of factor analysis, PCA does not assume the presence of latent variables. Instead, it focuses purely on summarizing the data by maximizing variance. How PCA Works  Standardization: If the variables are measured on different scales, they are often standardized.  Covariance Matrix: PCA calculates the covariance matrix of the standardized variables.  Eigenvalues and Eigenvectors: It extracts eigenvalues and eigenvectors from the covariance matrix. Each eigenvector corresponds to a principal component, and the eigenvalue represents the amount of variance explained by that component.  Principal Components: The components are ranked in order of explained variance. The first component explains the maximum variance, the second explains the next highest, and so on.  Dimension Reduction: We retain only those components that explain a significant portion of the variance (usually based on eigenvalues >1 rule or visual inspection using a scree plot). Applications of PCA  Dimensionality Reduction: PCA reduces data complexity by retaining only the most meaningful components.  Visualization: It’s particularly useful for visualizing high-dimensional data in a 2D or 3D plot.  Data Preprocessing: PCA is commonly used to preprocess data for other analyses, like clustering or regression, by reducing noise and multicollinearity. Key Points of PCA  PCA components are linear combinations of original variables.  It does not assume any underlying factors; it’s solely concerned with variance.  PCA is ideal when the primary goal is to reduce the dataset's dimensions while preserving as much information (variance) as possible. 2. Exploratory Factor Analysis (EFA) Purpose of EFA EFA is a statistical technique used to uncover the underlying structure of a relatively large set of variables. It assumes that latent constructs (factors) exist and are responsible for the correlations among observed variables. EFA is exploratory in nature, meaning it's used when the researcher does not have a specific hypothesis about the factor structure and wants to explore potential patterns within the data. How EFA Works  Correlation Matrix: EFA starts with calculating a correlation matrix among variables to find groups of variables that are highly correlated.  Factor Extraction: Several extraction methods are available, with Principal Axis Factoring and Maximum Likelihood being common options. These methods identify the underlying factors that explain correlations among variables.  Rotation: After initial factors are extracted, rotation methods (such as Varimax for orthogonal rotation or Promax for oblique rotation) make interpretation easier by simplifying the factor structure. Rotation helps in making factors more distinct and meaningful.  Factor Loadings: Each variable loads onto one or more factors. Factor loadings are coefficients that indicate the strength and direction of the relationship between each variable and the factor.  Number of Factors: Decisions about how many factors to retain are often guided by statistical criteria (e.g., eigenvalues >1) or inspection of a scree plot. Retaining the right number of factors is critical to balance model fit and interpretability. Applications of EFA  Scale Development: EFA helps in designing and validating psychometric scales by identifying groups of items that measure the same underlying construct.  Identifying Latent Constructs: In social sciences, EFA is frequently used to identify dimensions that represent latent constructs, like "customer satisfaction" or "academic performance."  Data Simplification: EFA reduces the number of observed variables by grouping them into factors, making it easier to interpret large datasets. Key Points of EFA  EFA assumes latent variables exist and cause correlations among observed variables.  It’s exploratory and does not impose any preconceived structure on the data.  Unlike PCA, EFA aims to model the underlying structure, making it suitable for discovering constructs rather than merely reducing dimensions. Differences Between PCA and EFA Feature PCA EFA Maximize explained variance, Identify underlying latent factors causing Goal data reduction observed correlations No assumptions about underlying Assumptions Assumes latent constructs exist factors Linear combinations of variables Factors with rotation for clearer Approach without rotation interpretation Dimension reduction, data Usage Scale development, identifying constructs visualization Output Components that explain variance Factors representing latent constructs Choosing Between PCA and EFA  Use PCA when the goal is purely dimensionality reduction without an interest in underlying constructs.  Use EFA when the goal is to uncover latent constructs or dimensions within the data. Step-by-step guide for performing Factor Analysis using Principal Component Analysis (PCA) and Exploratory Factor Analysis (EFA) in SPSS: Preliminary Steps Before running any factor analysis in SPSS, ensure you: 1. Define the variables: Confirm that all variables are continuous or ordinal. Factor analysis is generally not suited for categorical data. 2. Check correlations: Run a correlation matrix to verify that variables are reasonably correlated (a range of 0.3 to 0.8 is ideal). 3. Check sample size: Aim for at least 5 to 10 cases per variable to ensure stable factor analysis results. A. Performing Principal Component Analysis (PCA) in SPSS 1. Open the Data in SPSS  Open the dataset in SPSS with all the variables you wish to include in the PCA. 2. Navigate to the Factor Analysis Option  Go to Analyze > Dimension Reduction > Factor. 3. Select Variables  In the dialog box, move the variables you want to analyze into the Variables box. 4. Set the Extraction Method  Click on Extraction.  Select Principal Components as the extraction method (it is the default setting).  Choose Eigenvalues greater than 1 to determine the number of components to retain. Alternatively, you can use the Scree plot to visually assess the number of components.  Click Continue to return to the main dialog. 5. Set Rotation (Optional but Recommended)  Click Rotation.  Choose Varimax (for orthogonal rotation) or Promax (for oblique rotation) to make component interpretation easier.  Click Continue. 6. Generate Output Options  Click on Options.  Check Sorted by size to sort factor loadings in descending order, making them easier to interpret.  Select Suppress small coefficients (e.g., suppress coefficients below 0.3 or 0.4).  Click Continue. 7. Run PCA  Click OK to run the analysis. 8. Interpret the Output Factor Analysis: KMO and Bartlett's Test Kaiser-Meyer-Olkin Measure of Sampling.842 Adequacy. Bartlett's Test of Approx. Chi-Square 2268.354 Sphericity df 91 Sig..000 The KMO and Bartlett's Test results provide an initial assessment of the suitability of your data for Factor Analysis. Kaiser-Meyer-Olkin (KMO) Measure of Sampling Adequacy  The KMO value is 0.842, which is well above the minimum threshold of 0.6, typically considered acceptable. Values above 0.8 are considered "meritorious," indicating that the sample is adequate for conducting factor analysis.  A KMO value of 0.842 suggests that the correlations among variables are strong enough for a reliable factor analysis. This supports the validity of extracting factors from the dataset. Bartlett's Test of Sphericity  Bartlett's Test is used to determine if the correlation matrix is significantly different from an identity matrix (i.e., one in which all correlation coefficients are zero).  The Approximate Chi-Square value is 2268.354 with 91 degrees of freedom, and the p-value (Sig.) is 0.000, which is highly significant (p < 0.05).  This significant result indicates that there are relationships among the variables, supporting the appropriateness of factor analysis for this data. Conclusion With a high KMO value (0.842) and a significant Bartlett's Test (p = 0.000), the data meets the requirements for factor analysis. This suggests that the variables are likely to load onto underlying factors, which could represent the constructs of Organizational Support, Task Significance, Work Engagement, and Employee Tenure as defined in your items. Communalities Extractio Initial n OS1 1.000.725 OS2 1.000.598 OS3 1.000.692 OS4 1.000.728 ET1 1.000.720 ET2 1.000.683 ET3 1.000.638 TS1 1.000.741 TS2 1.000.672 TS3 1.000.660 WE1 1.000.780 WE2 1.000.703 WE3 1.000.646 WE4 1.000.817 Extraction Method: Principal Component Analysis. The Communalities table in Factor Analysis provides insight into the amount of variance in each variable that can be explained by the extracted factors. Here’s the interpretation of the results: Understanding Communalities  Initial values for all items are 1.000, which indicates that each item’s total variance is considered before extraction.  Extraction values show the proportion of each variable's variance that can be explained by the extracted factors after performing the factor analysis using Principal Component Analysis (PCA). Interpretation of Extraction Values The communalities after extraction reflect how much of each variable's variance is retained by the factor solution:  OS1 (0.725), OS2 (0.598), OS3 (0.692), and OS4 (0.728): For Organizational Support items, between 59.8% and 72.8% of the variance is explained by the factors. These are moderate-to-high communalities, indicating that these items are well represented by the factors extracted.   ET1 (0.720), ET2 (0.683), and ET3 (0.638): For Employee Tenure items, the communalities range from 63.8% to 72.0%, showing that these items also have a strong representation in the factor structure.  TS1 (0.741), TS2 (0.672), and TS3 (0.660): Task Significance items show that 66.0% to 74.1% of their variance is explained by the factors, indicating they are strongly represented in the factor solution.  WE1 (0.780), WE2 (0.703), WE3 (0.646), and WE4 (0.817): Work Engagement items show high communalities, ranging from 64.6% to 81.7%. These values suggest that Work Engagement items are very well captured by the extracted factors, with WE4 having the highest communality at 81.7%. Conclusion The extraction communalities show that each item retains a substantial proportion of its variance in the factor solution, especially items like WE4 (0.817) and TS1 (0.741). These high communalities confirm that the items are well-suited for factor analysis and effectively contribute to the factors identified. Total Variance Explained Extraction Sums of Squared Rotation Sums of Squared Initial Eigenvalues Loadings Loadings Compo % of Cumulat % of Cumulat % of Cumulati nent Total Variance ive % Total Variance ive % Total Variance ve % 1 4.709 33.638 33.638 4.709 33.638 33.638 2.954 21.103 21.103 2 2.183 15.592 49.230 2.183 15.592 49.230 2.720 19.431 40.534 3 1.575 11.253 60.483 1.575 11.253 60.483 2.074 14.816 55.350 4 1.336 9.544 70.027 1.336 9.544 70.027 2.055 14.676 70.027 5.640 4.573 74.600 6.544 3.888 78.488 7.505 3.608 82.096 8.478 3.416 85.512 9.409 2.922 88.435 10.376 2.685 91.120 11.367 2.622 93.742 12.350 2.502 96.244 13.294 2.098 98.341 14.232 1.659 100.000 Extraction Method: Principal Component Analysis. The Total Variance Explained table shows how much of the total variance in the data is captured by each extracted component, using Principal Component Analysis (PCA). Here’s an interpretation of the results: Initial Eigenvalues  Components with eigenvalues greater than 1 are typically considered significant, as they account for more variance than a single observed variable. Here, four components have eigenvalues greater than 1.  These four components explain 70.027% of the total variance in the data, which is a substantial amount. Generally, a cumulative variance of 60% or higher is considered acceptable in social sciences, so this result is satisfactory. Breakdown of Variance Explained  Component 1: Has an eigenvalue of 4.709, accounting for 33.638% of the variance.  Component 2: Has an eigenvalue of 2.183, explaining an additional 15.592% of the variance, bringing the cumulative explained variance to 49.230%.  Component 3: Has an eigenvalue of 1.575, contributing 11.253% more variance, totaling 60.483% cumulative variance.  Component 4: Has an eigenvalue of 1.336, adding 9.544% to the total, resulting in 70.027% cumulative explained variance. Rotation Sums of Squared Loadings After rotation, the variance distribution across components changes to improve interpretability:  Component 1 now explains 21.103% of the variance.  Component 2 explains 19.431%.  Component 3 explains 14.816%.  Component 4 explains 14.676%.  The total cumulative variance explained remains 70.027% after rotation, but the loadings are more evenly distributed across the components, which can make the factor structure clearer. Conclusion The four components together explain 70.027% of the total variance, which is a good level of variance explained, meeting the general threshold of at least 60% in social science research. This indicates that these four components provide a strong representation of the underlying constructs in your dataset. The chart shown is a Scree Plot from a Principal Component Analysis (PCA). Here’s an interpretation of the plot: Key Elements  Y-axis (Eigenvalue): Represents the eigenvalues for each component. An eigenvalue indicates the amount of variance captured by that component.  X-axis (Component Number): Represents each component in order of extraction. Interpretation of the Scree Plot  Sharp Drop: The plot shows a steep decline from Component 1 to Component 4, indicating that these first few components account for a substantial amount of variance in the data.  "Elbow" Point: After Component 4, the plot flattens out, meaning that subsequent components explain significantly less variance. This point, where the slope changes from steep to flat, is called the "elbow."  Optimal Number of Components: Typically, components before the "elbow" are retained, as they explain meaningful variance. In this case, the elbow appears at Component 4, suggesting that the first four components should be kept for further analysis. Conclusion The scree plot supports retaining four components for your analysis, as these components explain the most variance, while additional components contribute minimal variance. This aligns with the previous findings in the Total Variance Explained table, where the first four components accounted for 70.027% of the total variance. Component Matrixa Component 1 2 3 4 WE4.691 -.507 WE2.671 WE1.660 -.534 ET1.613.569 ET3.599.513 OS3.596.525 WE3.582 OS2.575 OS4.512.620 OS1.566.566 TS2.687 TS3.651 TS1.564.644 ET2.512.636 Extraction Method: Principal Component Analysis. a. 4 components extracted.  Rotated Component Matrixa Component 1 2 3 4 WE4.879 WE1.861 WE2.798 WE3.791 OS4.845 OS1.834 OS3.803 OS2.728 TS1.819 TS2.810 TS3.790 ET2.810 ET1.802 ET3.742 Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. a. Rotation converged in 5 iterations. The Rotated Component Matrix provides the factor loadings of each item on the four components after rotation, using Varimax with Kaiser Normalization. These loadings represent the correlations between each item and the factors, helping us to interpret the structure of the factors more clearly. Interpretation of Component Loadings A high loading (usually above 0.5) suggests that an item strongly associates with a particular component. Here’s the breakdown for each component:  1. Component 1: This component has high loadings for the Work Engagement (WE) items: o WE4 (.879), WE1 (.861), WE2 (.798), and WE3 (.791). o These high loadings indicate that Component 1 represents Work Engagement, as all items associated with this construct load strongly on this component. 2. Component 2: This component is characterized by high loadings for the Organizational Support (OS) items: o OS4 (.845), OS1 (.834), OS3 (.803), and OS2 (.728). o These items are all related to employees' perceptions of support and care from the organization, suggesting that Component 2 represents Organizational Support. 3. Component 3: This component has high loadings for the Task Significance (TS) items: o TS1 (.819), TS2 (.810), and TS3 (.790). o These items capture the meaningfulness and importance of the tasks performed within the organization, indicating that Component 3 represents Task Significance. 4. Component 4: This component shows high loadings for the Employee Tenure (ET) items: o ET2 (.810), ET1 (.802), and ET3 (.742). o These items reflect an employee’s sense of familiarity and experience within the organization due to their tenure, suggesting that Component 4 represents Employee Tenure. Conclusion The rotated component matrix reveals a clear factor structure with each component representing one of the four constructs:  Component 1: Work Engagement (WE)  Component 2: Organizational Support (OS)  Component 3: Task Significance (TS)  Component 4: Employee Tenure (ET) The rotation has improved interpretability by aligning items with their respective constructs, confirming that the items group as expected under each of the four factors. Component Transformation Matrix Component 1 2 3 4 1.601.519.397.459 2 -.677.731.080 -.010 3 -.209 -.297.914 -.181 4 -.369 -.327 -.018.870 Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. This is a Component Plot in Rotated Space from a Principal Component Analysis (PCA) with Varimax rotation. It provides a visual representation of how the items (variables) are distributed across the three main components in the rotated factor space. The plot is three- dimensional, with Component 1, Component 2, and Component 3 on the x, y, and z axes, respectively. Each labeled point represents an item from your dataset, and its position along each axis indicates its loading on that particular component. Conclusion This plot supports the distinctiveness and clarity of the four factors in your analysis:  Component 1: Likely represents Work Engagement.  Component 2: Likely represents Organizational Support.  Component 3: Likely represents Task Significance.  Component 4 (not directly labeled in the plot) appears to relate to Employee Tenure. The Component Plot in Rotated Space visually reinforces the results from the Rotated Component Matrix, confirming that each factor represents a unique set of related items. B. Performing Exploratory Factor Analysis (EFA) in SPSS 1. Open the Data in SPSS  Open the dataset in SPSS with all the variables you wish to include in the EFA. 2. Navigate to the Factor Analysis Option  Go to Analyze > Dimension Reduction > Factor. 3. Select Variables  Move the variables you want to analyze into the Variables box. 4. Set the Extraction Method  Click on Extraction.  Choose Principal Axis Factoring (or Maximum Likelihood, which is also common in EFA).  Choose Eigenvalues greater than 1 or select Scree plot to determine the number of factors to retain.  Click Continue. 5. Set Rotation Method  Click Rotation.  For uncorrelated factors, select Varimax (orthogonal rotation); for correlated factors, choose Promax (oblique rotation).  Click Continue. 6. Set Display and Output Options  Click on Options.  Choose Sorted by size to make interpretation easier.  Select Suppress small coefficients and specify a threshold (e.g., suppress coefficients below 0.3 or 0.4).  Check Coefficient Display Format for ease of interpretation.  Click Continue. 7. Run EFA  Click OK to run the analysis. 8. Interpret the Output  KMO and Bartlett's Test: Check these tests in the output. A KMO value >0.6 and a significant Bartlett’s Test (p < 0.05) indicate that your data is suitable for factor analysis.   Total Variance Explained: This shows how much of the variance in your data is explained by each factor. Ideally, the selected factors should explain at least 60% of the variance.  Scree Plot: Use this to determine the number of factors to retain. Look for the "elbow" in the plot.  Rotated Factor Matrix: This table shows the factor loadings of each variable on the factors. A loading above 0.5 indicates a strong association between the variable and the factor.  Interpret the Factors: Label each factor based on the high-loading variables it contains. These labels should describe the underlying theme or construct of each factor. Factor Rotation: Factor rotation is an essential step in factor analysis, enhancing the interpretability of the factors by adjusting the axis orientations in the factor space. Two primary types of factor rotation are Orthogonal rotation and Oblique rotation. 1. Factor Rotation Overview In factor analysis, the initial solution often results in factors that are difficult to interpret, with factor loadings spread across multiple variables. Factor rotation improves interpretability by maximizing high loadings and minimizing low loadings, effectively clarifying which variables load significantly on which factors. Rotation doesn’t change the underlying structure or the total variance explained by the factors; instead, it redistributes the variance among factors for clearer interpretability. 2. Types of Factor Rotation a. Orthogonal Rotation Orthogonal rotation maintains the right-angle (90 degrees) relationship between factors, meaning the factors remain uncorrelated (independent) from each other. This type of rotation is most suitable when factors are expected to be unrelated.  Characteristics of Orthogonal Rotation: o Factors are uncorrelated (i.e., they do not share variance). o Simpler interpretation due to the lack of correlation among factors. o Commonly used when factors are conceptually distinct.  Types of Orthogonal Rotation Methods: o Varimax Rotation: The most commonly used orthogonal rotation, Varimax aims to maximize the variance of squared loadings for each factor. This method enhances the clarity of which variables load strongly on each factor by increasing high loadings and reducing low loadings. Varimax is especially useful when seeking to identify clusters of variables with strong associations. o Quartimax Rotation: Quartimax minimizes the number of factors needed to explain each variable, focusing on simplifying the variables rather than the factors. It is less commonly used than Varimax, as it may lead to a "general factor" where many variables load onto a single factor.  o Equamax Rotation: Equamax combines Varimax and Quartimax approaches, seeking to balance the variance between factors and variables. However, it is less popular due to its complexity and less clear interpretative benefits.  Advantages: o Simplicity in interpretation since factors are uncorrelated. o The solution is easier to generalize if independent factors are expected in the data.  Disadvantages: o May not be suitable if factors are inherently related. o The constraint of keeping factors uncorrelated may limit the interpretability if the underlying constructs have some interdependence. b. Oblique Rotation Oblique rotation allows factors to be correlated, accommodating situations where underlying factors are conceptually or theoretically related. This rotation type is useful in social sciences, where variables often have some degree of interdependence.  Characteristics of Oblique Rotation: o Factors can be correlated, allowing for more realistic representation when constructs overlap. o Produces two matrices: The Pattern Matrix (factor loadings after rotation, similar to an orthogonal solution) and the Structure Matrix (correlations between variables and factors). o More flexible and often yields a more interpretable solution when factors are theoretically expected to be interrelated.  Types of Oblique Rotation Methods: o Promax Rotation: Promax rotation starts with an initial Varimax rotation and then raises loadings to a specified power to allow for correlations among factors. Promax is computationally efficient and suitable for large datasets, making it a popular choice when correlated factors are expected. o Direct Oblimin Rotation: The most commonly used oblique rotation method, Direct Oblimin, allows the degree of correlation between factors to be adjusted with a delta parameter. A lower delta value implies a higher correlation between factors. This method is useful when expecting factors to have a substantive correlation.  Advantages: o Provides a realistic structure if factors are naturally correlated. o Typically yields better interpretability in social science research where constructs are often interrelated.  Disadvantages: o More complex interpretation due to correlations among factors. o May require more careful reporting and explanation, particularly when differentiating between pattern and structure matrices. Parsimonious Factors: In research and data analysis, particularly in exploratory factor analysis and structural equation modeling, parsimonious factors refer to the selection of the simplest and most efficient set of factors that can adequately explain the relationships among observed variables. Parsimony is a guiding principle in model building, emphasizing simplicity and interpretability without sacrificing too much explanatory power. How to Achieve Parsimony in Factor Analysis? 1. Criteria for Factor Retention: The use of criteria such as eigenvalues greater than one (Kaiser’s criterion), scree plots, and parallel analysis helps to determine the number of factors to retain in a model. Often, these criteria suggest retaining a smaller number of factors that explain the majority of the variance. 2. Factor Loadings: Only variables with high factor loadings on a single factor should ideally be retained to ensure that each factor represents a distinct dimension. Variables with low or cross-loadings (loading significantly on multiple factors) are typically removed to achieve a more parsimonious structure. 3. Rotation Methods: Factor rotations, such as Varimax (orthogonal rotation) or Promax (oblique rotation), help simplify factor structures by making the loading pattern clearer, which aids in achieving a more parsimonious model. 4. Elimination of Redundant Factors: Factors that do not contribute significantly to the overall variance explained or are redundant with other factors can be eliminated, leading to a more efficient and parsimonious model. Summary Both PCA and EFA aim to distill large datasets into fewer, more interpretable factors. Parsimonious factors in PCA and EFA involve carefully selecting only the most meaningful components or factors, thus balancing simplicity and explanatory power. By focusing on parsimony, both PCA and EFA avoid overfitting, enhance replicability, and yield models that are simpler and easier to interpret—ideal for making data-driven decisions and identifying the underlying structure in data.