Statistical Significance Tests

Significance Tests

• Choosing a statistical test depends on the research question, data type (categorical or continuous), and design (independent or repeated).
• T-tests are used for comparing means between two groups.
• ANOVA is used for comparing means across more than two groups.
• Parametric tests for comparing means assume normality and equal variance.
• Parametric tests include t-tests and ANOVA.
• A one-sample t-test compares a sample mean to a known value.
• An independent samples t-test compares two different groups.
• A paired samples t-test compares two related measurements from the same group.
• One-way ANOVA tests differences between more than two groups for one independent variable.
• One-way repeated measured ANOVA tests the same group under different conditions or times.
• Factorial ANOVA evaluates multiple independent variables simultaneously.
• Repeated measures factorial ANOVA combines within and between-subjects factors across conditions.
• Split-plot ANOVA is used when one factor is repeated (within-subject) and another is between-subject.

Fundamentals of Data Modeling

• Modeling involves building a mathematical representation of data relationships.
• Models are used to understand or predict outcomes.
• Predictor variables are independent variables.
• Response variables are dependent variables.
• Residuals are the differences between observed and predicted values.
• Model parameters are the values that define the model.
• Errors represent the differences between observed and predicted values.
• Errors include random noise and bias.
• The modeling process involves defining the problem, choosing a method, training the model, validating and testing it, and interpreting the results.
• Prediction aims for accuracy, while inference aims for understanding.
• Parametric methods assume a fixed form (e.g., linear regression).
• Non-parametric methods do not assume a fixed form (e.g., k-NN).
• Models like deep learning are accurate but less interpretable.
• Supervised learning uses labeled data (e.g., regression, classification).
• Unsupervised learning finds patterns in unlabeled data (e.g., clustering).
• Regression predicts continuous outcomes.
• Classification predicts categories.
• Mean Squared Error (MSE) measures the average squared difference between predicted and true values.
• Test MSE is more important than training MSE for generalization.
• Overfitting occurs when a model learns noise from the training data.
• Overfitting hurts the model's performance on new data.

Simple Linear Regression

• Use Simple Linear Regression (SLR) when predicting a quantitative outcome from one predictor.
• Estimating coefficients is done via least squares.
• Least squares minimizes the sum of squared residuals.
• The coefficient represents the change in outcome per unit increase in the predictor.
• The intercept is the expected outcome when the predictor is zero.
• R-squared (R²) indicates the proportion of variance explained by the model.
• R-squared ranges from 0 to 1.
• The F-statistic tests if the model as a whole is significant.

Multiple Linear Regression

• Use Multiple Linear Regression (MLR) when predicting a response from multiple predictors.
• Estimating coefficients still uses least squares but with multiple inputs.
• Each coefficient estimates the effect of its predictor, holding others constant.
• Variable selection is important for reducing overfitting and improving interpretability.
• Selection can be completed using p-values, adjusted R², stepwise regression, or LASSO.
• R² increases with more predictors.
• Adjusted R² accounts for the number of predictors, helping avoid overfitting.
• Non-linear relationships can be captured via polynomial regression (e.g., adding x² terms).
• Qualitative predictors can be handled via dummy (indicator) variables.

Unsupervised Learning

• Unsupervised learning uncovers hidden structures without labeled outcomes.
• Challenges of the technique include the lack of ground truth, deciding the number of clusters, and interpreting results.
• Applications include customer segmentation, topic modeling, and anomaly detection.
• K-means partitions data into k groups by minimizing within-cluster variation.
• Hierarchical clustering builds a tree of clusters using distance measures (e.g., complete linkage, average linkage).

Sampling and Resampling

• Probability sampling means every unit has a known chance of being selected.
• Examples of probability sampling include random and stratified sampling.
• Non-probability sampling means there's no known chance of being selected.
• An example of non-probability sampling is convenience sampling.
• Resampling for validation assesses model performance using subsets of data.
• Validation set approach splits data once into training and validation sets.
• LOOCV (Leave-One-Out Cross-Validation) trains on all but one data point, repeating for each data point.
• K-fold CV splits data into k parts, trains on k-1 parts, and tests on the remaining part, repeating k times.

Analyzing Text, Images, and More

• Understanding text involves parsing, tokenizing, and interpreting unstructured language data.
• Manual qualitative analysis is guided by coding schemes.
• Software such as NVivo or Atlas.ti can also guide manual qualitiative analysis.
• Social media analytics track sentiment, engagement, and topic trends.
• Sentiment analysis uses NLP to classify text as positive, negative, or neutral.

AI, Reinforcement Learning, and LLMs

• The three waves of AI include: Rule-based systems, Machine learning, and Contextual AI.
• Reinforcement learning involves agents learning through trial-and-error.
• Rewards and punishments are often used to train agents.
• Three machine learning approaches: Supervised, Unsupervised, and Reinforcement.
• Neural Networks are layered models that capture complex patterns in data and are inspired by the brain
• Large Language Models (LLMs) use transformer architecture.
• This architecture utilizes attention mechanisms to model context and relationships in text.
• Chain-of-Thought encourages reasoning by showing intermediate steps.
• Reinforcement Learning with Human Feedback (RLHF) fine-tunes models using human preferences.
• Hallucination is when an AI model generates false or made-up information.