Introduction to Binary Logistic Regression

Introduction to Binary Logistic Regression

• Binary logistic regression models the probability of a binary outcome (e.g., success/failure, yes/no).
• It extends linear regression, using a logistic function for probability modeling.
• Predicted probabilities always fall between 0 and 1.
• Unlike linear regression, it handles categorical dependent variables.

Key Concepts

• Dependent Variable: A categorical variable with two outcomes (e.g., disease presence/absence, customer churn/retention).
• Independent Variables: Predictor variables, continuous or categorical, used to model the outcome probability.
• Logistic Function (Sigmoid Function): Transforms a linear combination of independent variables into a probability between 0 and 1, using the formula: (e^(z)) / (1 + e^(z)), where 'z' is the linear combination.

Model Building

• Model Equation: Describes the relationship between independent variables and the log-odds of the binary outcome.
  • Log-odds = ln[p/(1-p)] = β₀ + β₁X₁ + β₂X₂ + ... + βₙXₙ.
    • p is the outcome probability.
    • β₀ is the intercept.
    • βᵢ are coefficients for independent variable Xᵢ.
• Parameter Estimation: Maximum likelihood estimation (MLE) finds optimal coefficients (βs) to maximize the likelihood of observing the data.
• Model Evaluation: Measures the model's predictive ability.
  • Goodness-of-fit: Measures overall model fit to the data's distribution.
  • Hosmer-Lemeshow test: Assesses if predicted and observed probabilities match.
  • Pseudo-R-squared values: Measure variance explained, analogous to linear regression's R-squared, but adapted for logistic regression.
  • Classification Table: Presents model predictions as correct or incorrect.
  • Accuracy, Sensitivity, Specificity, and Precision: Quantify classification accuracy:
    • Accuracy=(TP + TN)/(TP + TN + FP + FN) (True Positives + True Negatives) / Total
    • Sensitivity (Recall) = TP/(TP + FN) (True Positives) / (True Positives + False Negatives)
    • Specificity = TN/ (TN + FP) (True Negatives) / (True Negatives + False Positives)
    • Positive Predictive Value (Precision) = TP/ (TP + FP) (True Positives) / (True Positives + False Positives)

Assumptions

• Independent Errors: Prediction errors are independent, crucial for MLE validity.
• Linearity: Independent variables have a linear relationship with log-odds, not necessarily probability.
• Proper Categorization: Accurate variable categorization is essential; incorrect categorization impacts results' validity and reliability; the dependent variable contains values that dictate the success or failure of an event.

Interpretation of Results

• Coefficients (βs): Represents the effect of a one-unit change in an independent variable on the log-odds of the outcome.
• Odds Ratios: Exp(βᵢ) shows the change in outcome odds for a one-unit change in an independent variable, providing a better measure of effect size.

Applications

• Medical Diagnosis: Predicts disease likelihood given symptoms.
• Marketing: Assesses customer purchase likelihood.
• Finance: Evaluates credit risk and default probability.
• Social Sciences: Models survey response likelihood.

Model Limitations

• Overfitting: Overly complex models can result, incorporating too many variables or interactions.
• Multicollinearity: High correlation among independent variables makes isolating individual effects difficult.
• Missing Data: Proper missing value handling is essential.
• Limited Generalizability: Model accuracy relies on training data quality; accurate models in one dataset may not generalize well to new data.