Binomial Confidence Intervals

• The document explores how to determine binomial confidence intervals for rare events, emphasizing the importance of defining the margin of error relative to the magnitude of the proportion

Key Concepts

• Confidence interval performance is assessed by coverage probability and interval width (margin of error)
• Focus is on rare-event probabilities and performance of four proportion interval estimators: Wald, Clopper-Pearson, Wilson, and Agresti-Coull
• Precision is defined by a relative margin of error, ensuring consistency with the proportion's magnitude
• Estimators are assessed by their ability to achieve desired coverage probability while meeting specified relative margin of error
• Both coverage probability and relative margin of error must be considered when estimating rare-event proportions
• All four interval estimators perform similarly within sample size and confidence level for a given framework
• Relative margin of error values identified result in satisfactory coverage and conservative sample size requirements
• Employs analytical evaluation, simulation, and application to recent studies

Introduction

• Applied statistics rely on constructing a confidence interval (CI) for a binomial proportion, p
• Many applications involve a large population where the event of interest is rare
• Proportion includes side effects, or defective components
• Order of magnitude includes Covid and safety incidents

Order of Magnitude

• Ascertaining the order of magnitude of p is critical in "large populations."
• Practical differences exist between p = 10^-4 and p = 10^-6 in populations of millions, impacting defect rates and product returns
• 10,000 observations are needed to get one event when p = 10^-4
• Specific guidance on sample size requirements is needed due to literature lacking coverage

Prior Work

• Constructing a CI for p has a wide literature, with comparative studies
• Existing studies tend to focus on situations where p is moderately large
• Little guidance exists for the scenario where p is small
• Little discussion on relative margin of error is needed in low p setting
• Relative margin of error essential where margin of error scales with magnitude for rare events.
• Focus on small p regime of p ∈ [10^-6,10^-1], where relative margin of error is essential
• Considers Wald interval, Clopper-Pearson, Wilson, and Agresti-Coull

Binomial Proportion Interval Estimators

• Assessing performance of Wald, Clopper-Pearson, Wilson, and Agresti-Coull without modification or continuity correction
• Wald interval is included because its most widely used
• Assesses Clopper and Pearson as an exact method
• Offers compromise between the Wald interval, and the Clopper-Pearson interval
• Assesses performance of the Wilson and Agresti-Coull intervals with popularity.

Estimating Proportions When No Events are Observed

• Considers situation where no events are observed that estimates rare-event probabilities
• Sample results in x = 0, and, hence, p = 0/n = 0, when p is likely small.
• Resulting intervals are very conservative for small n where no events are observed
• Interval examples W: [0, 0], CP: [0, 0.0362], WS: [0, 0.0370], AC: [-0.0074, 0.0444] where the Wald interval is degenerate
• The other intervals are too wide where a good estimate of magnitude is required
• Sample size of n=100 will not work
• Requires reasonable estimate of the order of magnitude of p.

Evaluation Criteria

• Coverage probability and expected width are the most commonly used CI evaluation criteria
• Considers an interval as non-rejected parameter values in a hypothesis test and discuss the p-confidence and p-bias criteria
• Employs root mean squared error and mean absolute deviation to measure CI performance

Coverage Probability

• Coverage probability can be interpreted as the computed interval's coverage percentage
• Denotes Lx and Ux as the lower and upper CI bounds and the expected coverage probability is given by

CPr(n,p) = Σ pˣ(1 − p)ⁿ⁻ˣ1(Lx ≤ p ≤ Ux)

Expected Width

EW(n, p) = Σ (n over x) pˣ(1-p)ⁿ⁻ˣ(Ux − Lx)

• Expected margin of error calculated as and the EMoE(n, p) = EW(n,p)/2

Calculating Sample Size

• Derives the sample size from the CI formula with fixed ER = €/p which is the first problem in CI estimation
• Used in conjunction with the Wald margin of error

n=z²/₂(1-p²)/ER²p*

• p* is the initial estimate

Initial Estimate of p

• Selecting a value for p* is required to make the earlier equation operational
• Can overcome the problem by using subject matter knowledge or results from a previous study
• Considers p*=0.5, given the focus of rare probabilities

Margin of Error Relative to p

• Enforces the equation ER ≤ 1 to ensure the margin of error is not larger than the order of magnitude of p*
• The er values are considered close to the bound of 1 results, intervals become greatly larger
• Decreased sample size, high precision not required
• Suggests ER ∈ [0.1,0.5] as reasonable, where interval is not huge, not requesting major sample sizes

Wald-Based Sample Size Comparison: Fixed e

• Displays calculated Wald sample sizes and coverage probabilities

Suitability of Er Scheme

• Checks the validity of using approximate CI estimators
• Examines the existing scheme to assess its compatibility with the qualification of np* > a and n(1 − p*) > a, where a ∈ {5,10}
• Suggests that the relative margin of error scheme, ER ∈ [0.1,0.5], lies below the threshold

Tolerances for Assessing CI Performance

• Suggests tolerances for assessing interval performance in terms of coverage probability and relative margin of error
• Suggests that e* ∈ {1,2,3} tolerates most analyses, as acceptable
• Tables presented provide results for the tolerance of α , and CPr values, for Wald values and for Er values.

Relative Margin of Error Central to Performance

• Assesses whether achieving a desired coverage probability while simultaneously satisfying a minimum margin results in consistent confidence intervals
• Notes cases where coverage is unsatisfactory but Wald and Wilson give similar output

CI Performance Tables

• Provides a 95% CI comparison for p* = p = 10–1
• Table cells color coded via tolerances
• Shows all requirements are not satisfied simultaneously
• Increased n results in Er ≤ 0.5 which is satisfied

Estimating a Rare Event with a Small Sample Size

• Highlights large sample sizes that have to accurately estimate a rare event