Probability Theory and Statistics Lecture Notes

Probability Theory and Statistics Lecture Notes

• Rostyslav Hryniv, Autumn 2018
• Ukrainian Catholic University - Computer Science and Business Analytics Programs
• 3rd term

Law of Large Numbers and Central Limit Theorem

• Inequalities and Weak Law of Large Numbers

  • Motivation for the law
  • Relevant inequalities
  • Description of the weak law

• Convergence and Strong Law of Large Numbers

  • Convergence in probability
  • Convergence almost everywhere
  • Description of the strong law

• Central Limit Theorem

  • Characteristic functions
  • Convergence in law
  • Convergence of sums

Frequentist Approach to Probability

• Definition

  • Frequentist probability defines an event's probability
  • Relative frequency in a large number of trials

• Justification by Law of Large Numbers

  • Problems with definition
  • Determining what constitutes "large enough"
  • Potential variations when repeated

Sums of Independent Identically Distributed Random Variables

• Independent Identically Distributed (i.i.d.) Random Variables

  • Defining i.i.d. random variables
  • Expectations (E(X₁)) and Variations (Var(X₁))

• Sums and Sample Mean (Mn)

  • Defining the sum (Sn)
  • Variance of Sₙ
  • Sample mean definition (Mn)
  • Expectation of sample mean (E(Mn))
  • Variance of sample mean (Var(Mn))

• Law of Large Numbers (informal)

  • Describing convergence to the true mean (μ)

Markov Inequality

• Definition

  • Defining a non-negative random variable (r.v. X)
  • Defining a constant (a > 0)
  • Defining the inequality: P(X ≥ a) ≤ E(X)/a

• Proof

  • Auxiliary r.v. (Ya) definition
  • Showing that Ya ≤ X
  • Using expectation to prove the inequality

• Example

  • Application of Markov Inequality

Chebyshev's Inequality

• Definition

  • Defining a random variable (X) with mean (μ) and variance (σ²)
  • Defining a constant (c > 0)
  • Defining the inequality: P(|X – μ| ≥ c) ≤ σ²/c²

• Proof

  • Showing that {|X – μ| ≥ c} = {(X – μ)² ≥ c²}
  • Using the Markov inequality

• Example

  • Application of Chebyshev's Inequality

One-Sided Chebyshev's Inequality

• Definition

  • Defining the one-sided inequality: P(X - μ ≥ c) ≤ σ²/(σ² + c²)

• Proof

  • Using the definition of the event to derive the inequality

• Example

  • Illustrative example using application of the one-sided Chebyshev's Inequality

Derivation of the Weak Law of Large Numbers (WLLN)

• Introduction of i.i.d. random variables and sample means

  • Identifying independent random variables (X₁, X₂,...)
  • Defining the sample mean (Mn) and its expectation (E(Mn)) and variance(Var(Mn))

• Applying Chebyshev's Inequality

  • Relation between sample mean and Chebyshev's inequality
  • Expressing the probability in terms of variance, sample size and constant

• Result

  • Probability converges to 0 as sample size increases
    • Result: the process eventually converges to the true mean (μ)

Weak Law of Large Numbers (WLLN)

• Theorem

• Independent and identically distributed random variables

• Defining the mean (μ)

  • Defining probability of difference: P(|Mn - μ| ≥ ε) = 0 as n approaches infinity

• Example

  • Describing the example with normal distribution (N (μ, σ²) and sample mean

Example: Opinion Poll

• Introduction

  • Sampling from a population
  • Assessing support for a topic (T)
  • Relationship between support rate (p) and sample observations

• Estimation of Support Rate

  • Using k/n as an estimate (p)
  • defining I and indicator (Ij) the Bernoulli indicator

• Discussion of Large Sample Size

  • Determining needed sample size (n) to achieve certain probability levels and desired precision

Convergence in Probability

• Definition

  • Sequence of r.v.'s (X₁, X₂,...) converging in probability to r.v.Y
  • Using the probability expression for the convergence

• WLLN Implication

  • The sample means (Sn) of i.i.d. random variables eventually converge to the underlying mean.

Convergence Almost Everywhere

• Definition

  • Defining convergence almost everywhere (a.e).
  • Expressing almost sure convergence (Xn → X)

• Example

  • Illustrative example with uniform distribution

• Theorem

  • Implication of convergence a.e. to convergence in probability

Strong Law of Large Numbers (SLLN)

• Theorem (SLLN)

  • Independent and identically random variables convergence

• Corollary (estimating cdf)

  • Defining the indicator random (variables) Ik
  • Convergence of empirical probability distribution

Empirical c.d.f.

• Definition

  • Defining the empirical c.d.f.

• Theorem (Glivenko-Cantelli)

Monte Carlo

• SLLN application in Monte Carlo simulation

• Examples

  • Illustrative example involving continuous function and uniformly distributed r.v.'s

Characteristic Functions

• Defining characteristic function: Φx(t) = E(eitx)
• Relation between characteristic functions and probability density functions
• Uniqueness and moments of distribution related to characteristic functions

Examples

• Normal Distribution

  • Defining normal distribution and expression of characteristic function

• Exponential Distribution

Convergence in Law

• Definition

  • Define convergence in law

• Example (Poisson Approximation)

  • Illustrative example including convergence in law for binomial and Poisson

Central Limit Theorem (CLT)

• Assumptions for CLT

  • Independent and identically distributed random values
  • Definition of the sample mean
  • Sample mean converges to underlying mean

• Standardized sums (Zn)

• Result of CLT

Berry-Esseen Theorem

• Rate of convergence in CLT

Opinion Polls

• Task (Opinion Polls) -Estimating fraction(p) that supports a topic(T)

• Indicators (Ik)

• Sample Means and CLT -Convergence of estimated fractions

• Example for n -Determining necessary sample size for a specific confidence level and interval

Example: Post Office Overweight

• Context -Determining the probability that a total weight is below a given value
• Random Variables (Xn)
• Calculations -Using sum of random variables and Chebyshev's Inequality.