Bayes Estimator: Bernoulli and Beta Distribution

Lecture 24 - Nov. 15, 2023: Bayes Estimator

• Bayes Estimator minimizes the Bayesian Risk.
  • The formula for Bayes Estimator is $\hat{\theta_B}(x) = \underset{\theta}{\operatorname{argmin}} \int L(\theta, \hat{\theta}) \pi_{\theta|X=x}(\theta) d\theta$ with $\pi_{\theta|X=x}(\theta)$ as the posterior distribution.
• If the Loss Function is $L(\theta, \hat{\theta}) = (\theta - \hat{\theta})^2$ (squared error risk), the Bayes Estimator simplifies to the posterior mean.
  • $ \hat{\theta_B}(x) = E[\theta|X=x] $.

Example of Bayes Estimator with Bernoulli and Beta Distribution

• Given $X_i \overset{iid}{\sim} Bernoulli(\theta), i = 1, \dots, n$ and $\theta \sim Beta(a, b)$.
• The Bayes Estimator of $\theta$ under squared error risk can be found.
• Likelihood is given by $ p(x|\theta) = \theta^{n\bar{x}} (1 - \theta)^{n - n\bar{x}} $.
• Prior distribution is $\pi(\theta) = \frac{\theta^{a-1} (1 - \theta)^{b-1}}{B(a, b)}$.
• Posterior distribution is $\theta|x \sim Beta(n\bar{x} + a, n - n\bar{x} + b)$.
• The Bayes Estimator is then $ \hat{\theta_B}(x) = \frac{n\bar{x} + a}{n + a + b} $, which can be expressed as a weighted average of the MLE and Prior Mean.
  • With the formula as $ \hat{\theta_B}(x) = w \cdot MLE + (1 - w) \cdot Prior Mean$ where $w = \frac{n}{n + a + b}$.

Conjugate Prior Defined

• A prior distribution is a conjugate prior for the likelihood function if the posterior distribution is in the same family as the prior distribution.
• The Beta distribution serves as the conjugate prior for the Bernoulli likelihood function

Example of Bayes Estimator with Normal Distribution

• Given $X_i \overset{iid}{\sim} N(\theta, \sigma^2), i = 1, \dots, n$ and $\theta \sim N(\mu, \tau^2)$.
• The Bayes Estimator of $\theta$ under squared error risk can be found.
• Likelihood is $p(x|\theta) = (\frac{1}{\sqrt{2\pi\sigma^2}})^n e^{-\frac{\sum_{i=1}^n (x_i - \bar{x})^2}{2\sigma^2}} e^{-\frac{n(\bar{x} - \theta)^2}{2\sigma^2}}$.
• Prior distribution is $\pi(\theta) = \frac{1}{\sqrt{2\pi\tau^2}} e^{-\frac{(\theta - \mu)^2}{2\tau^2}}$.
• Posterior distribution is $\theta|x \sim N(\frac{n\tau^2 \bar{x} + \sigma^2 \mu}{n\tau^2 + \sigma^2}, \frac{\sigma^2 \tau^2}{n\tau^2 + \sigma^2})$.
  • The Bayes Estimator is $ \hat{\theta_B}(x) = \frac{n\tau^2 \bar{x} + \sigma^2 \mu}{n\tau^2 + \sigma^2} = w \cdot MLE + (1 - w) \cdot Prior Mean$.
  • With the weight as $w = \frac{n\tau^2}{n\tau^2 + \sigma^2}$.

Remarks on Bayes Estimator

• The Bayes Estimator approximates to the MLE where the prior is "weak"
• The Bayes Estimator is generally a shrinkage estimator, shrinking the MLE towards the prior mean.
• With many data points the Bayes Estimator will be close to the MLE

Asymptotic Property of Bayes Estimator

• Under regularity conditions, the posterior distribution is asymptotically normal.
  • $\theta|x \overset{\cdot}{\sim} N(\hat{\theta}_{MLE}, I(\theta)^{-1})$ where $I(\theta)$ is the Fisher Information.
• The Bayes Estimator is consistent and asymptotically efficient.

Física

Vectors

Sum of Vectors

• Represented by placing vectors sequentially, preserving magnitude, direction, and sense.
• The resultant vector connects the origin of the first to the end of the last.
• Components from each axis are summed independently, following by magnitude calculated using the Pythagorean theorem and direction using the tangent function.

Product of Vectors

• Scalar product results in a real number as $ \vec{a} \cdot \vec{b} = |\vec{a}| \cdot |\vec{b}| \cdot \cos{\theta} $.
• Vector product yields another vector with magnitude calculated $|\vec{a} \times \vec{b}| = |\vec{a}| \cdot |\vec{b}| \cdot \sin{\theta}$.

Cinemática

Uniform Rectilinear Motion (MRU)

• Velocity is constant and trajectory is a straight line.
• Defined by $ v = \frac{\Delta x}{\Delta t} $ and $ x = x_0 + v \cdot t $.

Uniformly Varied Rectilinear Motion (MRUV)

• With constant acceleration and trajectory along a straight line..
• Formulas are $ a = \frac{\Delta v}{\Delta t} $, $ v = v_0 + a \cdot t $, $ x = x_0 + v_0 \cdot t + \frac{1}{2} \cdot a \cdot t^2 $ and $ v^2 = v_0^2 + 2 \cdot a \cdot \Delta x $.

Vertical Throw and Free Fall

• Special MRUV cases with acceleration due to gravity $ g = 9,8 m/s^2 $.
• Vertical throw has initial velocity differing from zero, while free fall starts from rest.

Uniform Circular Motion (MCU)

• Trajectory is a circumference and its angular speed is constant.
• Defined by relations such as $ \omega = \frac{\Delta \theta}{\Delta t} $, $ \theta = \theta_0 + \omega \cdot t $, $ v = \omega \cdot r $, and $ a_c = \frac{v^2}{r} = \omega^2 \cdot r $.

Uniformly Varied Circular Motion (MCUV)

• With constant angular acceleration.
• Characterized by equations mirroring MRUV but for angular parameters: $\alpha = \frac{\Delta \omega}{\Delta t}$, $ \omega = \omega_0 + \alpha \cdot t $, $ \theta = \theta_0 + \omega_0 \cdot t + \frac{1}{2} \cdot \alpha \cdot t^2 $, and $ \omega^2 = \omega_0^2 + 2 \cdot \alpha \cdot \Delta \theta $.

Oblique Throw

• A combination of MRU horizontally and MRUV vertically

Trabajo y Energía

Work

• Work done by a force is the force times the displacement times the cosine of the angle between them or $ W = \vec{F} \cdot \Delta \vec{x} = |\vec{F}| \cdot |\Delta \vec{x}| \cdot \cos{\theta}$

Kinetic Energy

• Energy due to the motion describes $ E_c = \frac{1}{2} \cdot m \cdot v^2.$

Potential Energy

• Energy due to position, either gravitational $ E_p = m \cdot g \cdot h. $ or elastic $ E_p = \frac{1}{2} \cdot k \cdot \Delta x^2. $

Theorem of Work and Energy

• states that the net work equals a body's change in kinetics or $W_{neto} = \Delta E_c$

Conservation of mechanical energy

• In an isolated system where only conservative forces act, mechanical energy is constant and is described as $ E_m = E_c + E_p = constante$

Potencia

• Power measures how quickly work gets done $ P = \frac{W}{\Delta t}$

Impulso y Cantidad de Movimiento

Impulse

• Calculated by product force over interval or $ \vec{I} = \vec{F} \cdot \Delta t $ and is.

Quantity of Motion

• Is is vector measuring mass related by the velocity $ \vec{p} = m \cdot \vec{v}$.

Teorema del impulso y la cantidad de movimiento

• Connects impulse equal change momentum or $ \vec{I} = \Delta \vec{p} $

Conservación de la cantidad de movimiento

• For conservation within isolated has equation $ \vec{p}_{total} = constante $

Estática

Concurrent Forces

• Force sum acting body result zero express condition $ \sum \vec{F} = 0$.

Momento de una fuerza

• Described by vector derived product vector representing then location relative Force denoted expression. $ \vec{M} = \vec{r} \times \vec{F}$

Fuerzas paralelas

• To ensure that bodies at rests equilibrium requirements require The sum must always be Zero as well moment about Zero too exist

Centro de gravedad

• Center indicates action for forces

Dinámica

Leyes de Newton

• Law Inertia objects maintain unless caused externally
• Second determines Pro Force derived multiplication Mass over. Accerelation express relations $\sum \vec{F} = m \cdot \vec{a}$
• Third defines action and opposing. Equal reverse in bodies

Fuerzas de rozamiento

• force by frictions either standing denoted relations ,$ F_r = \mu \cdot N$

Fuerzas elásticas

• Exert elasticity force in displacement denote following equation,$ F = -k \cdot \Delta x$

Fourier Series Summary

• Representation of periodic functions with sines and cosines
• Expressed as $f(x) = a_0 + \sum_{n=1}^{\infty} [a_n \cos(nx) + b_n \sin(nx)]$
• $a_0, a_n, b_n$ are Fourier coefficients that define amplitude and phase.

Calculating Fourier Coefficients

• Formulas to calculate coefficients:
  • $a_0 = \frac{1}{2\pi} \int_{-\pi}^{\pi} f(x) dx$
  • $a_n = \frac{1}{\pi} \int_{-\pi}^{\pi} f(x) \cos(nx) dx$
  • $b_n = \frac{1}{\pi} \int_{-\pi}^{\pi} f(x) \sin(nx) dx$

Square Wave Example

• Square wave function Fourier series:
  • $f(x) = \frac{4}{\pi} \sum_{n=1, 3, 5,...}^{\infty} \frac{1}{n} \sin(nx)$

Applications of Fourier Series

• Signal processing, image processing, physics, and engineering

GeoGebra Quick Start Guide

Interface

• Menu bar for file, edit, view, options, tools, window and help
• Toolbar for quick access to tools
• Algebraic view for math objects
• Graphic view for visual
• CAS, for symbolic calculations
• Construction protocol registers all steps

Inputting

• Input bar directly adds commands using on screen keyboard

Basic edits

Main Edits

Key Commands

• Basic operations with Point, line function transform

Extra suggestions

• Personalize with resources

Algèbre linéaire (Bernard Gostiaux)

• Table of Contents
  • Avant-propos
  • Notations

1. Espaces vectoriels

• Loi de composition interne
• Groupe
• Corps
• Espace vectoriel
• Sous-espace vectoriel
• Somme de sous-espaces vectoriels
• Intersection de sous-espaces vectoriels
• Somme directe de sous-espaces vectoriels
• Sous-espaces supplémentaires
• Espace vectoriel quotient

2. Applications linéaires

• Application linéaire
• Image d'une application linéaire
• Noyau d'une application linéaire
• Théorème du rang
• Applications linéaires et formes linéaires
• Hyperplan
• Projecteurs
• Symétries
• Homographies

3. Matrices

• Matrices
• Matrices carrées
• Matrices inversibles
• Transposition des matrices
• Matrices et applications linéaires
• Changement de bases
• Matrices équivalentes
• Matrices semblables
• Trace d'une matrice carrée

4. Déterminants

• Formes n-linéaires alternées
• Déterminant d'une famille de vecteurs
• Déterminant d'un endomorphisme
• Déterminant d'une matrice carrée
• Propriétés des déterminants
• Calcul des déterminants
• Applications des déterminants

5. Réduction des endomorphismes et des matrices carrées

• Éléments propres d'un endomorphisme
• Polynôme caractéristique
• Théorème de Cayley-Hamilton
• Polynôme minimal
• Diagonalisation
• Trigonalisation
• Décomposition de Dunford

6. Espaces préhilbertiens réels

• Produit scalaire
• Inégalité de Cauchy-Schwarz
• Norme associée à un produit scalaire
• Orthogonalité
• Procédé d'orthonormalisation de Gram-Schmidt
• Projections orthogonales
• Éléments de géométrie euclidienne
• Adjoint d'un endomorphisme
• Endomorphismes autoadjoints
• Endomorphismes orthogonaux

7. Formes bilinéaires

• Formes bilinéaires
• Formes bilinéaires symétriques
• Formes bilinéaires antisymétriques
• Orthogonalité
• Matrices congruentes
• Réduction de Gauss
• Formes quadratiques
• Formes quadratiques positives

8. Espaces hermitiens

• Formes hermitiennes
• Produit scalaire hermitien
• Orthogonalité
• Adjoint d'un endomorphisme
• Endomorphismes antihermitiens
• Endomorphismes unitaires

9. Solutions des exercices

Rules of Differentiation Summarized

Constant Function Rule

• For $f(x) = k$ (k is constant), $f'(x) = 0$.

Power Rule

• If $f(x) = x^n$, then $f'(x) = nx^{n-1}$.

Constant Multiple Rule

• For $f(x) = k \cdot g(x)$ (k is constant), $f'(x) = k \cdot g'(x)$.

Sum/Difference Rule

• For $f(x) = u(x) \pm v(x)$, $f'(x) = u'(x) \pm v'(x)$.

Product Rule

• If $f(x) = u(x) \cdot v(x)$, then $f'(x) = u'(x) \cdot v(x) + u(x) \cdot v'(x)$.

Quotient Rule

• If $f(x) = \frac{u(x)}{v(x)}$, then $f'(x) = \frac{u'(x) \cdot v(x) - u(x) \cdot v'(x)}{[v(x)]^2}$.

Chain Rule

• When $f(x) = u(v(x))$, $f'(x) = u'(v(x)) \cdot v'(x)$.

Derivatives of Trigonometric Functions:

• $f(x) = \sin(x) \implies f'(x) = \cos(x)$
• $f(x) = \cos(x) \implies f'(x) = -\sin(x)$
• $f(x) = \tan(x) \implies f'(x) = \sec^2(x)$

Derivatives of Exponential and Logarithmic Functions:

• $f(x) = e^x \implies f'(x) = e^x$
• $f(x) = \ln(x) \implies f'(x) = \frac{1}{x}$

Pythagorean Theorem Overview

• Right triangle: $a^2 + b^2 = c^2$
  • a and b are the legs
  • c is the hypotenuse
• Converse of the Pythagorea: If $a^2 + b^2 = c^2$, then $\angle C$ is a right angle.
• Pythagorean triples are integer triplets that fulfills the main equations
• Pythagorean Inequalities Theorem 1: If $c^2 > a^2 + b^2$, obtuse triangle.
• Pythagorean Inequalities Theorem 2: If $c^2 < a^2 + b^2$, acute triangle.

Research Notes

• Research is a careful and detailed study into a specific problem
• Types of research

Qualitative Research

• Explores ideas and creating a hypothesis or theory
• Tries to understand a group of people Data is collected through interviews and focus groups Tends to have small sample sizes Asks the question "why?" Data is descriptive

Quantitive Research

• Tests hypothesis of theories
• Tries to measure a group of people
• Data is collected through surveys and experiments
• Tends to have large sample sizes
• Asks the question "how many?"
• Data is numerical

Static Characteristics Overview

• Static Sensitivity: Ratio of output change to input change, $K = \frac{\Delta y}{\Delta x}$ Ideally the sensitivities should be high meaning the smaller change input will result of the larger change output.
• Linearity: Maximum deviation from linear response.
• Hysteresis: Difference in output depending on direction of input change.
• Resolution: Smallest detectable input change.
• Accuracy: How well a measurement aligns with a standard value.
• Precision: Consistency/repeatability of measurements.