H&H module 8:2

Convex Function Definition

• A function $f$ is convex on an interval $I$ if for all $x, y \in I$ and $t \in [0,1]$, $f(tx + (1-t)y) \leq tf(x) + (1-t)f(y)$.
• A function $f$ is concave on an interval $I$ if for all $x, y \in I$ and $t \in [0,1]$, $f(tx + (1-t)y) \geq tf(x) + (1-t)f(y)$.
• Geometrically, a convex function's secant (chord) lies above its graph.

Characterization of Differentiable Convex Functions

• A differentiable function $f$ is convex if and only if its derivative $f'$ is increasing.
• A differentiable function $f$ is concave if and only if its derivative $f'$ is decreasing.
• Geometrically, a convex function's tangents lie "below" its graph.

Characterization of Twice-Differentiable Convex Functions

• A twice-differentiable function $f$ is convex if and only if its second derivative $f'' \geq 0$.
• A twice-differentiable function $f$ is concave if and only if its second derivative $f'' \leq 0$.

Fundamental Examples of Convex/Concave Functions

• $x \mapsto e^x$ is convex on $\mathbb{R}$.
• $x \mapsto \ln(x)$ is concave on $\mathbb{R}_{+}^{*}$.
• $x \mapsto x^n$ is convex on $\mathbb{R}_{+}$ for $n \geq 1$.
• $x \mapsto |x|$ is convex on $\mathbb{R}$.

Jensen's Inequality

• For a convex function $f: I \rightarrow \mathbb{R}$ and $x_1, \ldots, x_n \in I$ and $\lambda_1, \ldots, \lambda_n \in [0,1]$ with $\sum_{i=1}^{n} \lambda_i = 1$:

  $f\left(\sum_{i=1}^{n} \lambda_{i} x_{i}\right) \leq \sum_{i=1}^{n} \lambda_{i} f\left(x_{i}\right)$

• If $\lambda_1 = \cdots = \lambda_n = \frac{1}{n}$, then $f\left(\frac{1}{n} \sum_{i=1}^{n} x_{i}\right) \leq \frac{1}{n} \sum_{i=1}^{n} f\left(x_{i}\right)$.

Algorithmic Game Theory - Lecture 14: Mechanism Design

• Mechanism Design without Money aims to implement a social choice function $f: \Theta \rightarrow O$.
• Revelation Principle states that if there exists a mechanism that implements $f$ in dominant strategies, then there exists a truthful mechanism that implements $f$ in dominant strategies.
• A truthful mechanism is when players maximize their utility by reporting their true type.

Implementation in Dominant Strategies

• A social choice function $f$ is implementable in dominant strategies if there exists a mechanism that implements $f$ in dominant strategies.

Gibbard-Satterthwaite Theorem

• If $O$ (outcomes) contains at least three distinct outcomes and $f: \Theta \rightarrow O$ is implementable in dominant strategies, then $f$ is dictatorial.

Proof Overview (Gibbard-Satterthwaite)

• Assume $f$ is onto (otherwise, we can restrict the range of $f$).
• $M = \prod_{i \in N} \Theta_{i}$ is the message space, $g: M \rightarrow O$ is the outcome function. By the revelation principle, we can assume that the mechanism is truthful.
• Define $f_i(\theta_{-i}) \in \underset{o \in O}{\operatorname{argmax}} u_{i}\left(o, \theta_{i}\right)$ for some $\theta_i$
• $f_i(\theta_{-i})$ is claimed to be independent of $\theta_{-i}$, else, the mechanism is dictatorial.

Thermodyanmics - Energy

• Energy cannot be created or destroyed (conservation of energy).

Types of Energy

• Kinetic Energy (KE): Energy of motion; given by $KE = \frac{1}{2}mv^2$.
• Potential Energy (PE): Stored energy; gravitational PE is $PE = mgh$.
• Internal Energy (U): Energy of molecules within a system (includes KE and PE).

Energy Transfer

• Work (W): Energy transfer when a force causes displacement; $W = F \cdot d$ or $W = \int{PdV}$ (area under P-V curve).
• Heat (Q): Energy transfer due to temperature difference; $Q = mc\Delta{T}$ for heating/cooling, $Q = mL$ for phase changes.

First Law of Thermodynamics

• $\Delta{U} = Q - W$: Change in internal energy equals heat added minus work done by the system.

Thermodynamic Processes

• Isobaric: Constant pressure; $W = P\Delta{V}$, $\Delta{U} = Q - P\Delta{V}$.
• Isochoric: Constant volume; $W = 0$, $\Delta{U} = Q$.
• Isothermal: Constant temperature; $\Delta{U} = 0$, $Q = W$, $W = nRT \ln{\frac{V_2}{V_1}}$.
• Adiabatic: No heat transfer; $\Delta{U} = -W$, $PV^{\gamma} = \text{constant}$, $W = \frac{P_2V_2 - P_1V_1}{1 - \gamma}$.

Heat Engines

• Convert heat into work through a cyclic process.
• Thermal Efficiency (e): $e = \frac{W_{net}}{Q_H} = 1 - \frac{Q_C}{Q_H}$ (where $W_{net}$ is net work, $Q_H$ is heat from hot reservoir, $Q_C$ is heat to cold reservoir).
• Carnot Engine: Theoretical maximum efficiency; $e_{carnot} = 1 - \frac{T_C}{T_H}$.

Refrigerators and Heat Pumps

• Use work to transfer heat from cold to hot.
• Coefficient of Performance (COP): Refrigerator $COP = \frac{Q_C}{W}$, Heat Pump $COP = \frac{Q_H}{W}$.
• Carnot Refrigerator/Heat Pump: Maximum COP; Refrigerator $COP_{carnot} = \frac{T_C}{T_H - T_C}$, Heat Pump $COP_{carnot} = \frac{T_H}{T_H - T_C}$.

Second Law of Thermodynamics

• Entropy (S): Measure of disorder; $\Delta{S} = \frac{Q}{T}$ for reversible processes.
• Statements: Heat flows spontaneously from hot to cold, entropy in a closed system increases or remains constant, $\Delta{S} \geq 0$.
• Implications: Impossible to create a perfect engine (100% efficiency), some energy is always lost as heat; tends towards greater disorder
• This implies it is impossible to create perfect engine and there tends to greater disorder.

Statistiques Inférence - Point Estimation Definition

• $X_1,...,X_n$ is an i.i.d. sample from $P_{\theta}$, where $\theta \in \Theta \subset \mathbb{R}^k$ is an unknown parameter.
• An estimator of $\theta$ is $T(X_1,...,X_n)$ which takes values in $\Theta$. Denoted as $\hat{\theta} = T(X_1,..., X_n)$.

Bias

• Bias of $\hat{\theta}$ is defined as $bias(\hat{\theta}) = \mathbb{E}[\hat{\theta}] - \theta$.
• $\hat{\theta}$ is unbiased if $bias(\hat{\theta}) = 0$, i.e., $\mathbb{E}[\hat{\theta}] = \theta$.

Mean Squared Error

• Defined as $MSE(\hat{\theta}) = \mathbb{E}[(\hat{\theta} - \theta)^2]$.
• Decomposition: $MSE(\hat{\theta}) = Var(\hat{\theta}) + bias(\hat{\theta})^2$.

Uniformly Minimum Variance Unbiased Estimator (UMVU)

• $\hat{\theta}$ is UMVU if:
  • $\hat{\theta}$ is unbiased.
• $Var(\hat{\theta}) \leq Var(\tilde{\theta})$ for all other unbiased estimator $\tilde{\theta}$ of $\theta$.

Fisher Information

• Quantifies the amount of information a RV $X$ contains about parameter $\theta$.
• $I(\theta) = \mathbb{E}\left[\left(\frac{\partial}{\partial \theta} \log f(X; \theta)\right)^2\right] = - \mathbb{E}\left[\frac{\partial^2}{\partial \theta^2} \log f(X; \theta)\right]$.
• For an i.i.d. sample, $I_n(\theta) = nI(\theta)$.

Cramér-Rao Bound

• Establishes a lower bound on the variance of any unbiased estimator of $\theta$: $Var(\hat{\theta}) \geq \frac{1}{I_n(\theta)}$.
• An estimator is efficient if it attains this bound.

Convergence

• $\hat{\theta}n \xrightarrow{P} \theta$ (convergence in probability) if $\lim{n \to \infty} P(|\hat{\theta}_n - \theta| > \epsilon) = 0$ for all $\epsilon > 0$.
• $\hat{\theta}n \xrightarrow{L^2} \theta$ (convergence in mean square) if $\lim{n \to \infty} \mathbb{E}[(\hat{\theta}_n - \theta)^2] = 0$.
• $\hat{\theta}n \xrightarrow{as} \theta$ (almost sure convergence) if $P(\lim{n \to \infty} \hat{\theta}_n = \theta) = 1$.
• $\hat{\theta}n \xrightarrow{d} \theta$ (convergence in distribution) if $\lim{n \to \infty} F_{\hat{\theta}n}(x) = F{\theta}(x)$.

Consistency

• An estimator $\hat{\theta}_n$ is consistent if it converges in probability to $\theta$: $\hat{\theta}_n \xrightarrow{P} \theta$.

Maximum Likelihood Estimator (MLE)

• The MLE of $\theta$ is the value that maximizes the likelihood function: $\hat{\theta}{MLE} = \arg \max{\theta \in \Theta} L(\theta; X_1,..., X_n)$.
• $L(\theta; X_1,..., X_n) = \prod_{i=1}^n f(X_i; \theta)$ is the likelihood function. Under regularity Conditions, MLE is consistent and asymptotically normal.
• $\sqrt{n}(\hat{\theta}_{MLE} - \theta) \xrightarrow{d} \mathcal{N}(0, I(\theta)^{-1})$.

Understanding The Mathematics of Deep Learning. Linear Algbra: Eigenvalue and Eigenvectors

• For $A$ in $R^{n \times n}$, if a vector $v$ in $R^n \ where\ v \neq 0$ and a $\lambda$ in $R $ exists, such that the following holds, then they are called the eigenvalue and eigenvector correspondingly for the corresponding matrix:

  $Av = \lambda v$

• Eigenvectors are defined for Sqaure matrices only

• If $v$ is an eigenvector, for any $c \neq 0$, then $cv$ is then too an eigenvector

• For a given eigenvalue $\lambda$, there are infinitely many eigenvectors

• Distinct eigenvectors can correspond to the one eigenvalue

Eigendecomposition

• A matrix $A \in \mathbb{R}^{n \times n}$ is diagonalizable if it is similar to a diagonal matrix, i.e., there exists an invertible matrix $P$ such that $P^{-1}AP$ is a diagonal matrix.
• Theorem: For $A \in \mathbb{R}^{n \times n}$, $A$ is diagonalizable if and only if $A$ has $n$ linearly independent eigenvectors.
• If $A \in \mathbb{R}^{n \times n}$ has $n$ distinct eigenvalues, then $A$ has $n$ linearly independent eigenvectors. This implies it is diagnalizable

MKTG203: Basics - Key Definitions

• Marketing Research: Defining a problem/opportunity, collecting & analyzing data, recommending actions.
• Exploratory Research: Preliminary info to define problems and suggest hypotheses.
• Descriptive Research: Better describe marketing problems.
• Casual Research: Test hypotheses about cause/effect.
• Secondary Data: Data recorded before the project. Example: Financial Statements
• Primary data: Data newly collected for the project through things like mechanical methods, polls, and more.
• Focus Group: Discussion with a group about a product before or after launch.

Advanced Control Systems - Stability Analysis

Definition of Stability

• Bounded-Input Bounded-Output stable (BIBO): Resulting in bounded output, from bounded input
• Asymptotic Stability: In the absence of any input, the output returns to equilibrium regardless of initial conditions.
• Marginal Stability: In the abscence of any input, the output remains bounded, but does not return to equilibrium.
• Interal Stability implies Input-Output Stability

Types of Analysis

Routh Hurtwiz Criterion

• A means of determining the number of roots of the characteristic equation without solving for these roots
• Procedure: By constructing from from the coefficients of the chracteristic polynomial
• Stability Condition: All elements in the first column of the Routh array must have the same sign(Postive).

Bode Plots

• Using the magnitude and phase plots for assessing stability
• Gain Margin (GM): Amount of gain, to make loop gain unity at phase.
• Phase Margin (PM): The amount of phase lag, to make the phase angel -180 at the frequency and Gain crossover freqency
• **Stability Condition: For both, GM and PM must be postive

Definition of Matrix Multiplication

• For matrices A with dimensions m,n and B with dimensions n,p, the product is of dimesions m,p. Where each member being defined by the following:

$$ c_{ik} = \sum_{j=1}^{n} a_{ij} b_{jk} \text{ where: } i = 1, \dots, m \text{ und } k = 1, \dots, p $$

properties

• Assoiciativity
• Distributivity
• non commutative