Channel Capacity and Shannon's Theorem

Channel Capacity

• Channel capacity represents the maximum rate of reliable information transfer through a communication channel.

Shannon Capacity Theorem

• Defines channel capacity (C) as the maximum mutual information I(X; Y) between input (X) and output (Y) over all input distributions p(x).
• $C = \max_{p(x)} I(X; Y)$ bits per channel use
• $C = B \log_2(1 + \frac{S}{N})$ bits per second
  • B denotes bandwidth (Hz)
  • S denotes signal power
  • N denotes noise power

Channel Capacity Properties

• The channel capacity depends on bandwidth and signal-to-noise ratio (SNR).
• Higher bandwidth and SNR lead to higher channel capacity.
• Channel capacity is a theoretical upper limit, not always achievable in practice.

Channel Capacity Implications

• Channel capacity sets a fundamental limit on communication channel data transmission rates.
• Can be used for communication systems design and assessment.
• Facilitates comparison across different communication channels

Discrete Memoryless Channel (DMC)

• A DMC uses discrete input and output symbols.
• The output depends only on the current input symbol.
• Capacity formula: $C = \max_{p(x)} I(X; Y)$, where $I(X; Y)$ is the mutual information between input $X$ and output $Y$.

Mutual Information

• Mutual information $I(X; Y)$ measures the amount of information the output $Y$ provides about the input $X$.
• Defined as $I(X; Y) = H(X) - H(X|Y)$, where $H(X)$ is the entropy of $X$ and $H(X|Y)$ is the conditional entropy of $X$ given $Y$.

Entropy

• Entropy of a discrete random variable X, $H(X) = - \sum_{x \in \mathcal{X}} p(x) \log p(x)$
• Conditional entropy of X given Y, $H(X|Y) = - \sum_{x \in \mathcal{X}} \sum_{y \in \mathcal{Y}} p(x, y) \log p(x|y)$

Finding DMC Capacity

• Maximizing mutual information $I(X; Y)$ over all input distributions $p(x)$ determines DMC capacity.
• The Blahut-Arimoto algorithm is an iterative method to find the maximizing input distribution
• Karush-Kuhn-Tucker (KKT) conditions provide necessary conditions for optimality.

Binary Symmetric Channel (BSC)

• A BSC is a DMC with binary input and output alphabets: $\mathcal{X} = {0, 1}$ and $\mathcal{Y} = {0, 1}$.
• Characterized by 'p', the probability of a bit flip during transmission.

BSC Transition Probabilities

• $p(y = 0 | x = 0) = 1 - p$
• $p(y = 1 | x = 0) = p$
• $p(y = 0 | x = 1) = p$
• $p(y = 1 | x = 1) = 1 - p$

BSC Capacity

• A uniform input distribution ($p(x = 0) = p(x = 1) = 0.5$) maximizes mutual information for a BSC.
• Mutual information: $I(X; Y) = 1 - H(p)$, where $H(p)$ is the binary entropy function.
• BSC capacity: $C = 1 - H(p)$
• Noiseless channel (p=0) has C = 1 bit per channel use
• Completely noisy channel (p=0.5) has C = 0 bits per channel use

Machine Learning Definition

• Gives computers the ability to learn without being explicitly programmed.
• Improves performance at task T with experience E, measured by performance metric P

Types of Machine Learning

• Supervised Learning
• Unsupervised Learning

Supervised Learning

• Computers are taught with labeled data
  • Regression predicts a continuous value output such as predicting a house price based on size.
  • Classification predicts a discrete value output such as identifying a tumor as benign or malignant.

Unsupervised Learning

• Algorithms find structure in unlabeled data.
  • Clustering groups similar data points together such as grouping customers into market segments.
  • Non-clustering is used for anomaly detection, dimensionality reduction and association rule learning.

Example Machine Learning Algorithms

• Linear Regression
• Logistic Regression
• Neural Networks
• Support Vector Machines
• K-Means
• Principal Component Analysis
• Anomaly Detection

Linear Regression Model Representation

• Features
• Input variable $x$
• Output / target variable $y$
• Training example $(x,y)$
• $i^{th}$ training example $(x^{(i)},y^{(i)})$
• m = number of training examples

Hypothesis

• Linear Equation: $h(x)=\theta_0 + \theta_1x$
• Parameters: $\theta_i$

Cost Function

• $J(\theta_0, \theta_1) = \frac{1}{2m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)}) - y^{(i)})^2$
• Goal is to minimize $J(\theta_0, \theta_1)$ with respect to $\theta_0$ and $\theta_1$

Gradient Descent Algorithm

• Repeat until convergence {
  • $\theta_j := \theta_j - \alpha \frac{\partial}{\partial \theta_j}J(\theta_0, \theta_1) \qquad$ (for j = 0 and j = 1)
• $\alpha$ = learning rate

Gradient Descent Intuition

• $\alpha$ Too Small: slow convergence
• $\alpha$ Too Large: may overshoot, fail to converge, or diverge

Gradient Descent for Linear Regression

• Update $\theta_0$ and $\theta_1$ simultaneously.

Update Rules

• $\theta_0 := \theta_0 - \alpha \frac{1}{m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)}) - y^{(i)})$
• $\theta_1 := \theta_1 - \alpha \frac{1}{m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)}) - y^{(i)})x^{(i)}$

Exchange Rate

• Exchange rate is the price of one currency in terms of another.
  • Example is USD/JPY = 110

Exchange Rate Systems

• Floating Exchange Rate System
  • Supply and demand determine exchange rates in the forex market
• Fixed Exchange Rate System
  • Exchange rates are fixed by the government or central bank, requiring intervention to maintain the rate
• Managed Float Exchange Rate System
  • Market forces primarily determine rates, but the government/central bank intervenes occasionally

Factors Affecting Exchange Rates

• Economic Factors

  • Relative Inflation Rates: Higher inflation depreciates currency
  • Relative Interest Rates: Higher interest rates appreciate currency
  • Current Account Deficit/Surplus: Deficit depreciates and surplus appreciates currency -Economic Growth: Stronger growth appreciates currency

• Political Factors -Political Stability: Stability attracts investment, appreciates currency -Government Policies: Fiscal and monetary policies impact rates

• Market Psychology -Speculation: Speculative trading causes short-term fluctuations -Investor Confidence: Confidence influences currency demand

Exchange Rate Calculations

• Direct Quote: Value of foreign currency in terms of home currency (USD/CHF = 1.10)
• Indirect Quote: Value of home currency in terms of foreign currency (CHF/USD = 0.91)
• Cross Rate: Exchange rate between two non-official currencies (EUR/GBP)

Cross Rate Calculation Formula

• $\frac{EUR}{GBP} = \frac{USD/GBP}{USD/EUR}$
• Given USD/EUR = 1.10 and USD/GBP = 1.30, EUR/GBP = 1.30 / 1.10 = 1.1818 (1 EUR = 1.1818 GBP)

Impact of Exchange Rates

• Exports: Weaker currency makes exports cheaper
• Imports: Weaker currency makes imports more expensive
• Inflation: Weaker currency can increase inflation
• Economic Growth: Exchange rates affect export competitiveness and import costs

Obesity Definition

• Obesity is defined as a body mass index (BMI) of 30 kg/m2 or more.

Guideline Coverage

• Obesity in children (aged 2 years and over), young people and adults.

Guideline Recommendations

• Preventing obesity
• Identifying people at risk of, or with obesity
• Assessing people at risk of, or with obesity
• Managing obesity

Prevention Strategies

• Advise about the benefits of a healthy diet and physical activity.
• Encourage small, sustainable changes to diet and activity levels.
• Provide information about local support services.
• Collaborate to create environments supporting healthy eating and physical activity.

Prevention Target

• Everyone has a role.
• Focus on those at higher risk; lower socioeconomic groups and certain ethnic groups (Black, Asian).
• Target interventions at key life stages (pregnancy, early childhood).

Identification: Waist Circumference

• Measure waist circumference of adults with BMI < 35 kg/m2 to assess risk.
• Consider waist-to-hip ratio in addition to BMI and waist circumference.
• Raised waist circumference indicates increased risk of type 2 diabetes, cardiovascular disease, and mortality.

Waist Circumference Risk Levels

• High Risk:
  • Men: 94 cm or more
  • Women: 80 cm or more
• Very High Risk:
  • Men: 102 cm or more
  • Women: 88 cm or more

Assessment Guidelines

• Establish willingness to make lifestyle changes.
• Identify underlying medical conditions.
• Consider individual circumstances (lifestyle, culture, socioeconomic status).
• Discuss goals and expectations for weight management.

Management Guidelines

• Combination of diet, exercise, and behavioral strategies.
• Consider drug treatment for BMI of 30 kg/m2 or more when lifestyle interventions alone have not been successful.
• Consider bariatric surgery for BMI of 40 kg/m2 or more, or BMI of 35 kg/m2 or more with significant health conditions, when other treatments have failed.

Channel capacity is the maximum rate of reliable information transfer. Shannon's theorem defines capacity based on bandwidth and signal-to-noise ratio. It's a key concept for communication systems design.