Optimization Problems with Derivatives

Applications of the Derivative: Optimization

• Optimization involves finding the minimum or maximum value of a function.
• For example, minimizing the perimeter of a rectangular area with a fixed size.
• To solve optimization problems, express the function in terms of a single variable using given constraints.
• Find critical points by setting the derivative equal to zero.
• Use the second derivative test to confirm whether the critical point is a minimum or maximum.

Optimization - Rectangular Area Example

• Objective: Minimize perimeter P = 2x + 2y of a rectangle with area xy = 2000 m².
• Expressed y in terms of x: y = 2000/x.
• Substitute into the perimeter equation: P(x) = 2x + 4000/x.
• Found the derivative: P'(x) = 2 - 4000/x².
• Set P'(x) = 0 and solved for x: x = √2000 = 10√20.
• Minimum was verified using the second derivative: P''(x) = 8000/x³ > 0.
• Solved for y: y = 10√20.
• Calculated the minimum perimeter: P = 40√20.

Optimization - Point on a Line Example

• Problem: Find the point on the line y = 4x + 7 closest to the origin (0,0).
• Minimized the distance squared: f(x, y) = x² + y².
• Substitute y = 4x + 7: f(x) = 17x² + 56x + 49.
• The derivative was found: f'(x) = 34x + 56.
• Set f'(x) = 0 and solved for x: x = -56/34 = -28/17.
• The minimum was verified using the second derivative: f''(x) = 34 > 0.
• Solved for y: y = 7/17.
• Closest point to the origin: (-28/17, 7/17).

Applications of the Derivative: Related Rates

• Related Rates problems involve finding the rate of change of one quantity in terms of the rate of change of another.
• These problems often rely on implicit differentiation.

Related Rates - Melting Snowball Example

• Problem: A snowball melts, its radius decreases at 1 cm/min. How fast is the volume decreasing when r = 5 cm?
• Volume of a sphere: V = (4/3)πr³.
• Differentiated with respect to time t: dV/ dt = 4πr² (dr/ dt).
• Specified dr/ dt = -1 cm/min, r = 5 cm.
• Solution: dV/ dt = -100π cm³/min.

Propositional Logic Definition

• Propositional logic is a formal system that deals with propositions and their relationships.
• A proposition is a statement that can be either true or false, but not both.

Types of Propositions

• Simple propositions express a single thought.
• Compound propositions are formed by connecting simple propositions.

Logical Connectives

• Conjunction (and): $P \land Q$ means P and Q.
• disjunction (or): $P \lor Q$ means P or Q.
• Negation (not): $\neg P$ means not P.
• Conditional (if... then): $P \rightarrow Q$ means if P then Q.
• Biconditional (if and only if): $P \leftrightarrow Q$ means P if and only if Q.

Truth Tables

Tautology, Contradiction, and Contingency

• A tautology is always true.
• A contradiction is always false.
• A contingency can be either true or false.

Logical Equivalence

• Two propositions are logically equivalent if they have the same truth table.
• Example: $P \rightarrow Q \equiv \neg P \lor Q$

De Morgan's Laws

• $\neg (P \land Q) \equiv \neg P \lor \neg Q$
• $\neg (P \lor Q) \equiv \neg P \land \neg Q$

Logical Implication

• P implies Q if, whenever P is true, Q is also true.

Validity of Arguments

• An argument is valid if the conclusion is a logical consequence of the premises.

Negation of Compound Propositions

• $\neg (P \land Q) \equiv \neg P \lor \neg Q$
• $\neg (P \lor Q) \equiv \neg P \land \neg Q$
• $\neg (P \rightarrow Q) \equiv P \land \neg Q$

Introduction to Optimization

• The problem of optimization is defined as minimize f(x)
• $f: \mathbb{R}^n \rightarrow \mathbb{R}$ is the objective function.

Types of Optimization

• Without constraints: $x \in \mathbb{R}^n$
• With constraints: $x \in \Omega \subset \mathbb{R}^n$

Types of Optima

• Global: $x^* \in \mathbb{R}^n \mid f(x^*) \leq f(x) \quad \forall x \in \mathbb{R}^n$
• Local: $x^* \in \mathbb{R}^n \mid \exists \epsilon > 0 \quad f(x^) \leq f(x) \quad \forall x \in \mathbb{R}^n, |x - x^| < \epsilon$

Optimality Conditions

• Without constraints:
  • If $f \in C^1$ and $x^$ is a local minimum, then $\nabla f(x^) = 0$
  • If $f \in C^2$ and $x^$ is a local minimum, then $\nabla f(x^) = 0$ and $\nabla^2 f(x^*)$ is positive semidefinite.
  • If $f \in C^2$, and $\nabla f(x^) = 0$ and $\nabla^2 f(x^)$ is positive definite, then $x^*$ is a strict local minimum.

Optimization with Constraints

• Minimize f(x) subject to $h_i(x) = 0, \quad i = 1, \dots, m$ and $g_j(x) \leq 0, \quad j = 1, \dots, p$.
• $f, h_i, g_j \in C^1$
• Lagrangian function:
  • $\mathcal{L}(x, \lambda, \mu) = f(x) + \sum_{i=1}^m \lambda_i h_i(x) + \sum_{j=1}^p \mu_j g_j(x)$

Karush-Kuhn-Tucker (KKT) Conditions

• If $x^$ is a local minimum, then exist $\lambda^ \in \mathbb{R}^m$ and $\mu^* \in \mathbb{R}^p$ such that:
  • $\nabla_x \mathcal{L}(x^, \lambda^, \mu^*) = 0$
  • $h_i(x^*) = 0, \quad i = 1, \dots, m$
  • $g_j(x^*) \leq 0, \quad j = 1, \dots, p$
  • $\mu_j^* \geq 0, \quad j = 1, \dots, p$
  • $\mu_j^* g_j(x^*) = 0, \quad j = 1, \dots, p$ (Complementary Slackness Condition)

Numerical Methods for Optimization

• Gradient Descent: $x_{k+1} = x_k - \alpha_k \nabla f(x_k)$
• Newton's Method: $x_{k+1} = x_k - [\nabla^2 f(x_k)]^{-1} \nabla f(x_k)$
• Quasi-Newton Methods: BFGS, DFP

Methods for Constrained Optimization

• Sequential Quadratic Programming (SQP)
• Penalty Methods
• Barrier Methods

Convergence Analysis

• Global Convergence: The algorithm converges to a global minimum from any starting point.
• Local Convergence: The algorithm converges to a local minimum if the starting point is close enough.
• Convergence rate:
  • Linear
  • Superlinear
  • Quadratic

Additional Optimization Notes

• If f is convex, then every local minimum is a global minimum.
• Duality includes: primal problem, dual problem, complementary slackness
• Robustness is the ability to maintain performance despite errors.

What is Machine Learning?

• Enables computers to adapt behavior through experience, improving performance P at a task T based on experience E.
• Definition based on Tom Mitchell's work.

Machine Learning Chess Example

• Task T: Playing chess
• Performance measure P: % of games won against opponents
• Training experience E: Playing practice games against itself

Other Machine Learning Examples

• Detecting if emails a spam -> # of emails correctly classified
• Diagnosing patients -> # of patients correctly diagnosed
• Recommending movies -> Difference between your rating and the system's prediction |

Machine Learning Types

• Supervised learning
• Unsupervised learning
• Semi-supervised learning
• Reinforcement learning

Supervised Learning

• Uses labeled data for training.
• Includes classification (e.g., spam detection) and regression (e.g., stock price prediction).
• Involves training a model f on input-output pairs (x, y) to approximate y = f(x).

Unsupervised Learning

• Training data has no labels, with algorithms finding patterns in the data.
• Ex: Clustering (grouping customers).
• Ex: Dimensionality reduction (reducing the number of features in a dataset).

Semi-Supervised Learning

• Data contains a few labels.
• Images are classified.
• Algorithm trains on inputs x, where some inputs have labels y.
• Use both laballed and unlabelled data.

Reinforcement Learning

• Training based on rewards for good actions.
• Game trains the agent by maximizing the reward system.

Model Selection

• Overfitting occurs when a model learns noise and is too complex.
• Underfitting occurs when a model is too simple.
• The bias-variance tradeoff manages overfitting and underfitting.

Machine Learning Jargon

• Key terms include features, labels, training data, test data, model parameters, and hyperparameters.

Algorithmic Trading

• Automated order execution based on pre-programmed instructions, factoring in:
  • Price
  • Timing
  • Volume

Trend Following Strategies

• Utilizes moving averages and channel breakouts for trend detection.

Arbitrage Opportunities

• Exploits price discrepancies across different markets.

Market Making

• Generates profits from the bid-ask spread.

Mean Reversion

• Capitalizes on prices reverting to their average.

Statistical Arbitrage

• Employs statistical models that identify and exploit pricing inefficiencies.

TWAP (Time Weighted Average Price)

• Minimizes market impact by executing large orders over periods of time/ chunks.

VWAP (Volume Weighted Average Price)

• Average executrion price is close to the volume-weighted average price.

Implementation Shortfall

• Minimizes the difference between the actual and ideal execution price.

Advantages of Algorithmic Trading

• Speed and Efficiency: Algorithms react quickly to market changes.
• Reduced Emotional Bias: Rational trading decisions are achieved by this.
• Backtesting Capabilities: Strategies can be evaluated with historical data.
• Allows for portfolio Diversification.

Disadvantages of Algorithmic Trading

• Technical Issues such as software glitches may occur, leading to loss.
• Over-Optimization: Historical data may lead to worse performance in a live environment.
• Algo's struggle to adapt: Market Complexity or rapid dynamics change and cause issues.

Simple Moving Average Crossover Strategy

• Strategy: Buy when the short-term moving average crosses above the long-term moving average, sell when it crosses below.
• Libraries: Utilizes yfinance, pandas, and numpy.
• Implemented by code:
  • Downloads financial data, and calculates moving averages.
  • Generates trading signals.
  • Backtests the strategy, assessing portfolio performance, calculates the Sharpe ratio.
  • Visualizes with plots of the historical price data

Simple Moving Average Crossover Strategy Code Explanation

• Import Libraries: Import yfinance for data, pandas for manipulation, numpy for numerical operations.
• Download Data: Download data for Apple Inc. ('AAPL') from Yahoo Finance using yfinance.
• Calculate Moving Averages: Calculate short-term 20-day and long-term 50-day simple moving averages (SMAs) of the prices.
• Generate Signals: Buy if the short-term SMA crosses above, sell if crosses below.
• Backtesting: Trading strategy: simulate performance.
  • Calculated daily positions in AAPL.
  • Calculated portfolio value, cash balance, and total returns.
• Plotting*: Shows the strategy.
  • Plots moving averages.
  • Highlights signals.
• Evaluation: Evaluates performance with the Sharpe ratio.

What is Static Electricity

• Static Electricity: A imbalance of electric charge across a surface
• Static Electricity: The excess of either positive or negative charges

How do objects become charged?

• Charging via Friction: Electrons are transferred
• Charging via Conduction: Electrons move from a charged object to uncharged object
• Charging via Induction: Rearrangement of uncharged objects occurs

Static Electricity: Conductors

• Materials that allow electron travel such as metals
• Atoms loosely hold electrons

Static Electricity: Insulators

• Materials prevent electron travel such as plastic
• Atoms tightly hold electrons

Static Electricity: Grounding

• Connect a charged object to the Earth to neutralize charges

Static Electricity: Coulomb's Law

• $F=k \frac{q_{1} q_{2}}{d^{2}}$
  • F = Force (N)
  • $q_1, q_2$ = charge (C)
  • d = distance between objects (m)
  • $k=9.0 \times 10^{9} \mathrm{Nm}^{2} / \mathrm{C}^{2}$

Static Electricity: Electrical Fields

• Regions where charges objects exert force
• Field lines point away from positive, close to negative.

Static Electricity: Electrical Potential

• Electrical potential energy per unit charge measured in volts
• Known as voltage

Static Electricity: Potential Difference

• Difference between two points, also measured in volts
• Known as voltage

Static Electricity: Capacitance

• Capacity of energy storage, measured in Farads or F
• $C=\frac{Q}{V}$
  • C = Capacitance (F)
  • Q = Charge (C)
  • V = Voltage (V)

Static Electricity: Diagram

• Showing negative charges repelling and neutralizing.

Pythagorean Theorem: Parts

• Hypotenuse: Longest side, opposite the right angle.
• Legs: The shorter sides forming the right angle.

Pythagorean Theorem: Definition

• $a^2 + b^2 = c^2$ where a and b and the legs, and c is the hypotenuse

Pythagorean Theorem: Hypotenuse Length

• Can be found when given the legs: $c = \sqrt{a^2 + b^2}$

Pythagorean Theorem: Leg Length

• Can be found when given the hypotenuse and another leg: $b = \sqrt{c^2 - a^2}$

Pythagorean Theorem: Triples

• 3, 4 and 5
• 5, 12 and 13
• 8, 15 and 17
• 7, 24 and 25

Pythagorean Theorem: Applications

• Can be used for: height of buildins, distance, or the length of other variables

Pythagorean Theorem: Problems

• Solutions in order: 10, 8, 13, 7