Machine Learning System Design

Machine Learning System Design

• Includes error analysis, satisficing and optimizing metrics, and how to partition and iterate on train, dev, and test sets.

Error Analysis

• Manually examine misclassified dev set examples to understand error types.
• Evaluate error analysis by categorizing errors and calculating the percentage of total errors for each category to identify areas for improvement.
• Cleaning incorrectly labeled data should be applied consistently to dev and test sets, especially if it significantly impacts performance.

Build System Quickly and Iterate

• Quickly set up dev/test set and metric, then build an initial system quickly.
• Prioritize next steps using Bias/Variance analysis & Error analysis.

Bias/Variance Analysis

• Training error and dev error are compared to determine if avoidable bias or variance is high, guiding improvement strategies.
• Human-level error compared to training and dev error helps determine focus.
• Error analysis helps decide if fixing dog or blurry image misclassifications would be more productive.
• Bias/Variance analysis determines whether bias reduction might be more promising than variance reduction, as compared to human level performance.

Satisficing and Optimizing Metrics

• Combines multiple evaluation metrics, such as accuracy and running time, into a single evaluation process.
• Accuracy (to maximize) and running time (≤ 100ms) are used to build an overall metric.

Optimizing and Satisficing Metrics

• Accuracy is chosen as the optimizing metric, striving for the best possible accuracy
• Running Time is the satisficing metric and just needs to be "good enough," it just needs to be ≤ 100ms.
• Choose the one which maximizes accuracy, subject to running time ≤ 100ms.

Definition

• A reasonable overall cost balances metrics.
• $Cost = Accuracy - 0.5 * RunningTime$
• Combines them into a single overall metric
• By defining how to trade off one vs. the other.

Train/Dev/Test Distributions

• Choose a dev set and test set to reflect what type of examples you expect to get in the future.
• Dev and test sets should be from the same distribution, or as close as feasible.

Size of Dev and Test Sets

• Set your test set to be big enough to give high confidence in the overall performance of your system.
• Set your dev set to be big enough to detect meaningful differences between the algorithms you are trying.
• If you have 10,000,000 examples, 10,000 examples in dev and test set is sufficient.

When to Change Dev/Test Sets and Metrics

• Algorithm is performing poorly on black cats that have a bias in the data set.
• Change the dev/test set and evaluation metric to improve performance.
• If doing well on metric + dev/test set doesn't correspond to doing what you actually need, change your metric and/or dev/test set.
• Even if Algorithm A has lower error, it is actually worse for your user satisfaction, then change your evaluation metric/dev set to better capture this.

Changing the Evaluation Metric

• Give greater weight to misclassified "edgy" photos
• Example forumla: $Error = \frac{1}{\sum w^{(i)}} \sum_{i=1}^{m}w^{(i)}I{y_{pred}^{(i)} \ne y^{(i)}}$

Algorithmic Game Theory (AGT)

• AGT combines game theory with algorithm design, aiming to create algorithms for strategic agents and understand the price of anarchy.
• Game Theory is the study of strategic interaction among rational, self-interested agents.
• Algorithm Design is the design of efficient algorithms in various computational settings.

Selfish Routing

• Model includes a network of nodes and edges with latency functions on each edge, where users choose paths to travel between source and target.
• Understand how Total flow, Cost and Social Cost works in Path analysis

Wardrop Equilibrium

• A flow in Wardrop equilibrium ensures no user can improve travel time by unilaterally changing their route, meaning all flow travels on the shortest paths.

Price of Anarchy

• The Price of Anarchy (PoA) measures the loss of social welfare due to selfish behavior.
• $$PoA = \frac{\text{Cost of Nash Equilibrium}}{\text{Cost of Optimal Solution}}$$

Braess's Paradox

• Adding a new road to a network can increase overall travel time due to new, worse Nash equilibrium.
• Without edge $(B, C)$, the cost is $1.5$ and the social cost is $1.5$ and an equal flow
• However, with the edge $(B, C)$, the Wardrop equilibrium social cost rises to $2$.

Bounding the Price of Anarchy

• With linear latency functions, $l_e(x) = a_e x + b_e$, the Price of Anarchy is at most $4/3$.
• For polynomial latency functions of degree $d$, the Price of Anarchy is $\Theta(d / \log d)$.

Algorithmic Trading and Order Book Events

• Electronic trading is dominant making up 60% of US equities and 80% of FX trading.
• Algorithmic trading (AT) is the use of computer programs to automate trading decisions and is not synonymous with High-frequency trading (HFT).
• HFT a specialized subset of AT characterized by extremely short time horizons, high message rates, high order cancellation rates, and end-of-day flat inventory

Order Book

• Orders to buy or sell assets are collected in an "order book" organized by price and time of arrival.
• Market buy orders are matched against the lowest-price asks, while market sell orders are matched against the highest-price bids.
• Limit orders are added to the order book at a specific price unless immediately matched.

Order Book Events

• Limit Order Arrival - A new limit order is placed in the order book
• Market Order Arrival - An existing limit order is executed by a market order
• Limit Order Cancellation - An existing limit order is cancelled

Some Uses of Order Book Data

• Market Surveillance to detect and prevent illegal trading activity, such as "quote stuffing".
• Algorithmic Trading is used to respond optimally to order book events to provide or take liquidity.
• Market Making quotes both bid and offer prices in order book to earn "spread" but must manage risk of being "picked off" by informed traders.
• Other uses include, optimal execution, order book asymmetry evaluation, market survelliance

Research Questions

• Research questions include the informativeness of events, modeling arrival patterns, effects on market quality and use in improving algorithms.

Cardiovascular System

• The cardiovascular system has arteries, capillaries, and veins is a system of components.

Arteries

• Carry blood away from the heart.
• Arteries branch to Arterioles then to Capillaries

Capillaries

• Microscopic vessels site of exchange with tissues.
• Composed of thin walls made of a single layer of endothelial cells

Veins

• Carry blood back to the heart.
• Formed by Capillaries joining to Venules which feed into Veins

Blood Vessel Structure

• 3 tunics exist in Blood vessels

Tunica intima

• Endothelium innermost layer
• Made of simple squamous epithelium

Turnica Media

• Middle layer
• Made of smooth muscle that controls vasoconstriction and vasodilation

Tunica Externa

• Outermost layer
• Made of collagen fibers for protection/ support

Structural Differences

• Arteries have thicker tunica media layer than veins
• Veins have valves to prevent backflow of blood

Capillary Exchange

• Achieved by diffusion, filtration, and osmosis.
• Diffusion- Movement of molecules from high to low concentration, exchanging O2, CO2, glucose, and hormones
• Filtration- Fluid moves out of the Blood stream, is driven by blood pressure
• Osmosis - Fluid moves into capillary, driven by osmotic pressure

Blood Pressure

• Blood pressure: Force of blood against blood vessel walls, measured as Systolic/Diastolic pressure in mm Hg.
• Systolic: ventricles contract
• Diastolic: ventricles relax

Factors affecting BP

• Factors affecting BP include cardiac output, blood volume, and peripheral resistance.
• $\uparrow$ CO $\rightarrow$ $\uparrow$ BP
• $\uparrow$ BV $\rightarrow$ $\uparrow$ BP
• Peripheral resistance- Affects BP via vasoconstriction and blood viscosity: $\uparrow$ PR $\rightarrow$ $\uparrow$ BP

Control of Blood Pressure

• The nervous system and hormones work together.

Nervous System

• Baroreceptors detect changes in BP and are located in aorta and carotid arteries.
• Cardiovascular center in medulla oblongata controls heart rate and blood vessel diameter. Sympathetic and parasympathetic nervous systems regulate BP: Sympathetic nervous system: $\uparrow$ BP while Parasympathetic nervous system: $\downarrow$ BP

Hormones

• Control BP though the blood
• Epinephrine/ Norepinephrine: $\uparrow$ BP
• Antidiuretic hormone (ADH): $\uparrow$ BP
• Atrial natriuretic peptide (ANP): $\downarrow$ BP
• The Renin-angiotensin-aldosterone system (RAAS): $\uparrow$ BP

Cardiovascular Disease

• CVD is impacted by atherosclerosis, hypertension, stroke and myocardial infarction.

Atherosclerosis

• Accumulation of plaque in arteries
• Accumulation leads to:
• $\downarrow$ Blood flow
• $\uparrow$ Blood pressure
• Clot formation

Hypertension

• High blood pressure is classified as 140/90 mm Hg or higher
• Risk factors:
• Genetics, age, race, obesity, stress, smoking

Stroke

• If blood supply to brain is interrupted this can lead to brain damage.

Myocardial Infarction

• If blood supply to heart is blocked, this leads to heart damage.

Risk Factors (CVD)

• Genetics, age, and race are uncontrollable risk factors
• Smoking, obesity, high cholesterol, hypertension, diabetes and stress are controllable

CVD Prevention

• CVD can be prevented through a healthy diet, exercising regularly, maintaining a healthy weight, not smoking, managing stress, and controlling blood pressure and cholesterol.

Algorithmic Complexity

• Algorithmic complexity is a measure of how much time or space is needed for an algorithm to complete.

Time Complexity

• The time complexity of an algorithm is a measure of the amount of time it takes for an algorithm to complete.

Space Complexity

• The space Complexity of an algorithm is a measure of the amount of memory is takes for an algorithm to complete.

Big O Notation

• Used to describe the performance or complexity of an algorithm, describing the worst-case scenario
• O(1) takes a constant time
• O(log n) takes a logarithmic time
• O(n) takes a linear time
• O(n log n) takes a super-linear time
• O(n^2) takes a quadratic time
• O(2^n) takes an exponential time
• O(n!) takes a factorial time

Common Data Structure Operations

• Arrays are quick for Access, but slow for Search, Insertion and Deletion
• LinkedLists are quick for Insertion and Deletion, but slow for Search and Access
• Hash Tables, are quick for Insertion, Search and Deletion

Common Algorithm Time Complexities

• Binary search is quicker and better than Liner Search
• The sort time can really depend on the volume of data and technique used.

Lecture 19: Fri. March 1

• Focused on Center of Mass and Probability analysis.

Center of Mass

• Describes position of the center of mass in reference to x and y axis.

Example 1

• Details finding center of mass

Probability

• Contains formuals need to calculate such as $\mu =$ mean $= \int_{-\infty}^{\infty} x f(x) ,dx$
• Contains description in reference to Calculus.

Fonction logarithme népérien

• The Logarithme népérien function: definition and utilization with numbers

Définition

• The fonction logarithme népérien, notée $ln$, est définie sur $]0; +\infty[$ et est la primitive de la fonction $x \longmapsto \frac{1}{x}$ qui s'annule en $1$.

Propriétés algébriques

• For all strictly positive numbers $a$ and $b$ and for any relative integer $n$, the function adheres to the following properties:
• $ln(ab) = ln(a) + ln(b)$
• $ln(\frac{1}{a}) = -ln(a)$
• $ln(\frac{a}{b}) = ln(a) - ln(b)$
• $ln(a^n) = n \cdot ln(a)$
• $ln(1) = 0$
• $ln(e) = 1$

Étude de la fonction

• Provides limits, the derivation table
• Graphs
• How the Logarithme népérien function actually works.

Applications

• Dives in to the application of how they work.

Algorithmic Trading

• Automated trading executes trades based on pre-defined instructions.

• Algorithmic Trading Users

• Institutional Investors: pension funds, mutual funds

• Investment Banks

• Hedge Funds

• Retail Investors

List of Algo Trading Pros

• Trades executed at the best possible price
• Order placement is instant and accurate
• Reduced transaction costs
• Simultaneous order execution
• Backtesting capabilities

List of Algo Trading Cons

• Need programming knowledge
• Need knowledge about financial markets
• Requires constant monitoring
• Can lead to "flash crashes"
• Subject to system errors

Types of Trading Strategies Employed

• Exploiting price differences in different markets
• Capitalize on predicted trades due to index fund rebalancing
• Implement complex mathematical models
• Utilize moving averages and channel breakouts

Bernoulli's Principle

• Increase in the speed of a fluid occurs simultaneously with a decrease in pressure or a decrease in the fluid's potential energy P + 1/2 \rho V^2 + \rho gh = constant
• Bernoulli's principle can be applied to various aspects of fluid dynamics, such as:
• Lift Generation in Aircraft Wings/ Airfoil
• Venturi Effect - The Venturi effect is a phenomenon where the pressure of a fluid decreases as it flows through a constricted section of a pipe or tube with a narrow section. This effect is a direct application of Bernoulli's principle
• Atomizers and Sprayers utilize Bernoulli's principle to disperse liquids into fine droplets.
• Curveballs in Sports such as baseball, players can impart spin to the ball to make it curve in flight, in regards to magnus effect and Bernoulli's principle

Complex Numbers

• A complex number is a number of the form $a + bi$, where
• $a$ and $b$ are real numbers
• $i$ is the imaginary unit, defined as $i = \sqrt{-1}$

Components

• Parts that are involed in a comples number

Real Part

• The a = Re(z) Part

Imaginary Part

• The b =Im(z) part

Basic math operations on complex numbers.

• Addition, subtraction, multiplication and division

Complex Conjugate

• It's definition and properties
• Definition- If $z = a + bi$, then the complex conjugate of $z$ is denoted as $\overline{z}$ and is defined by $\overline{z} = a - bi$.