Reinforcement Learning and MDPs

Reinforcement Learning (RL)

• RL used for stock trading strategies by modeling trading as a Markov Decision Process (MDP)
• Intelligent agent interacts with environment, observes states, takes actions, and receives rewards
• Agent uses policy to decide actions
• Goal: learn policy maximizing expected cumulative rewards
• RL unique feedback loop; not found in supervised or unsupervised learning

Markov Decision Processes (MDPs) and Dynamic Programming

• Markov chains: fixed number of states, random transitions

• Transition probabilities depend only on current and next state, not past states (memoryless)

• MDPs: agent can choose actions, transition probabilities depend on actions, state transitions yield rewards

• Goal: find policy maximizing cumulative rewards

• Bellman Optimality Equation: calculates optimal value (maximum expected reward)

• Value of a state: immediate reward + discounted value of future states weighted by probabilities

• Dynamic Programming: complex problems into smaller, simpler subproblems

Value Iteration Algorithm

• Calculates optimal value of all states using Bellman Optimality Equation
• Initializes state values to 0
• Iteratively updates state values until convergence
• Once optimal values known, optimal policy derived by maximizing rewards
• Example: Q-value iteration (works with state-action values Q(s,a) instead of state values V(s))
• Requires knowing transition probabilities (T) or rewards (R)

Q-Learning

• Model-free RL algorithm; estimates Q-values without knowing T or R
• Learns from experience, observing transitions (s → s′ with reward r)
• Updates Q-value estimates for (s, a) based on error (difference from target and current Q-value)
• ε-greedy policy: balances exploration (random actions) and exploitation (highest Q-value action)
• Temporal Difference (TD) learning similar but with state values instead of state-action values
• Q-Value iteration converges faster (less than 20 iterations) than Q-Learning (8,000 iterations)

Deep Q-Learning

• Large, complex environments
• Scalable neural network approximates Q-values via a Deep Neural Network (DNN)
• Experiences stored in replay memory
• Reduces correlations and stabilizes learning by randomly sampling experiences
• Faster training and more stable learning

Policy Gradient

• Policy-based method: learns optimal policy directly, not the value function
• Parameterizes policy function (e.g., neural network)
• Policy outputs probability distribution over actions for a given state
• Optimizes policy to maximize cumulative reward by defining and maximizing objective function to generate the best probabilities to be used as actions over time
• Policy gradient is an optimisation problem, gradient ascent is used; find the values of parameters that maximise objective function
• Can't calculate true gradient due to computational cost of all possible trajectories
• Policy Gradient Theorem reformulates objective function for gradients calculation

Explore the concepts of Reinforcement Learning (RL) and Markov Decision Processes (MDPs) through this quiz. Learn how intelligent agents make decisions to maximize their rewards while understanding the unique feedback mechanisms involved. Test your knowledge on dynamic programming and the Bellman Optimality Equation as you delve into advanced trading strategies.