Chapter 4 - Medium

Policy-Based Reinforcement Learning

• Policy-based reinforcement learning directly optimizes the policy, making it suitable for continuous action spaces.
• Value-based methods are difficult to use in continuous action spaces because they require discretizing the action space or finding the maximum value action, which is computationally infeasible.

Batch Accumulation

• Accumulating gradients over multiple episodes and then updating the parameters.
• This approach helps to reduce variance and improve learning stability.

Bias-Variance Trade-Off

• Balancing the trade-off between bias (error due to approximations) and variance (error due to randomness) to ensure stable and efficient learning.
• This trade-off is crucial in policy-based methods to achieve optimal performance.

Actor-Critic Algorithm

• The actor-critic algorithm combines policy-based (actor) and value-based (critic) approaches to reduce variance and improve learning stability.
• The algorithm updates the actor and critic parameters simultaneously using the TD error and policy gradient.

Temporal Difference Bootstrapping

• A method that updates value estimates based on estimates of future values.
• It combines the ideas of dynamic programming and Monte Carlo methods.
• The update rule for the state-value function V is: V(st) ← V(st) + α[rt+1 + γV(st+1) − V(st)]

Baseline Subtraction with Advantage Function

• The advantage function measures how much better an action is compared to the average action in a given state.
• The advantage function is defined as A(s, a) = Q(s, a) − V(s).
• Baseline subtraction with the advantage function helps to reduce variance and improve learning stability.

Generic Policy Gradient Formulation

• The policy gradient formulation is: ∇θJ(θ) = E[∇θ log π(a|s, θ)A(s, a)].

Asynchronous Advantage Actor-Critic

• The A3C algorithm runs multiple agents in parallel to update the policy and value functions asynchronously.
• This approach helps to speed up learning by exploring the environment simultaneously and asynchronously updating the global policy and value functions.

Trust Region Policy Optimization

• TRPO ensures policy updates are within a trust region to maintain stability and prevent large, destabilizing updates.
• The algorithm updates the policy parameters using the conjugate gradient method and the trust region.

Entropy and Exploration

• Entropy is added to the objective function to encourage exploration by preventing premature convergence.
• Soft Actor-Critic is an algorithm that incorporates entropy maximization into the policy update to balance exploration and exploitation.

Deterministic Policy Gradient

• DDPG combines DQN and policy gradients for environments with continuous actions.
• The deterministic policy gradient is: ∇θJ(θ) = E[∇aQ(s, a)∇θμ(s, θ)].

Hands-On: PPO and DDPG MuJoCo Examples

• PPO is a method that simplifies TRPO while retaining performance.
• DDPG is applied to control tasks in MuJoCo to demonstrate policy learning in continuous action spaces.

Locomotion and Visuo-Motor Environments

• Locomotion and visuo-motor environments focus on training agents to move and interact with their surroundings using visual inputs and motor actions.
• Examples include training agents to walk, run, or fly using policy-based methods.

Benchmarking

• Benchmarking is used to evaluate and compare the performance of reinforcement learning algorithms.
• Examples include evaluating different policy-based methods on common benchmarks like MuJoCo locomotion tasks or Atari games.

Learn about the actor-critic algorithm, bias-variance trade-off in policy-based methods, and actor-critic bootstrapping in reinforcement learning.