Self-Driving Cars Lecture 4 - Reinforcement Learning PDF

Self-Driving Cars Lecture 4 – Reinforcement Learning Prof. Dr.-Ing. Andreas Geiger Autonomous Vision Group University of Tübingen / MPI-IS Agenda 4.1 Markov Decision Processes 4.2 Bellman Optimality and Q-Learning 4.3 Deep Q-Learning...

Self-Driving Cars Lecture 4 – Reinforcement Learning Prof. Dr.-Ing. Andreas Geiger Autonomous Vision Group University of Tübingen / MPI-IS Agenda 4.1 Markov Decision Processes 4.2 Bellman Optimality and Q-Learning 4.3 Deep Q-Learning 2 4.1 Markov Decision Processes Reinforcement Learning So far: I Supervised learning, lots of expert demonstrations required I Use of auxiliary, short-term loss functions I Imitation learning: per-frame loss on action I Direct perception: per-frame loss on affordance indicators Now: I Learning of models based on the loss that we actually care about, e.g.: I Minimize time to target location I Minimize number of collisions I Minimize risk I Maximize comfort I etc. Sutton and Barto: Reinforcement Learning: An Introduction. MIT Press, 2017. 4 Types of Learning Supervised Learning: I Dataset: {(xi , yi )} (xi = data, yi = label) Goal: Learn mapping x 7→ y I Examples: Classification, regression, imitation learning, affordance learning, etc. Unsupervised Learning: I Dataset: {(xi )} (xi = data) Goal: Discover structure underlying data I Examples: Clustering, dimensionality reduction, feature learning, etc. Reinforcement Learning: I Agent interacting with environment which provides numeric reward signals I Goal: Learn how to take actions in order to maximize reward I Examples: Learning of manipulation or control tasks (everything that interacts) Sutton and Barto: Reinforcement Learning: An Introduction. MIT Press, 2017. 5 Introduction to Reinforcement Learning Agent Reward rt State st Action at Next state st+1 Environment I Agent oberserves environment state st at time t I Agent sends action at at time t to the environment I Environment returns the reward rt and its new state st+1 to the agent Sutton and Barto: Reinforcement Learning: An Introduction. MIT Press, 2017. 6 Introduction to Reinforcement Learning I Goal: Select actions to maximize total future reward I Actions may have long term consequences I Reward may be delayed, not instantaneous I It may be better to sacrifice immediate reward to gain more long-term reward I Examples: I Financial investment (may take months to mature) I Refuelling a helicopter (might prevent crash in several hours) I Sacrificing a chess piece (might help winning chances in the future) Sutton and Barto: Reinforcement Learning: An Introduction. MIT Press, 2017. 7 Example: Cart Pole Balancing I Objective: Balance pole on moving cart I State: Angle, angular vel., position, vel. I Action: Horizontal force applied to cart I Reward: 1 if pole is upright at time t https://gym.openai.com/envs/#classic_control 8 Example: Robot Locomotion I Objective: Make robot move forward I State: Position and angle of joints I Action: Torques applied on joints I Reward: 1 if upright & forward moving http://blog.openai.com/roboschool/ https://gym.openai.com/envs/#mujoco 9 Example: Atari Games I Objective: Maximize game score I State: Raw pixels of screen (210x160) I Action: Left, right, up, down I Reward: Score increase/decrease at t http://blog.openai.com/gym-retro/ https://gym.openai.com/envs/#atari 10 Example: Go I Objective: Winning the game I State: Position of all pieces I Action: Location of next piece I Reward: 1 if game won, 0 otherwise www.deepmind.com/research/alphago/ www.deepmind.com/research/alphago/ 11 Example: Self-Driving I Objective: Lane Following I State: Image (96x96) I Action: Acceleration, Steering I Reward: - per frame, + per tile https://gym.openai.com/envs/CarRacing-v0/ 12 Reinforcement Learning: Overview Agent Reward rt State st Action at Next state st+1 Environment I How can we mathematically formalize the RL problem? 13 Markov Decision Process Markov Decision Process (MDP) models the environment and is defined by the tuple (S, A, R, P, γ) with I S : set of possible states I A: set of possible actions I R(rt |st , at ): distribution of current reward given (state,action) pair I P (st+1 |st , at ): distribution over next state given (state,action) pair I γ: discount factor (determines value of future rewards) Almost all reinforcement learning problems can be formalized as MDPs 14 Markov Decision Process Markov property: Current state completely characterizes state of the world I A state st is Markov if and only if P (st+1 |st ) = P (st+1 |s1 ,..., st ) I ”The future is independent of the past given the present” I The state captures all relevant information from the history I Once the state is known, the history may be thrown away I The state is a sufficient statistics of the future 15 Markov Decision Process Reinforcement learning loop: I At time t = 0: I Environment samples initial state s0 ∼ P (s0 ) Agent I Then, for t = 0 until done: rt st at I st+1 Agent selects action at I Environment samples reward rt ∼ R(rt |st , at ) Environment I Environment samples next state st+1 ∼ P (st+1 |st , at ) I Agent receives reward rt and next state st+1 How do we select an action? 16 Policy A policy π is a function from S to A that specifies what action to take in each state: I A policy fully defines the behavior of an agent I Deterministic policy: a = π(s) I Stochastic policy: π(a|s) = P (at = a|st = s) Remark: I MDP policies depend only on the current state and not the entire history I However, the current state may include past observations 17 Policy How do we learn a policy? Imitation Learning: Learn a policy from expert demonstrations I Expert demonstrations are provided I Supervised learning problem Reinforcement Learning: Learn a policy through trial-and-error I No expert demonstrations given I Agent discovers itself which actions maximize the expected future reward I The agent interacts with the environment and obtains reward I The agent discovers good actions and improves its policy π 18 Exploration vs. Exploitation How do we discover good actions? Answer: We need to explore the state/action space. Thus RL combines two tasks: I Exploration: Try a novel action a in state s , observe reward rt I Discovers more information about the environment, but sacrifices total reward I Game-playing example: Play a novel experimental move I Exploitation: Use a previously discovered good action a I Exploits known information to maximize reward, but sacrifice unexplored areas I Game-playing example: Play the move you believe is best Trade-off: It is important to explore and exploit simultaneously 19 Exploration vs. Exploitation How to balance exploration and exploitation? -greedy exploration algorithm: I Try all possible actions with non-zero probability I With probability  choose an action at random (exploration) I With probability 1 −  choose the best action (exploitation) I Greedy action is defined as best action which was discovered so far I  is large initially and gradually annealed (=reduced) over time 20 4.2 Bellman Optimality and Q-Learning Value Functions How good is a state? The state-value function V π at state st is the expected cumulative discounted reward (rt ∼ R(rt |st , at )) when following policy π from state st :   X V π (st ) = E[rt + γrt+1 + γ 2 rt+2 +... |st , π] = E  γ k rt+k st , π  k≥0 I The discount factor γ < 1 is the value of future rewards at current time t I Weights immediate reward higher than future reward (e.g., γ = 21 ⇒ γ k = 11 , 12 , 14 , 18 , 16 1 ,...) I Determines agent’s far/short-sightedness I Avoids infinite returns in cyclic Markov processes 22 Value Functions How good is a state-action pair? The action-value function Qπ at state st and action at is the expected cumulative discounted reward when taking action at in state st and then following the policy π:   X Qπ (st , at ) = E  γ k rt+k st , at , π  k≥0 I The discount factor γ ∈ [0, 1] is the value of future rewards at current time t I Weights immediate reward higher than future reward (e.g., γ = 21 ⇒ γ k = 11 , 12 , 14 , 18 , 16 1 ,...) I Determines agent’s far/short-sightedness I Avoids infinite returns in cyclic Markov processes 23 Optimal Value Functions The optimal state-value function V ∗ (st ) is the best V π (st ) over all policies π:   X V ∗ (st ) = max V π (st ) V π (st ) = E  γ k rt+k st , π  π k≥0 The optimal action-value function Q∗ (st , at ) is the best Qπ (st , at ) over all policies π:   X Q∗ (st , at ) = max Qπ (st , at ) Qπ (st , at ) = E  γ k rt+k st , at , π  π k≥0 I The optimal value functions specify the best possible performance in the MDP I However, searching over all possible policies π is computationally intractable 24 Optimal Policy If Q∗ (st , at ) would be known, what would be the optimal policy? π ∗ (st ) = argmax Q∗ (st , a0 ) a0 ∈A I Unfortunately, searching over all possible policies π is intractable in most cases I Thus, determining Q∗ (st , at ) is hard in general (for most interesting problems) I Let’s have a look at a simple example where the optimal policy is easy to compute 25 A Simple Grid World Example states actions = { 1. right ? 2. left reward: r = −1 for 3. up ? each transition 4. down } Objective: Reach one of terminal states (marked with ’?’) in least number of actions I Penalty (negative reward) given for every transition made 26 A Simple Grid World Example ? ? ? ? Random Policy Optimal Policy I The arrows indicate equal probability of moving into each of the directions 27 Solving for the Optimal Policy Bellman Optimality Equation I The Bellman Optimality Equation is named after Richard Ernest Bellman who introduced dynamic programming in 1953 I Almost any problem which can be solved using optimal control theory can be solved via the appropriate Bellman equation Richard Ernest Bellman Sutton and Barto: Reinforcement Learning: An Introduction. MIT Press, 2017. 29 Bellman Optimality Equation The Bellman Optimality Equation (BOE) decomposes Q∗ as follows: Q∗ (st , at ) = E rt + γrt+1 + γ 2 rt+2 +... |st , at     BOE ∗ 0 = E rt + γ max 0 Q (st+1 , a ) st , at a ∈A This recursive formulation comprises two parts: I Current reward: rt I Discounted optimal action-value of successor: γ max Q∗ (st+1 , a0 ) 0 a ∈A We want to determine Q∗ (st , at ). How can we solve the BOE? I The BOE is non-linear (because of max-operator) ⇒ no closed form solution I Several iterative methods have been proposed, most popular: Q-Learning Sutton and Barto: Reinforcement Learning: An Introduction. MIT Press, 2017. 30 Proof of the Bellman Optimality Equation Proof of the Bellman Optimality Equation for the optimal action-value function Q∗ : Q∗ (st , at ) = E rt + γrt+1 + γ 2 rt+2 +... |st , at     X = E γ k rt+k |st , at  k≥0   X = E rt + γ γ k rt+k+1 |st , at  k≥0 ∗ = E [rt + γV (st+1 )|st , at ]   ∗ 0 = E rt + γ max 0 Q (st+1 , a )|st , at a Sutton and Barto: Reinforcement Learning: An Introduction. MIT Press, 2017. 31 Bellman Optimality Equation Why is it useful to solve the BOE? I A greedy policy which chooses the action that maximizes the optimal action-value function Q∗ or the optimal state-value function V ∗ takes into account the reward consequences of all possible future behavior I Via Q∗ and V ∗ the optimal expected long-term return is turned into a quantity that is locally and immediately available for each state / state-action pair I For V ∗ , a one-step-ahead search yields the optimal actions I Q∗ effectively caches the results of all one-step-ahead searches Sutton and Barto: Reinforcement Learning: An Introduction. MIT Press, 2017. 32 Q-Learning Q-Learning: Iteratively solve for Q∗   ∗ ∗ 0 Q (st , at ) = E rt + γ max 0 Q (st+1 , a ) st , at a ∈A by constructing an update sequence Q1 , Q2 ,... using learning rate α: Qi+1 (st , at ) ← (1 − α)Qi (st , at ) + α(rt + γ max 0 Qi (st+1 , a0 )) a ∈A = Qi (st , at ) + α (rt + γ max Qi (st+1 , a0 ) − Qi (st , at )) a0 ∈A | {z } prediction | {z } target | {z } temporal difference (TD) error I Qi will converge to Q∗ as i → ∞ Note: policy π learned implicitly via Q table! Watkins and Dayan: Technical Note Q-Learning. Machine Learning, 1992. 33 Q-Learning Implementation: I Initialize Q table and initial state s0 randomly I Repeat: I Observe state st , choose action at according to -greedy strategy (Q-Learning is “off-policy” as the updated policy is different from the behavior policy) I Observe reward rt and next state st+1 I Compute TD error: rt + γ max Qi (st+1 , a0 ) − Qi (st , at ) a0 ∈A I Update Q table What’s the problem with using Q tables? I Scalability: Tables don’t scale to high dimensional state/action spaces (e.g., GO) I Solution: Use a function approximator (neural network) to represent Q(s, a) Watkins and Dayan: Technical Note Q-Learning. Machine Learning, 1992. 34 4.3 Deep Q-Learning Deep Q-Learning Use a deep neural network with weights θ as function approximator to estimate Q: Q(s, a; θ) ≈ Q∗ (s, a) Q(s, a; θ) Q(s, a1 ; θ),...Q(s, am ; θ) θ θ s a s Mnih et al.: Human-level control through deep reinforcement learning. Nature, 2015. 36 Training the Q Network Forward Pass: Loss function is the mean-squared error in Q-values:  2  rt + γ max Q(st+1 , a0 ; θ) − Q(st , at ; θ)     L(θ) = E   a0 | {z }  prediction | {z } target Backward Pass: Gradient update with respect to Q-function parameters θ: " 2 # ∇θ L(θ) = ∇θ E rt + γ max 0 Q(st+1 , a0 ; θ) − Q(st , at ; θ) a Optimize objective end-to-end with stochastic gradient descent (SGD) using ∇θ L(θ). Mnih et al.: Human-level control through deep reinforcement learning. Nature, 2015. 37 Experience Replay To speed-up training we like to train on mini-batches: I Problem: Learning from consecutive samples is inefficient I Reason: Strong correlations between consecutive samples Experience replay stores agent’s experiences at each time-step I Continually update a replay memory D with new experiences et = (st , at , rt , st+1 ) I Train on samples (st , at , rt , st+1 ) ∼ U (D) drawn uniformly at random from D I Breaks correlations between samples I Improves data efficiency as each sample can be used multiple times In practice, a circular replay memory of finite memory size is used. Mnih et al.: Human-level control through deep reinforcement learning. Nature, 2015. 38 Fixed Q Targets Problem: Non-stationary targets I As the policy changes, so do our targets: rt + γ max Q(st+1 , a0 ; θ) 0 a I This may lead to oscillation or divergence Solution: Use fixed Q targets to stabilize training I A target network Q with weights θ− is used to generate the targets: " 2 # L(θ) = E(st ,at ,rt ,st+1 )∼U (D) rt + γ max 0 Q(st+1 , a0 ; θ− ) − Q(st , at ; θ) a I Target network Q is only updated every C steps by cloning the Q-network I Effect: Reduces oscillation of the policy by adding a delay Mnih et al.: Human-level control through deep reinforcement learning. Nature, 2015. 39 Putting it together Deep Q-Learning using experience replay and fixed Q targets: I Take action at according to -greedy policy I Store transition (st , at , rt , st+1 ) in replay memory D I Sample random mini-batch of transitions (st , at , rt , st+1 ) from D I Compute Q targets using old parameters θ− I Optimize MSE between Q targets and Q network predictions " 2 # 0 − L(θ) = Est ,at ,rt ,st+1 ∼D rt + γ max 0 Q(st+1 , a ; θ ) − Q(st , at ; θ) a using stochastic gradient descent. Mnih et al.: Human-level control through deep reinforcement learning. Nature, 2015. 40 Case Study: Playing Atari Games Agent ; ; ; Environment Objective: Complete the game with the highest score Mnih et al.: Human-level control through deep reinforcement learning. Nature, 2015. 41 Case Study: Playing Atari Games Q(s, a; θ): Neural network with weights θ Output: Q values for all (4 to 18) Atari actions FC-Out (Q values) (efficient: single forward pass computes Q for all actions) FC-256 32 4x4 conv, stride 2 16 8x8 conv, stride 2 ;;; Input: 84 × 84 × 4 stack of last 4 frames ; (after grayscale conversion, downsampling, cropping) Mnih et al.: Human-level control through deep reinforcement learning. Nature, 2015. 42 Case Study: Playing Atari Games Mnih et al.: Human-level control through deep reinforcement learning. Nature, 2015. 43 Deep Q-Learning Shortcomings Deep Q-Learning suffers from several shortcomings: I Long training times I Uniform sampling from replay buffer ⇒ all transitions equally important I Simplistic exploration strategy I Action space is limited to a discrete set of actions (otherwise, expensive test-time optimization required) Various improvements over the original algorithm have been explored. 44 Deep Deterministic Policy Gradients DDPG addresses the problem of continuous action spaces. Problem: Finding a continuous action requires optimization at every timestep. Solution: Use two networks, an actor (deterministic policy) and a critic. µ(s; θµ ) Q(s, a; θQ ) θµ θQ s s a = µ(s; θµ ) Actor Critic Lillicrap et al.: Continuous Control with Deep Reinforcement Learning. ICLR, 2016. 45 Deep Deterministic Policy Gradients I Actor network with weights θµ estimates agent’s deterministic policy µ(s; θµ ) I Update deterministic policy µ(·) in direction that most improves Q I Apply chain rule to the expected return (this is the policy gradient): ∇θµ Est ,at ,rt ,st+1 ∼D Q(st , µ(st ; θµ ); θQ ) = E ∇at Q(st , at ; θQ ) ∇θµ µ(st ; θµ )     I Critic estimates value of current policy Q(s, a; θQ ) I Learned using the Bellman Optimality Equation as in Q Learning: h
2 i ∇θQ Est ,at ,rt ,st+1 ∼D rt + γQ(st+1 , µ(st+1 ; θµ− ); θQ− ) − Q(st , at ; θQ ) I Remark: No maximization over actions required as this step is now learned via µ(·) Lillicrap et al.: Continuous Control with Deep Reinforcement Learning. ICLR, 2016. 46 Deep Deterministic Policy Gradients Experience replay and target networks are again used to stabilize training: I Replay memory D stores transition tuples (st , at , rt , st+1 ) I Target networks are updated using “soft” target updates I Weights are not directly copied but slowly adapted: θQ− ← τ θQ + (1 − τ )θQ− θµ− ← τ θµ + (1 − τ )θµ− where 0 < τ  1 controls the tradeoff between speed and stability of learning Exploration is performed by adding noise ∇θµ to the policy µ(s): µ(s; θµ ) + N Lillicrap et al.: Continuous Control with Deep Reinforcement Learning. ICLR, 2016. 47 Prioritized Experience Replay Prioritize experience to replay important transitions more frequently I Priority δ is measured by magnitude of temporal difference (TD) error: δ = rt + γ max 0 Q(st+1 , a0 ; θQ− ) − Q(st , at ; θQ ) a I TD error measures how “surprising” or unexpected the transition is I Stochastic prioritization avoids overfitting due to lack of diversity I Enables learning speed-up by a factor of 2 on Atari benchmarks Schaul et al.: Prioritized Experience Replay. ICLR, 2016. 48 Learning to Drive in a Day Real-world RL demo by Wayve: I Deep Deterministic Policy Gradients with Prioritized Experience Replay I Input: Single monocular image I Action: Steering and speed I Reward: Distance traveled without the safety driver taking control (requires no maps / localization) I 4 Conv layers, 2 FC layers I Only 35 training episodes Kendall, Hawke, Janz, Mazur, Reda, Allen, Lam, Bewley and Shah: Learning to Drive in a Day. ICRA, 2019. 49 Learning to Drive in a Day Kendall, Hawke, Janz, Mazur, Reda, Allen, Lam, Bewley and Shah: Learning to Drive in a Day. ICRA, 2019. 50 Other flavors of Deep RL Asynchronous Deep Reinforcement Learning Execute multiple agents in separate environment instances: I Each agent interacts with its own environment copy and collects experience I Agents may use different exploration policies to maximize experience diversity I Experience is not stored but directly used to update a shared global model I Stabilizes training in similar way to experience replay by decorrelating samples I Leads to reduction in training time roughly linear in the number of parallel agents Mnih et al.: Asynchronous Methods for Deep Reinforcement Learning. ICML, 2016. 52 Bootstrapped DQN Bootstrapping for efficient exploration: I Approximate a distribution over Q values via K bootstrapped ”heads” I At the start of each epoch, a single head Qk is selected uniformly at random I After training, all heads can be combined into a single ensemble policy Q1 QK θQ 1... θQK θshared s Osband et al.: Deep Exploration via Bootstrapped DQN. NIPS, 2016. 53 Double Q-Learning Double Q-Learning I Decouple Q function for selection and evaluation of actions to avoid Q overestimation and stabilize training. Target: DQN : rt + γ max 0 Q(st+1 , a0 ; θ− ) a DoubleDQN : rt + γQ(st+1 , argmax Q(st+1 , a0 ; θ); θ− ) a0 I Online network with weights θ is used to determine greedy policy I Target network with weights θ− is used to determine corresponding action value I Improves performance on Atari benchmarks van Hasselt et al.: Deep Reinforcement Learning with Double Q-learning. AAAI, 2016. 54 Deep Recurrent Q-Learning Add recurrency to a deep Q-network to handle partial observability of states: FC-Out (Q-values) LSTM Replace fully-connected layer with recurrent LSTM layer 32 4x4 conv, stride 2 16 8x8 conv, stride 2 ;;;; Hausknecht and Stone: Deep Recurrent Q-Learning for Partially Observable MDPs. AAAI, 2015 55 Faulty Reward Functions https://blog.openai.com/faulty-reward-functions/ 56 Summary I Reinforcement learning learns through interaction with the environment I The environment is typically modeled as a Markov Decision Process I The goal of RL is to maximize the expected future reward I Reinforcement learning requires trading off exploration and exploitation I Q-Learning iteratively solves for the optimal action-value function I The policy is learned implicitly via the Q table I Deep Q-Learning scales to continuous/high-dimensional state spaces I Deep Deterministic Policy Gradients scales to continuous action spaces I Experience replay and target networks are necessary to stabilize training 57