Reinforcement Learning: An Introduction

Introduction to Machine Learning: Reinforcement Learning

• Reinforcement learning aims to develop a system (agent) that improves performance through interactions within an environment.
• Information about the current state of the environment includes a reward signal.
• The agent uses reinforcement learning to learn a series of actions that maximizes rewards through trial-and-error or planning.
• Reinforcement learning’s history dates back to the 1950s.

Categorizing Machine Learning

• Machine learning is categorized into supervised learning, unsupervised learning, and reinforcement learning.
• Supervised learning uses features and labels to make predictions,
• Unsupervised learning identifies patterns in data
• Reinforcement learning uses rewards to determine the best actions.

Differences from Supervised/Unsupervised Learning

• Unlike supervised learning, reinforcement learning does not require a supervisor to guide the training process
• Reinforcement learning obtains data dynamically from the environment.
• Reinforcement learning runs inferences repeatedly, navigating through the real-world environment, unlike classification problems.
• Reinforcement learning differs from unsupervised learning; it not only focuses on finding hidden structure in unlabeled data.

Reinforcement Learning vs. Supervised Learning

• Supervised learning learns from labeled samples.
• Reinforcement learning is suitable when training samples are unavailable beforehand and when the agent needs to learn from its own experience in an unknown environment like robotics, game playing, or industrial controllers.

Reinforcement Learning Applications

• Reinforcement learning solves real-world problems like control tasks or decision tasks and it is used to operate systems that interact with the real world, like robots or drones learning to pick and place devices.
• Researchers demonstrated in 2013 that a system could outperform humans in Atari games using raw pixels as inputs.
• Google acquired DeepMind for over $500 million in 2014.
• Reinforcement learning involves mapping situations to actions to maximize a numerical reward signal.

Markov Decision Process Components

• The Markov Decision Process has five components
• Agent: The model being built and trained
• Environment: The real-world the agent interacts with
• State: The current state of the world, including the position of surrounding objects
• Action: Steps the agent takes to interact with its environment
• Reward: The positive or negative stimuli from the environment as a result of the agent’s actions.

Simplified Training Environment

• Training involves identifying the shortest path from a starting point to a finish line, using a grid where each square represents a state.
• The vehicle can move up or down while facing the goal.

Training Scores

• Each square is assigned a score to incentivize certain behavior.
• Squares at the track's edges are marked as "stop states," indicating failure.
• Providing high reward for squares on the center line and low elsewhere incentivizes driving down the track's center.

Episodes and Iterations

• Reinforcement training involves the vehicle exploring a grid until reaching out of bound or reaching an assigned destination.
• Reinforcement learning algorithms are trained by repeated optimization of cumulative rewards.
• Models learn, actions resulting in the highest cumulative reward on route to the goal.

Exploration and Convergence

• Exploration in reinforcement learning refers to an agent's random experimentation within its environment to discover new strategies or actions that may lead to better outcomes.
• Convergence in reinforcement learning refers to the process where an agent's learning algorithm stabilizes over time, leading to consistent and optimal or near-optimal behavior

Exploration vs Exploitation

• Exploration: It involves trying out new strategies or actions with the hope of discovering even better ways to achieve the desired goal.
• Exploitation: It involves using the knowledge and strategies the agent already possesses to make decisions that are expected to yield immediate rewards

Total Reward & Episodes

• Total Reward in RL represents the total reward an RL agent earns in a single episode.
• A higher total reward indicated better performance.
• Episodes are one complete run in the environment until a set termination is finished.

Key Elements of Reinforcement Learning System

• Policy defines the learning agent's way of behaving,
• Reward signal defines the goal for the agent
• Value function determines what is good in the long run.
• Model of the environment mimics behavior and allows infrences

State in Reinforcement Learning

• State in policy and value functions determines the agent's actions based on the current state.
• The model predicts the next state and reward.

Reinforcement Learning Model Training

• Over time, the reinforcement learning model learns from its experience.
• Experiences are used collected to update the neural network to create new experiences.
• Policy gradients and deep Q-networks (DQN) one of the most important techniques used in deep Reinforcement Learning.

Markov Decision Process Defined

• A Markov decision process (MDP) is a control process providing a mathematical framework for modeling decision making in random and controlled situations.
• MDP is a straightforward framing of learning from interaction to achieve a goal, is used to describe an environment for reinforcement learning where the environment is fully observable

Q-Learning and Policy Gradients

• Q-Learning is a value-based reinforcement learning algorithm to find optimal action-selection policy using a Q function.
• Policy Gradients is a method to directly optimize in the action space.
• Q-learning learns a single deterministic action from possible actions, meanwhile policy gradients and other direct policy searches learn a map that works in continuous action spaces and can be stochastic

Value Learning Problem

• The value learning problem highlights the differences between humans and computers and how they think, it stems from how difficult for computers to determine what to value.

Applications of Reinforcement Learning

• Reinforcement Learning can be applied to solve real world problems like: autonomous driving, securities trading, neural network architecture search, simulated training of robots and AI Agents for playing video games.

Autonomous Driving

• Reinforcement Learning is useful for learning driving strategies, and helps autonomous vehicles to interact in virtual environments.

Securities Trading

• Reinforcement Learning has many applications to the complex financial marketplace
• It can be used to develop trading algorithms for complex financial strategies.

Neural Network Architecture Search

• Reinforcement Learning has many applications to the complex financial marketplace
• It can be used to develop trading algorithms for complex financial strategies.

Simulating Robots

• Reinforcement learning can train train robots in simulated environments before deploying them in the real world, and for completing tasks efficiently.

AI Agents

• Reinforcement Learning is useful for training AI agents to adapt to enviroments in a game.

RL General Considerations

• The Reinforcement Learning is powerful, that may be considered as overkill due to efficiency problems and computational complexity.
• Reinforcement Learning is highly based on data.
• Also is assumed that Markovian environment simplifies decision-making and makes learning more efficient.

RL Overkill Scenario Comparison

• Simple decision tasks should use the most traditional supervised learning techniques such as Support Vector Machines, Convolutional NNs, Decision Trees, etc
• Optimization tasks can be perform by Heuristic techniques and genetical or statistical optimization algorithms
• Task with predefined rules based on the same rule and dynamic programming
• Complex problems can benefit from Robot Learning to make Al and Strategy easier

Considerations for Reinforcement Learning

• Training can take a long time and is not always stable.
• The complexity can takes a long time on hardware
• Models Algorithms may have problems to converge
• Exploitation should have an Al in order to work well.

Working Class Enviroments

• The model of the system that is giving observations and rewards to the agent.
• There must be a relation between model and agent and the capacity to communicate
• All enviroments should be able to handle actions ,set the number of episodes and checks the end of the episode.

CartPole Reinforcement Enviroment

• CartPole is a classic RL environment with the goal to balance a pole on a moving cart by applying actions to move the cart left or right.
• The observations provide the state of the cart-pole system with four values (Cart Position, Cart Velocity, Pole Angle and Pole Velocity) and the Actions (Agent's Choices) allows 2 options move left and rigth

  RL General Goals

• The goal is to gives a reward per step
• Exploration vs. Exploitation helps balance exploring new strategies

OpenAI gym is open source and can be integrated with many other systems.

• All rewards gives to RL agents can be analyzed and improved

Key steps for QValue Implementation

• The program restarts when the episode concludes
• An algorithm runs for the first time and set the values
• When a value is found repeats the process until find the right solution