Reinforcement Learning: An Introduction

• Reinforcement Learning (RL) is a machine learning type where an agent learns decision-making through environment interaction.
• The agent takes actions across varying environmental states, receiving rewards or penalties as feedback.
• The agent's goal is to develop a policy that maximizes the cumulative reward over time.
• RL differs from supervised learning, instead relying on trial and error, environmental exploration, and feedback utilization to improve future behavior.
• RL emphasizes maximizing long-term rewards, which is useful for solving sequential decision-making problems.

Key RL Components

• Agent: The learner and decision-maker.
• Environment: The external world, which interacts with the agent.
• The agent's current situation or configuration is known as "State."
• Action: The set of possible moves an agent can make.
• Reward: Feedback or a scalar used to evaluate an agent action.
• Policy: A strategy that maps state to actions.
• Value Function: A function estimating a state's value relating to future rewards.
• Exploration vs. Exploitation: A dilemma where you must find a balance between exploring new actions for information and current ones to maximize rewards.

Self-Driving Car Example Components

• Agent: The self-driving car. It determines driving actions in varying conditions, such as accelerating, braking, turning, or remaining in the current lane.

• Environment: The city and roads, including traffic lights, pedestrians, weather, road signs, and lanes.

• State: The car's current situation, including its position, speed, distance from other cars, traffic light status, and weather.

• Actions: Steering, changing lanes, accelerating, decelerating, and stopping.

• Reward: Feedback signals evaluating agent actions. Positive rewards include staying in lanes, maintaining safe distances, and reaching destinations. Negative rewards include, collisions or traffic violations.

• Policy: Determines the best course of action depending on the current state. For instance, it dictates stopping at a red light or yielding to pedestrians.

• Value Function: Estimated long-term reward in the current state, it helps prioritize the best thing to do. High value is associated with safe driving and destination proximity, while low value relates to collisions or being far from the destination.

• Exploration vs. Exploitation: Balancing the trade-off between trying new routes or maneuvers to gather more information and using proven safe driving strategies

• Reinforcement learning depends on a constantly changing environment.

• RL aims to determine the most effective sequence of actions, not categorization or labeling.

• An agent explores, interacts with, and learns via trial and error.

• An agent contains a function that maps state observations, or inputs, to the actions, or outputs.

• RL calls this a "policy" deciding action based on input observations.

• In a self-driving car, observations are the steering wheel angle, acceleration, and speed, the vision sensor recognizes data that is used in conjunction with a policy that outputs servo commands.

• The environment generates a reward telling the agent how well the actuator commands did, with rewards reflecting if the car successfully stays on the road or if it has an accident.

• The agent uses reinforcement learning algorithms to figure out the best course of action as the optimal action produces the most rewards in the long run.

• A policy can be explained as logic and tunable parameters

• Reinforcement learning algorithms can tune parameters, focusing on structure, which can optimize results.

RL Project Stages

• Environment: Choose an environment where the agent can learn, it can be a real environment or a simulated one.
• Rewards: Establish a reward mechanism that incentivizes desired agent behaviors.
• Policy: Represent the agent's decision-making function through explicit rules, parameters, or a neural network.
• Training: Use algorithms to train the agent and refine the policy parameters.
• Deploy: Test and implement the agent in a real-world setting.
• The "environment" constitutes everything outside the agent that sends actions and generates rewards.
• "Model-free reinforcement learning" enables the agent to interact without prior knowledge of dynamics.
• Agents can learn how to maximize rewards or mitigate aversive scenarios when using model-free RL.
• Model-free RL helps you equip an RL agent to learn optimal policies.

Model-Based RL

• Agents are given a map to help them and reduce exploration during the process.
• Model-based RL lowers learning times, because you can guide the agent away from states with low reward.

Real vs Simulated Environments

• In real environments, nothing represents the environment’s elements more accurately than the real environment
• No time has to be spent creating a model.
• Training may require constant changing of the real environment
• Simulated conditions provide speed and the ability to produce different situations
• There is no hardware damage in simulated conditions.
• A "function" realizes the reward signal.
• This function takes an agent's action with a current state and provides a scalar value.

Reward Aspects

• Rewards are impacted by the behavior (action given a state).
• This function's design has sparse rewards, rewards every time step, or only at the episode's conclusion. It can have large calculations and parameters.
• If there are not many restrictions on rewards functions, you must be mindful of "sparsity", this is when rewards only happen after actions.
• When the agent stumbles for a very long time it has a hard time receiving rewards. Shaping rewards is giving the agent a smaller reward, such as a robot moving 1m towards the 10m goal.
• Engineering rewards requires domain knowledge.
• In exploration vs exploitation, agents must choose where to take the actions it knows most about, or explore new parts and collect the most rewards.
• It is important to occasionally let the agent explore, and expand policies for new states.
• Balancing exploration and exploitation has the agent settle throughout the learning process, taking more actions throughout the role.
• Value focuses on assessing the state and helps collect the most amounts of rewards.

Short-Termism Value

• The instant benefit is when taking or being in certain states
• Value represents all the rewards that have an expected average moving forward.
• The best bet isn't the best at the beginning, receiving awards after sequential actions.
• It's more advantageous to be short-sighted when creating value
• Short-term value increases by discounting rewards by larger amounts in the future, done by setting a discount factor.
• Policy is a function that creates an algorithm that provides its state and all rewards.

Q-Table Policies

• In an environment that has discrete numbers, one can use a simple way to have policies represented.
• Tables represent an array of numbers where inputs have lookups and act as outputs.
• Q tables map all states to the designated value.
• A policy can check if values are given and then it selects actions Agents with "Q-tables" find the values for each stated pair.
• Once the table is full, it will choose the action that creates values for most rewards
• Neural neworks create the policy used by algorithms in the agency.

Machine Learning Models

• Creating and making a new network uses various interconnected algorithms.
• There are different techniques that are used when constructing a machine learning model, such as Actor-Critic.
• In continuous states and actions, it takes so much training in order to work at the beginning, otherwise known as, Dimensionality.
• A neural network is required, that way it handles what the network is meant for, or whatever the algorithm may be used for.

Policy Function-Based Learning Algorithms:

• Used to train neural networks that take in actions.
• The network is the policy that directly tells the agency what to take