Reinforcement Learning Concepts Quiz

A policy, denoted by π, is a function that maps each state of the environment to an action. It dictates the agent's behavior, determining which action it takes in a given state.

The value of a policy represents the expected cumulative reward the agent will receive by following that policy from a specific starting state.

A finite-horizon model considers a limited number of steps (T) in the future. The agent aims to maximize the expected reward within this timeframe.

An infinite-horizon model allows for an unlimited sequence of actions. However, future rewards are discounted to ensure that the total expected reward remains finite.

The discount factor (γ) determines how much future rewards are valued compared to immediate rewards.

Bellman's equation is a fundamental equation used to calculate the value of a state or state-action pair. It states that the value of a state is equal to the expected reward for taking the best action and then transitioning to the next state.

The optimal policy (π*) is the policy that maximizes the expected cumulative reward for all states.

The value of a state-action pair (Q(s,a)) represents the expected cumulative reward for taking action a in state s and then following the optimal policy thereafter.

Dynamic programming methods are used when you perfectly know the reward and next state probability distributions, but they can be computationally expensive.

When you don't know the reward or next state probability distributions, you need to explore the environment and learn from the sampled experience.

The environment's behavior is unknown; you need to experiment to understand how the system works.

Updating the value of the current state (action) based on the reward received in the next time step.

The difference between the predicted value of the current state and the actual value observed after taking an action.

A way to balance exploration and exploitation. You randomly select an action with a probability ε to explore, and you choose the best action with probability 1-ε to exploit.

Starting with a high exploration rate (ε) and gradually decreasing it to encourage exploitation as you gather more knowledge of the environment.

A method for finding an optimal policy by iteratively updating value estimates based on the temporal differences and rewards received.

The optimal policy is determined by choosing the action that maximizes the expected value in the next state, given a current state. It utilizes the optimal value function and a greedy approach to select the action that yields the highest cumulative reward.

A method used to find the optimal value function. It involves iteratively updating the values of states until they converge to a stable solution. The process stops when the maximum difference between values in consecutive iterations falls below a certain threshold.

An algorithm that directly updates the policy rather than relying on the convergence of values. It alternates between evaluating the value function for a given policy and improving the policy based on the evaluated values.

The idea behind Policy Iteration is to repeatedly improve a policy until it converges to optimal. This involves evaluating the current policy, generating a better policy based on the evaluation, and repeating the process until no further improvements can be made.

Policy Iteration assumes the environment is known, including the transition probabilities and reward functions. It leverages this knowledge to find the optimal policy through iterative updates.

The approach of finding the optimal policy in reinforcement learning, where the environment is known and the optimal value function is determined through the Bellman equation.

The environment's dynamics are known, which includes the transition probabilities between states and the rewards associated with taking actions. There is no need to explore the environment.

A policy is considered soft if it allows for the possibility of choosing any action in a given state, with a non-zero probability.

The softmax function is used to transform values (like Q-values) into probabilities, ensuring that the probability of choosing an action is always greater than zero.

The temperature parameter (T) in the softmax function controls the exploration-exploitation balance. A high temperature (T) encourages exploration by making all actions almost equally likely, while a low temperature favors exploitation by giving higher probabilities to actions with higher values.

Annealing is a technique used to manage the exploration-exploitation trade-off by gradually decreasing the temperature parameter (T) over time. This allows the agent to start with a more exploratory behavior and gradually shift towards exploiting the best actions it has discovered.

In a deterministic environment, every state-action pair has a single, predictable reward and next state. This simplifies the learning process, as the agent can reliably predict the consequences of its actions.

Bellman's equation is a fundamental equation used to calculate the value of a state or state-action pair, taking into account the immediate reward and the future rewards that can be obtained by taking a particular action.

The Bellman equation is used as an update rule to estimate the value of state-action pairs. By iteratively applying Bellman's equation, the agent can gradually improve its estimates of the values of different actions in different states.

The Q-value is a measure of how good taking a particular action in a particular state is. It takes into account both the immediate reward and the future rewards that can be obtained by taking that action and then following the optimal policy.

The estimated value of the current state is updated by considering the discounted value of the next state's value and adding the immediate reward. This update is called a backup.

In this scenario, the immediate rewards are either 0 or 100, depending on whether the goal state is reached. The discount factor (γ) is 0.9.

It is not necessary to model the reward or next state functions directly in this environment. We focus on learning the optimal policy through the estimated value function.

Q-values represent the expected cumulative reward for taking a specific action in a particular state. They steadily increase as better paths with higher cumulative rewards are discovered.

Q-values only increase and never decrease. As the agent discovers better paths, it updates its estimates based on the maximum cumulative rewards, resulting in higher Q-values.

When the environment involves non-deterministic aspects, such as an opponent or randomness, the agent keeps track of averages (expected values) instead of assigning direct values. This helps to account for the varying outcomes.

Even if the agent aims for a specific direction, it may deviate due to randomness. The agent needs to adjust its estimates to account for these deviations, taking into account the expected value of the outcome.

Due to non-deterministic outcomes, we cannot directly assign a reward or next state value. Instead, we keep a running average of rewards and next states observed over time.

Q-learning is a reinforcement learning algorithm that learns the optimal Q-values, which represent the expected cumulative reward for taking a specific action in a particular state and then following the optimal policy. It uses the Bellman equation to update the Q-values based on the current state, action, reward, and the maximum Q-value for the next state.

The Q-learning algorithm iteratively updates the Q-value for each state-action pair using a temporal difference (TD) approach. The update rule involves adding a weighted difference between the current Q-value estimate and a backed-up estimate, which considers the current reward and the maximum Q-value for the next state.

The learning rate (η) in the Q-learning update rule controls how much the current Q-value is adjusted based on the new information. A larger η means a faster learning rate but a smaller η makes the algorithm more stable. Over time, η is gradually decreased to ensure convergence to the optimal Q-values.

The discount factor (γ) in the Q-learning update rule determines how much future rewards are valued compared to immediate rewards. A larger γ means that the algorithm prioritizes future rewards, while a smaller γ focuses more on immediate rewards.

Q-learning is an off-policy method because it uses the maximum Q-value for the next state, regardless of the actual policy being followed. This means that the algorithm can learn the optimal policy even if it is not following it during training.

Sarsa is an on-policy version of Q-learning that uses the current policy to select the next action and its corresponding Q-value to update the Q-value of the current state-action pair.

Sarsa updates the Q-values based on the current state, action, reward, next state, and next action, which is determined by the current policy.

The main difference between Q-learning and Sarsa lies in their policy selection. Q-learning uses the optimal policy (greedy selection) to determine the next state's action, while Sarsa uses the current policy to determine the next action.