Charta is on Google Play: practise on the go. Get the app
Charta

Describe the operation of reinforcement learning and calculate the Q-values used when making decisions.

FRM® Part I · Quantitative Analysis · Machine-Learning Methods · FRM1.QA.14.i

The idea

In reinforcement learning an agent observes a state, takes an action, receives a reward and moves to a new state, aiming to maximise discounted cumulative reward. The Q-value Q(s,a) estimates the long-run reward of taking action a in state s and then acting well. Q-learning updates it step by step. Qnew = Qold + α × [R + γ × max Q(next state) − Qold]. Here α is the learning rate and γ is the discount factor, between 0 and 1. The bracket is the gap between the better estimate and the old one. The maximum is taken over actions in the next state, whether or not the agent actually chooses that action. The agent balances exploration, trying actions at random with probability ε, against exploitation of the highest Q-value.

Formulas

Q-learning update

Qnew = Qold + α × [R + γ × max Q(s′) − Qold]

Qold
the current estimate of Q(s, a)
α
the learning rate
R
the reward just received
γ
the discount factor, between 0 and 1
max Q(s′)
the highest Q-value over actions in the next state s′

Use the maximum over next-state actions even when exploration leads the agent to take a different action.

A practice question

From the question bank, with the reason each wrong option is wrong.

In reinforcement learning, what does a Q-value Q(s, a) represent?

  1. The estimated long-run discounted reward from taking action a in state s and acting well afterwards. - correct
  2. The probability that the agent is in state s at any given time.Why not: Q-values are not state probabilities.
  3. The immediate reward alone, ignoring all future states and rewards.Why not: Q-values include discounted future rewards.
  4. The labelled correct action that a teacher would choose in state s.Why not: There are no labelled answers in reinforcement learning.

Q(s,a) is the expected cumulative discounted reward, so the agent can compare actions in each state.

A harder one

A Q-learning agent has Q(s, a) = 2.0. It takes a, earns R = 1.0, and the best Q-value in the next state is 4.0. The learning rate is 0.1 and the discount factor 0.9. Using Q(new) = Q(old) + alpha x [R + gamma x max Q(next) - Q(old)], what is the updated Q-value?

  1. 4.60Why not: Replaces Q with the target, equivalent to a learning rate of 1.
  2. 2.30Why not: Omits the discount factor on the next-state value.
  3. 2.26 - correct
  4. 2.46Why not: Forgets to subtract the old Q inside the bracket.

Target = 1.0 + 0.9 x 4.0 = 4.6; error = 4.6 - 2.0 = 2.6; new Q = 2.0 + 0.1 x 2.6 = 2.26.

The hardest tier

Action a in state s earns a reward of 1 and moves the agent to state s1 with probability 0.6, where the best Q-value is 5, or to state s2 with probability 0.4, where the best Q-value is 2. The discount factor is 0.9. What is Q(s, a) when it equals the reward plus the discounted expected best next-state value?

  1. 5.50Why not: Uses the best state's value of 5, ignoring the transition probabilities.
  2. 4.80Why not: Omits the discount factor on the expected future value.
  3. 3.42Why not: Omits the immediate reward of 1.
  4. 4.42 - correct

Expected next value = 0.6 x 5 + 0.4 x 2 = 3.8; Q = 1 + 0.9 x 3.8 = 4.42.

In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.

Start free →

More in Machine-Learning Methods