Subjects › AGI Agents (6-week course)
W2 · Q-learning & SARSA
Learning values from experience: TD targets, on- vs off-policy, and exploration.
The idea
Without a model you learn from sampled transitions: Q(s,a) ← Q(s,a) + α[target − Q(s,a)]. Q-learning's target uses max_a' Q(s',a') (off-policy — learns the greedy policy while behaving ε-greedily); SARSA uses the action actually taken next (on-policy — learns the policy it follows, so it is more careful near cliffs). In your repo: Week02_RL_DeepMind/q_learning_agent.py; Exercise 2.2 (sarsa_update).
Lesson
No lesson built yet. The worked solutions below are the material it would be assembled from.
Questions (5)
| Question | Type | Difficulty | Drift | Accuracy |
|---|---|---|---|---|
| A Q-learning agent uses α = 0.5 and γ = 0.95. Currently Q(s, a) = 2. It takes a, receives r = −1 and lands in … | numeric | Easy -1.30 | — | — (0) |
| A Q-learning agent steps into lava: r = −10 and done = True. What is the TD target for this transition? | mcq | Easy -0.60 | — | — (0) |
| Exercise 2.2's sarsa_update runs with α = 0.5 and γ = 0.9. Currently Q(s, a) = 0. The agent receives r = −1 an… | numeric | Moderate 0.30 | — | — (0) |
| On a grid where the shortest route runs along the edge of the lava, two agents both act ε-greedily with ε = 0.… | mcq | Hard 1.00 | — | — (0) |
| You have logs of another agent's play, as (state, action, reward, next state, next action) records, and want t… | short | Very hard 1.60 | — | — (0) |