Subjects › AGI Agents (6-week course)
W2 · MDPs & value functions
States, actions, rewards, discounting, the Bellman equation, value and policy iteration.
The idea
A value function answers 'how good is it to be here, if I behave well from now on?'. The Bellman equation writes that recursively: V(s) = max_a Σ P(s'|s,a)[r + γV(s')]. γ trades near rewards against far ones. Value iteration applies the backup until it stops changing; policy iteration alternates evaluate ↔ improve. In your repo: Week02_RL_DeepMind/value_function.py; Exercise 2.1 (policy_iteration).
Lesson
No lesson built yet. The worked solutions below are the material it would be assembled from.
Questions (5)
| Question | Type | Difficulty | Drift | Accuracy |
|---|---|---|---|---|
| An agent receives rewards 0, 0 and then +10 on its first three steps, and the episode ends. With γ = 0.9, what… | numeric | Easy -1.30 | — | — (0) |
| Week 2's stretch goal adds a second goal. Goal A is worth +10 and is reached on step 10 at the earliest; goal … | mcq | Easy -0.50 | — | — (0) |
| Value iteration is backing up state s with γ = 0.9. Action 'right' succeeds with probability 0.8, moving to s′… | numeric | Moderate 0.30 | — | — (0) |
| Exercise 2.1 asks why policy iteration usually needs far fewer improvement steps than value iteration needs sw… | mcq | Hard 0.90 | — | — (0) |
| Run value_function.py and compare the two printouts. V*(start) falls from −1.09 at slip = 0.0 to −4.65 at slip… | short | Very hard 1.60 | — | — (0) |