Subjects › AGI Agents (6-week course)
W2 · Dyna, MCTS & the DeepMind lineage
Planning with a learned model, tree search with UCT, and how AlphaGo/MuZero combine them.
The idea
Dyna-Q replays imagined transitions from a learned model between real steps — planning as extra learning. MCTS builds a search tree, balancing exploitation and exploration with UCT, and AlphaGo mixes the random-rollout return z with a learned value estimate v: (1 − λ)·v + λ·z (Exercise 2.3). MuZero = Week 1 (learned model) + Week 2 (search + value learning). DQN is Q-learning with a network, a replay buffer and a target network. In your repo: Week02_RL_DeepMind/dyna_q.py, mcts.py, dqn_skeleton.py; Exercises 2.3–2.4.
Lesson
No lesson built yet. The worked solutions below are the material it would be assembled from.
Questions (5)
| Question | Type | Difficulty | Drift | Accuracy |
|---|---|---|---|---|
| A Dyna-Q agent with planning_steps = 10 takes one real step. Compared with plain Q-learning, what extra work d… | mcq | Easy -1.30 | — | — (0) |
| Exercise 2.3's mixed_leaf_value evaluates a leaf as (1 − λ)·v + λ·z, where v is the value estimate and z is th… | numeric | Moderate -0.30 | — | — (0) |
| In mcts.py, selection picks the child that maximises Q + c·√(ln N_parent / N_child), with c = 1.4 and Q = W / … | numeric | Moderate 0.40 | — | — (0) |
| DQN adds a target network: a copy of the Q-network used only to compute r + γ·max_a′ Q_target(s′, a′), and syn… | mcq | Hard 0.90 | — | — (0) |
| The Week 2 milestone asks: 'How is MuZero Week 1 + Week 2?' In 2–4 sentences, say what MuZero takes from each … | short | Very hard 1.60 | — | — (0) |