Subjects › AGI Agents (6-week course)
W3 · Traces, failures & self-improvement
Reading traces with the F1–F8 failure codes, Reflexion, self-consistency, and before/after evals.
The idea
Agents fail in recognisable ways: F1 hallucinated observation, F2 ungrounded answer, F3 tool misuse, F4 loop, F5 premature stop, F6 over-search, F7 goal drift, F8 injection-following. Reflexion stores a verbal lesson and retries (learning without gradients — lost when the context is dropped). Self-consistency samples several answers and votes. Every change needs a before/after number, not vibes. In your repo: Week03_ReAct_LangGraph/reasoning_trace.md, react_agent.py (run_with_reflexion, Exercise 3.1), exercises/self_consistency.py (Exercise 3.5).
Lesson
No lesson built yet. The worked solutions below are the material it would be assembled from.
Questions (5)
| Question | Type | Difficulty | Drift | Accuracy |
|---|---|---|---|---|
| Tag this trace (from an agent with no repeated-action guard) with its failure code: THOUGHT I need Aurora St… | mcq | Easy -1.30 | — | — (0) |
| Experiment C adds a poisoned KB entry. Tag this trace: QUESTION How many scientists work at Aurora Station? … | mcq | Moderate -0.20 | — | — (0) |
| Exercise 3.5: self_consistency(llm, question, n=5) samples five answers, extracts the text after the last 'Ans… | numeric | Hard 0.50 | — | — (0) |
| Reflexion (Exercise 3.1) is said to 'learn' without any gradient update. Where is what it learned actually sto… | short | Hard 1.00 | — | — (0) |
| Suppose each sampled answer to a question is independently correct with probability 0.6, and every wrong sampl… | numeric | Very hard 1.60 | — | — (0) |