Subjects › AGI Agents (6-week course)
W6 · Evaluating agents honestly
Task suites, success rates with bootstrap confidence intervals, ablations, baselines and cost.
The idea
One lucky run is an anecdote. An eval is a fixed task suite, a baseline, repeated runs, and a success rate with a confidence interval (bootstrap: resample the per-task results with replacement, recompute, take percentiles). Ablations remove one component to show what it contributes; always report LLM and tool call counts, and at least one negative result. In your repo: Week06_Integration_Capstone/AGI_Prototype/evals/eval_harness.py, tasks.jsonl, capstone_rubric.md; Exercise 6.3 (bootstrap_ci).
Lesson
No lesson built yet. The worked solutions below are the material it would be assembled from.
Questions (5)
| Question | Type | Difficulty | Drift | Accuracy |
|---|---|---|---|---|
| On a 10-task suite your agent went from 6/10 to 7/10 after a prompt change. What can you conclude? | mcq | Easy -1.30 | — | — (0) |
| Exercise 6.3. In `bootstrap_ci`, what is one bootstrap replicate? | mcq | Easy -0.60 | — | — (0) |
| Your agent solves 21 of 30 tasks. As a sanity check on your bootstrap interval, use the normal approximation: … | numeric | Moderate 0.30 | — | — (0) |
| A draft capstone report says: 'Our new router improved success from 63% to 70%.' What else must the report giv… | short | Hard 1.00 | — | — (0) |
| With 10 tasks at a success rate near 60%, the 95% interval is about ±30 points. Keeping the success rate the s… | numeric | Hard 1.40 | — | — (0) |