Subjects › AGI Agents (6-week course)
W5 · Guardrails, injection & red-teaming
Least privilege, confirmation for irreversible actions, indirect prompt injection, and measuring recall and false positives.
The idea
Treat tool output as data, never as instructions: that is a structural defence, while a regex filter is only a heuristic one. Spotlighting marks untrusted text so the model can tell it apart. Give tools the least privilege they need and require confirmation for anything irreversible. A red-team harness must report false-positive rate on benign controls as well as recall on attacks — a filter that blocks everything has perfect recall and is useless. In your repo: Week05_Alignment_Safety/safety_filters.py, red_team_eval.py, safety_tests.md; Exercises 5.2 and 5.3.
Lesson
No lesson built yet. The worked solutions below are the material it would be assembled from.
Questions (5)
| Question | Type | Difficulty | Drift | Accuracy |
|---|---|---|---|---|
| A lazy guardrail blocks every request it sees. On a red-team suite of attacks plus benign controls, what are i… | mcq | Easy -1.30 | — | — (0) |
| The shipped red-team suite has six injection cases, inj-01 to inj-06. The regex PromptInjectionFilter blocks i… | numeric | Easy -0.60 | — | — (0) |
| After extending it, your suite has 24 attacks and 8 benign controls. Your pipeline blocks 20 of the attacks an… | numeric | Moderate 0.40 | — | — (0) |
| Why is 'treat tool output as data, never as instructions' a structural defence, while the regex PromptInjectio… | short | Hard 0.90 | — | — (0) |
| Exercise 5.2. Before wrapping a tool's output in `<untrusted_data>` … `</untrusted_data>`, `detect_indirect_in… | mcq | Hard 0.70 | — | — (0) |