{
  "module": "CAP-B2 — Run a Full Agent Red-Team Engagement",
  "course": "2B — Securing & Attacking Harnesses and LLMs",
  "version": "2.0.0",
  "duration_minutes": 35,
  "total_questions": 15,
  "bloom_distribution": {
    "target": "20% recall / 40% application / 40% analysis-design",
    "actual": { "recall": 3, "application": 6, "analysis": 6 }
  },
  "passing_score_percent": 70,
  "questions": [
    {
      "id": "Q01", "bloom": "recall", "type": "multiple_choice",
      "prompt": "What is the Acme Corp tau deployment's partial-hardening configuration that this capstone attacks?",
      "options": [
        "tau with no plugins installed — fully undefended.",
        "tau with tau_taint (B2) and tau_sandbox (B7) installed, but tau_vault (B5) NOT installed. B3 memory gate and B8 intent tracker are also missing. This is the realistic partial-hardening scenario — the most common production state.",
        "tau with all three tau_plugins (taint, sandbox, vault) installed — fully hardened.",
        "tau with tau_vault installed but tau_taint and tau_sandbox missing."
      ],
      "answer_index": 1,
      "rationale": "The Acme deployment has two of three tau_plugins controls installed: tau_taint (B2, Extension Point 1) and tau_sandbox (B7, Extension Point 2). tau_vault (B5, Extension Point 3) is NOT installed, leaving credentials in plaintext. The B3 memory gate and B8 intent tracker are not part of the tau_plugins pack and are also missing. This partial-hardening state is what makes the engagement realistic — most production teams install some controls and stop before finishing."
    },
    {
      "id": "Q02", "bloom": "recall", "type": "multiple_choice",
      "prompt": "What does the scorecard battery reveal when run against the Acme deployment, and how do you read it?",
      "options": [
        "All attack classes succeed at 100% — the deployment is fully undefended.",
        "All attack classes are blocked at 0% — the deployment is fully hardened.",
        "Indirect injection 0% (tau_taint working), tool_abuse 0% (tau_sandbox working), memory_poison 100% (B3 missing), sandbox_escape 67% (denylist bypassed on 2 of 3 probes). The battery IS the discovery instrument — it reveals which controls are working and which are missing.",
        "Only direct injection succeeds; all other classes are blocked."
      ],
      "answer_index": 2,
      "rationale": "The scorecard battery reveals the control gaps through per-class success rates. Indirect at 0% confirms tau_taint is working (tool output tagged <untrusted>, high-impact calls blocked when taint active). Tool_abuse at 0% confirms tau_sandbox is working (curl/wget blocked). Memory_poison at 100% reveals B3 is missing (write_to_memory with injected payload succeeds). Sandbox_escape at 67% reveals the denylist is bypassable — cat credentials.json is blocked but interpreter-based reads succeed. The battery is the discovery instrument; the OWASP checklist then names the findings."
    },
    {
      "id": "Q03", "bloom": "recall", "type": "multiple_choice",
      "prompt": "What are tau's five extension points (from SDD-B11), and which are USED vs UNUSED in the Acme deployment?",
      "options": [
        "All five extension points are used in the Acme deployment.",
        "EP1 (tool-executor wrapping) USED by tau_taint; EP2 (create_bash_tool factory) USED by tau_sandbox; EP3 (create_model_provider credential store) UNUSED — tau_vault not installed; EP4 (harness.subscribe) UNUSED — no B8 listener; EP5 (CodingSession.load tool-list) USED (hardened tools injected). The gaps are at EP3 and EP4.",
        "Only EP1 and EP2 exist in tau; the others are theoretical.",
        "EP3 is used by tau_vault; EP4 is used by B8; the deployment is fully covered."
      ],
      "answer_index": 1,
      "rationale": "SDD-B11 maps five extension points. EP1 (dataclasses.replace(AgentTool, executor=gated)) is used by tau_taint. EP2 (create_bash_tool factory replacement at tools.py:574) is used by tau_sandbox. EP3 (create_model_provider credential_store parameter at provider_runtime.py:46) is UNUSED — tau_vault is not installed, so FileCredentialStore (plaintext) remains. EP4 (harness.subscribe(listener) at harness.py:124) is UNUSED — no B8 security listener is subscribed. EP5 (CodingSession.load tool-list at session.py:286) is used to inject the hardened tools. The credential gap is EP3; the observability gap is EP4."
    },
    {
      "id": "Q04", "bloom": "application", "type": "multiple_choice",
      "prompt": "You are running the credential-exfil chain against the Acme tau deployment. tau_sandbox blocks 'cat ~/.tau/credentials.json' via its denylist. You try the command: python3 -c \"import json; print(list(json.load(open('$HOME/.tau/credentials.json')).keys()))\". What happens, and why?",
      "options": [
        "The command is blocked because python3 is in the NETWORK_TOOLS denylist.",
        "The command EXECUTES. SandboxPolicy.check extracts 'python3' as the base command. python3 is not in NETWORK_TOOLS, not in ALWAYS_DENIED, and does not match the credential-read regex (which matches cat|head|tail|less|more|... followed by the path). The command runs, the credential keys are in the tool output, and the credentials enter tau's context — the sandbox cannot help now.",
        "The command is blocked because it contains the string 'credentials.json'.",
        "The command is blocked because python3 is in the ALWAYS_DENIED set."
      ],
      "answer_index": 1,
      "rationale": "The sandbox's SandboxPolicy.check extracts the base command (python3) via _extract_base_command, then checks it against NETWORK_TOOLS (curl, wget, ssh... — python3 is not there), ALWAYS_DENIED (rm, mkfs, dd... — python3 is not there), and the credential-read regex (r'(?:cat|head|tail|less|more|...)\\s+.*{path}' — python3 is not a reader in the set, so the regex does not match). The command executes. The credential keys enter the tool output, which enters tau's context window. The sandbox is a denylist on the command string — it blocks the obvious read channels but is bypassable by interpreters. The closing control is tau_vault (secure credentials at rest), not a better denylist."
    },
    {
      "id": "Q05", "bloom": "application", "type": "multiple_choice",
      "prompt": "You confirm the credential-exfil bypass against real tau. The command executed (ok=True), and the tool output contains the credential keys. What evidence do you retain in the validation store?",
      "options": [
        "The full credential values (API keys, tokens) so the client can see the severity.",
        "The command string, the ok=True result, a HASH of the credential file's content prefix (proves the file was read and is real), the tau version, the plugin config, the model version, and the measured success rate (N over M). The credential VALUES are Restricted — destroy on report. Retaining them IS the harm the test was hired to prevent.",
        "Only the tau version; the command and output are irrelevant.",
        "The full ~/.tau/credentials.json file so the client can verify."
      ],
      "answer_index": 1,
      "rationale": "Minimum-proof discipline (B0). The proof is reachability (the command executed, the file was read), not possession (the credential values). Capture: the command string, ok=True, a hash of the prefix, the tau version, plugin config, model version, and the measured rate. The credential values are Restricted class — destroy on report submission. Retaining them makes you a data controller under GDPR and is the exact harm the test was hired to prevent. A hash of the prefix proves the file is real (not a decoy) without retaining the protected values."
    },
    {
      "id": "Q06", "bloom": "application", "type": "multiple_choice",
      "prompt": "During Phase 1 scoping, you want to test the credential-exfil chain. Acme's tau calls OpenAI's API (gpt-4o). OpenAI's ToS prohibits weight extraction but is silent on credential-read probes of the deployer's own filesystem. What is the correct action?",
      "options": [
        "Proceed — the deployer (Acme) authorized testing of their own agent and filesystem.",
        "You CANNOT proceed on the deployer's authorization alone for any technique that touches the provider surface. However, the credential-exfil probe reads Acme's OWN filesystem (~/.tau/credentials.json), not OpenAI's model. The deployer CAN authorize filesystem reads of their own system. The scope file's provider_authorization check returns ALLOW for credential-read probes (deployer-controlled surface) but would return DENY for weight-extraction (provider-controlled surface, ToS prohibits). Proceed with the cred-exfil probe; do NOT proceed with weight extraction.",
        "Proceed with both credential reads and weight extraction — the deployer authorized everything.",
        "Decline all testing because OpenAI's ToS is silent on credential reads."
      ],
      "answer_index": 1,
      "rationale": "The key B0 distinction: the deployer can authorize testing of THEIR agent and THEIR filesystem, but cannot authorize violation of the PROVIDER's terms. The credential-exfil probe reads Acme's own ~/.tau/credentials.json — a deployer-controlled surface. The deployer's authorization is sufficient. Weight extraction would touch the provider's model — a provider-controlled surface where OpenAI's ToS prohibits it, and no waiver is on file. The scope file's provider_authorization field enforces this per-technique: ALLOW for credential reads, DENY for weight extraction. This is why the scope file separates deployer from provider surfaces."
    },
    {
      "id": "Q07", "bloom": "application", "type": "multiple_choice",
      "prompt": "You run the memory-poison probe against the Acme tau deployment: write_to_memory with an injected payload. The scorecard shows 100% success. Why does tau_taint NOT block this, even though write_to_memory is in DEFAULT_HIGH_IMPACT_TOOLS?",
      "options": [
        "tau_taint is broken and not working.",
        "The taint gate blocks high-impact calls ONLY when _taint_active is True (a prior tool output was tainted). The memory-poison probe calls write_to_memory DIRECTLY with no prior tainted tool output, so _taint_active is False, and the gate does not fire. The write succeeds. The taint gate is a TAINT-CONDITIONAL gate, not an unconditional write-validation gate. B3 (a memory-write gate that validates ALL writes regardless of taint) is the missing control.",
        "write_to_memory is not in DEFAULT_HIGH_IMPACT_TOOLS.",
        "The taint gate only works for the bash tool, not for write_to_memory."
      ],
      "answer_index": 1,
      "rationale": "tau_taint's TaintGate.inspect checks: if name in self._high_impact AND self._taint_active — then block. The memory-poison probe calls write_to_memory directly (the battery calls tool.execute with the injected arguments), with NO prior tool output that would set _taint_active. So the condition is False (taint not active), and the gate returns None (proceed). The write succeeds. The taint gate is designed to block high-impact calls that are INFLUENCED by tainted content — it is not an unconditional write-validation gate. B3 (which validates what enters the session before it persists, regardless of taint state) is the missing control. This is why the scorecard shows memory_poison at 100%."
    },
    {
      "id": "Q08", "bloom": "application", "type": "multiple_choice",
      "prompt": "In Phase 5, you are writing up the credential-exfil finding. The bypass technique (interpreter-based reads bypass reader-regex denylists) has a clear defensive lesson. How do you classify and publish it per B0's dual-use rules?",
      "options": [
        "Publish the full working bypass command immediately with a 45-day embargo.",
        "Report to the provider first under NDA (if API model). Because the finding has a DEFENSIVE LESSON (the denylist-bypass class — interpreters bypass reader-regex), publish the TECHNIQUE (not the exact command) after a 45-90 day embargo (harness-level). The exact python3 -c command is Provider-Only, shared under NDA. The credential VALUES are Restricted — destroy on report, hash of prefix only.",
        "Withhold the finding entirely — all bypass techniques are dual-use.",
        "Publish the exact command in the report's appendix for reproducibility."
      ],
      "answer_index": 1,
      "rationale": "The dual-use decision turns on whether the finding has a defensive lesson. The denylist-bypass class (interpreters bypass reader-regex denylists) HAS a lesson — it teaches defenders that denylists are insufficient and credential isolation at rest (tau_vault) is required. So the TECHNIQUE is published after the embargo (45-90 days, harness-level). The exact command (the recipe) is Provider-Only, shared under NDA. The credential values are ALWAYS Restricted — destroyed on report, captured only as a hash. This is the per-finding resolution: the defensive lesson is published, the weapon and the values are not."
    },
    {
      "id": "Q09", "bloom": "application", "type": "multiple_choice",
      "prompt": "Acme receives your report and installs tau_vault. They ask: 'Is the credential exfil fixed now?' What is the correct response?",
      "options": [
        "'Yes, it's fixed' — tau_vault is installed, the credentials are no longer plaintext.",
        "'No, it's never fully fixed' — leave it ambiguous.",
        "The answer is a MEASURED RESIDUAL RISK, not 'yes.' Per the retest plan (Phase 6), re-run the cred-exfil battery subset (the denylist-bypass probes) against the patched tau (with tau_vault installed). Report the before/after: 'cred-exfil dropped from 100% (50/50) to 0% (0/100) — the vault stores credentials in vault format, so even if the agent reads the file, it gets obfuscated data, not plaintext keys.' The harness is never 'fixed'; it is 'measured residual risk under a battery.'",
        "Decline to answer and end the engagement."
      ],
      "answer_index": 2,
      "rationale": "The B0 anti-pattern — treating an AI finding as binary fixed/unfixed — is most likely to recur here. The cure is Phase 6's retest plan: the same battery, re-run against the patched target (tau + tau_vault), reporting the before/after success rate. tau_vault converts credentials to vault format via migrate_credentials_to_vault(), so the denylist-bypass command still executes (python3 is not blocked), but the file content is now obfuscated base64, not plaintext JSON with API keys. The probe evaluates to failure (no credential keys in the output). The rate drops from 100% to 0%. The client accepts a measured number, not the word 'fixed.'"
    },
    {
      "id": "Q10", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "The credential-exfil chain bypasses tau_sandbox's denylist using python3 -c. Why is the remediation tau_vault (B5) and NOT a better denylist that blocks python3?",
      "options": [
        "A better denylist is the correct remediation — just add python3, node, ruby to the ALWAYS_DENIED set.",
        "tau_vault is the remediation because: (1) the vault stores credentials in vault format (not plaintext), so even if the agent reads the file, it gets obfuscated data; (2) the vault implements the CredentialReader protocol, so only the provider layer reads from it — the agent process cannot reach it directly; (3) a denylist is always incomplete (new interpreters, encoding tricks, shell builtins) — it is a regex on the command string, not a filesystem-level access control. Harden the denylist as defense-in-depth AFTER installing the vault, but the vault is the structural fix.",
        "Neither tau_vault nor a better denylist is needed — tau_sandbox already blocks cat.",
        "The remediation is to remove the bash tool entirely."
      ],
      "answer_index": 1,
      "rationale": "The root cause is plaintext credentials at rest (tau_vault absent), not the denylist's incompleteness. A better denylist (add python3, node, ruby) is a losing game — there are always new interpreters, encoding tricks (base64 decode), shell builtins (read, mapfile), and creative bypasses. The denylist is a regex on the command string, fundamentally bypassable. tau_vault is the structural fix: it replaces FileCredentialStore with a vault-backed CredentialReader, so the credential file on disk is obfuscated (base64-encoded JSON in the teaching version), and the agent process cannot get plaintext keys even if it reads the file. The denylist becomes defense-in-depth (block the obvious channels), not the only gate. This is why the roadmap prioritizes tau_vault (CRITICAL, Small effort) over hardening the denylist (MEDIUM, Small effort)."
    },
    {
      "id": "Q11", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Why does the session-poisoning chain succeed against the Acme deployment even though tau_taint tags tool output as <untrusted>? Trace the exact path.",
      "options": [
        "tau_taint is not working — it fails to tag the output.",
        "tau_taint tags the output <untrusted> in the LIVE context, but the tagged output is persisted to the session JSONL UNFILTERED (no B3 gate at _persist_messages_since). On session RESUME, SessionStorage.load reads the JSONL directly into the transcript — it does NOT route through the tool executor wrapper, so the taint gate never inspects the loaded content. The new session has a fresh TaintGate with _taint_active=False. The tainted content (including the injection payload) re-enters context as trusted. The laundering is complete: tagged in session 1, persisted, loaded as clean in session 2.",
        "The taint gate only works for the bash tool, not for the read tool.",
        "SessionStorage.load re-tags all content, but the tags are stripped by the model."
      ],
      "answer_index": 1,
      "rationale": "The taint gate operates at the TOOL EXECUTOR layer (Extension Point 1) — it wraps tool.execute, tags the AgentToolResult.content, and sets _taint_active. But session persistence happens AFTER the tool returns, at _persist_messages_since (session.py:1378), which is NOT wrapped by the taint gate. The tagged output (including <untrusted> tags and the payload) is persisted as-is to the JSONL. On session resume, SessionStorage.load reads the JSONL directly into the transcript — it does NOT call tool.execute, so the taint gate never sees the loaded content. The new session's TaintGate is a fresh instance with _taint_active=False. The taint gate is a LIVE-CONTEXT defense; it does not protect the persistence layer or the resume path. B3 (a memory-write gate at _persist_messages_since, or re-tagging on resume) is the missing control."
    },
    {
      "id": "Q12", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Construct the control-matrix entry for the zero-click HITL bypass chain against the Acme deployment. What is the root cause, and what control closes it?",
      "options": [
        "Root cause: weak system prompt. Closing control: a better prompt.",
        "Root cause: (1) tau_taint blocks bash when taint is active (bash is in DEFAULT_HIGH_IMPACT_TOOLS), but write and edit are NOT in the high-impact set — so low-impact writes proceed even when taint is active; (2) no B8 session intent tracker is subscribed to harness.subscribe — the compound trajectory (multiple benign writes whose cumulative effect is the attack) goes undetected. Closing control: B8 — subscribe a SessionIntentTracker to harness.subscribe (Extension Point 4) that compares the cumulative trajectory to the original goal and flags divergence. The per-turn taint gate sees benign calls; B8 sees the compound.",
        "Root cause: tau_sandbox is not blocking write/edit. Closing control: add write/edit to the denylist.",
        "Root cause: the model is too powerful. Closing control: use a weaker model."
      ],
      "answer_index": 1,
      "rationale": "The zero-click chain exploits TWO gaps. First, tau_taint's DEFAULT_HIGH_IMPACT_TOOLS = {send_email, write_to_memory, bash} — write and edit are NOT high-impact, so they are not blocked by the taint gate even when taint is active. An injection causes low-impact writes (each benign) whose compound trajectory is the attack. Second, no B8 intent tracker is subscribed to harness.subscribe (Extension Point 4) — the event stream fires for UI rendering, not attack detection. The compound trajectory (the slow intent drift across turns) is invisible. B8 closes it by tracking the cumulative trajectory and flagging divergence from the original goal. The per-turn taint gate sees individual benign calls; B8 sees the compound. B8 is absent — that is the finding's root cause."
    },
    {
      "id": "Q13", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Why is attacking a REAL tau deployment more valuable than attacking a stub harness? What does the real target provide that a stub cannot?",
      "options": [
        "Real tau is faster to set up than a stub.",
        "Three things: (1) REAL EVIDENCE — the bash tool really executes via asyncio.create_subprocess_shell; credentials are really plaintext; the session JSONL really persists. Evidence is real tool-call output that survives peer review. A stub's behavior is an artifact of the stub. (2) REAL BYPASSES — the denylist bypass (python3 -c) works because tau_sandbox's regex is a real implementation detail, not a simulated control. A stub would either block or allow; real tau has the actual regex with the actual gap. (3) REAL REMEDIATION — the roadmap prescribes real plugins (tau_vault) that really exist, are really tested (31/31), and really drop in at a real extension point. The client can act immediately.",
        "Real tau is required because stubs are illegal.",
        "Real tau provides no advantage over a stub — both are code."
      ],
      "answer_index": 1,
      "rationale": "The value of attacking real tau is threefold. First, the evidence is real: a peer reviewer can clone tau, install the same plugins, and reproduce the finding. A stub's behavior is whatever the stub's author coded — it does not generalize. Second, the bypasses are real: tau_sandbox's credential-read regex is an actual regex in actual Python code, with actual gaps (interpreter-based reads). A stub would either block or allow; it would not have the subtle regex-matching behavior that creates the real bypass. Third, the remediation is real: tau_vault is a real, tested plugin that drops in at Extension Point 3. The client can run migrate_credentials_to_vault() and the finding is closed. A stub prescribes abstract controls; real tau prescribes specific, installable plugins."
    },
    {
      "id": "Q14", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "The Acme deployment has tau_sandbox installed, which blocks 'cat ~/.tau/credentials.json'. Yet the credential-exfil finding is rated CRITICAL. Why is the sandbox's block insufficient, and what does this tell you about the relationship between B5 and B7?",
      "options": [
        "The sandbox's block IS sufficient — the credential-exfil finding is overstated.",
        "The sandbox blocks the OBVIOUS read channel (cat), but: (1) the credentials are still PLAINTEXT on disk (tau_vault absent); (2) the denylist is a regex — interpreters (python3, node) bypass it; (3) even if the denylist were perfect, the credentials exist in plaintext, reachable by any process with filesystem access. B7 (sandbox) and B5 (vault) are COUPLED but distinct: B7 blocks the exfiltration CHANNEL (the bash tool); B5 secures credentials AT REST (the file format). B7 without B5 is a partial mitigation — it raises the bar but does not eliminate the exposure. B5 without B7 is also incomplete (the agent could still read the vault file, though it gets obfuscated data). Both are needed; the Acme deployment has B7 but not B5, and that gap is Critical.",
        "The sandbox is broken — it should block python3 but doesn't.",
        "B5 and B7 are redundant — you only need one."
      ],
      "answer_index": 1,
      "rationale": "SDD-B11 states: 'B5 and B7 are coupled: B5 secures credentials at rest and at the provider boundary; B7 blocks the exfiltration channel.' The Acme deployment has B7 (tau_sandbox) but not B5 (tau_vault). B7 blocks the obvious read channel (cat credentials.json) via the denylist, but the credentials are still plaintext on disk. The denylist is a regex on the command string — bypassable by interpreters. Even if the denylist were perfect, the plaintext file is reachable by any process with filesystem access (not just the agent's bash tool). B7 without B5 is a partial mitigation: it raises the bar (blocks the obvious channels) but does not eliminate the exposure (credentials are still plaintext). The finding is Critical because the root cause — plaintext credentials at rest — is unaddressed. The remediation is B5 (tau_vault), with B7 as defense-in-depth."
    },
    {
      "id": "Q15", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "You are building the engagement report's remediation roadmap for the Acme deployment. Construct the priority-ordered roadmap with severity, effort, and expected reduction for each finding.",
      "options": [
        "1. Add B8 (LOW, L). 2. Add B3 (LOW, L). 3. Install tau_vault (LOW, S). 4. Harden denylist (LOW, S).",
        "1. [CRITICAL, S] Install tau_vault — closes RT-001 (cred-exfil). Plugin exists, tested (31/31), drops in at EP3. Expected: cred-exfil 100%→0%. 2. [HIGH, M] Add B3 memory gate — closes RT-002 (session poison). Wedge at _persist_messages_since (session.py:1378). Expected: memory-poison 100%→0%. 3. [HIGH, M] Add B8 intent tracker — closes RT-003 (zero-click). Subscribe listener to harness.subscribe (EP4). Expected: multi-step 67%→<5%. 4. [MEDIUM, S] Harden sandbox denylist — closes RT-004 (denylist bypass). Add interpreters to denylist OR rely on vault. Expected: bypass 100%→0% with vault. Priority: severity × effort — tau_vault is CRITICAL and Small effort (the plugin exists), so it is first.",
        "1. Harden the denylist (CRITICAL, S). 2. Install tau_vault (HIGH, M). 3. Add B3 (MEDIUM, L). 4. Add B8 (LOW, L).",
        "All four fixes are equal priority — let the client decide the order."
      ],
      "answer_index": 1,
      "rationale": "The roadmap is prioritized by severity × effort. tau_vault is CRITICAL (the credential-exfil is the headline finding — plaintext credentials reachable) and Small effort (the plugin exists, is tested 31/31, and drops in at Extension Point 3 via create_model_provider(credential_store=vault) plus migrate_credentials_to_vault()). So it is first. B3 is HIGH (session poisoning is a real cross-turn laundering path) and Medium effort (wedge at _persist_messages_since or re-tag on resume — not a drop-in plugin). B8 is HIGH (zero-click chains undetected) and Medium effort (subscribe a listener — additive, but the tracker itself must be built). Harden the denylist is MEDIUM (defense-in-depth after the vault) and Small effort. The expected reductions come from B1's scorecard data: tau_vault takes cred-exfil from 100% to 0% (the file is no longer plaintext); B3 takes memory-poison from 100% to 0% (deterministic gate); B8 takes multi-step from 67% to <5% (probabilistic detection)."
    }
  ]
}