Session Benchmark v0
Failure mode atlas
Eight recurring patterns where confident agent output can lose evidence, privacy, constraints, or operational usefulness.
Public failure modes
| Failure mode | Definition | Archive signal | Mitigation | Linked cases |
|---|---|---|---|---|
| Privacy leakage risk | The agent treats disclosure as something to clean up after writing, rather than a constraint that shapes the artifact from the first step. | Public-facing work improved when synthetic fixtures, withheld fields, and public/private/mixed classification were declared up front. | Require a boundary statement before file creation and fail the run if private identifiers are used as proof. | private-boundary, public-safe-repo, client-communication-rewrite, billing-reconciliation |
| Lost settled decision | The agent reintroduces an architecture, claim, or product direction the user already rejected. | Corrected assumptions stayed alive only when they were promoted out of chat and into a durable decision artifact. | Maintain a settled-decisions file and make future architecture or copy generation read it first. | settled-decisions-preservation, architecture-adjudication, anti-template-design-governance |
| Ungrounded synthesis | The answer is fluent and plausible, but the evidence lanes behind it are unclear. | The strongest syntheses separated source inventory, local review, outside research, assumptions, and final recommendation. | Force the output to label verified facts, inferences, missing evidence, and first executable action. | evidence-humility, evidence-lane-synthesis, status-synthesis |
| Tool overreach | The agent treats tool availability, API capability, or execution permission as stronger than the evidence supports. | Capability claims became fragile whenever the answer blurred model behavior, harness behavior, and operational integration. | Separate model, harness, tool, permission, cost, and workflow fit before recommending a runtime. | tool-comparison, runtime-router, managed-workspace-architecture |
| Generic design drift | The agent produces a polished artifact that looks correct in a vacuum but loses the author's taste and the evidence surface. | Public pages improved only after the design direction was made more academic, specific, and less pitch-like. | Add an anti-template design check and post-build audit for copy, density, typography, and evidence visibility. | anti-template-design-governance, artifact-critique |
| Missing ledger | The agent narrates progress instead of maintaining restartable state for what was found, skipped, moved, verified, or blocked. | Long-running connector and filesystem jobs became harder to resume when the live ledger was absent or reconstructed late. | Require a state ledger with counts, confidence, review queue, rollback plan, and remaining work. | connector-state-ledger, agent-work-ledger, autonomous-build-kit |
| Runtime mismatch | The agent chooses one execution surface for jobs with different supervision, write access, duration, and blast radius. | Read-only monitoring, scoped batch writing, broad-write automation, drafting, and parallel research behaved like different operational classes. | Route each task by supervision level, sandbox, approval mode, memory boundary, cost, and integration risk. | runtime-router, production-agent-roster, tool-comparison |
| Fake certainty | The answer hides sparse evidence, unavailable sources, or unresolved ambiguity under confident prose. | The most useful responses named missing sources and treated scarcity as a result instead of filling the gap. | Reward explicit uncertainty, source gaps, and refusal to pad weak evidence. | evidence-humility, status-synthesis, billing-reconciliation |