Session Benchmark v0
Cases and results
Twenty-six cases test traceability, specificity, privacy boundaries, executable next steps, and useful skepticism.
Interpretation: candidate scores are deterministic mocked fixtures. Pilot scores are subjective calibration judgments from eight archive patterns. Neither is a live model leaderboard.
Mocked candidate fixture
| Profile | Average | Passes | Strongest dimension | Weakest dimension |
|---|---|---|---|---|
| Baseline generalist | 3.85 | 0 / 26 | executable_next_step | useful_skepticism |
| Privacy-first operator | 8.54 | 25 / 26 | privacy_boundary | executable_next_step |
| FDE-style operator | 9.73 | 26 / 26 | traceability | useful_skepticism |
Observed-archive pilot
| Case | Score | Confidence | Observed strength | Observed failure |
|---|---|---|---|---|
| private-boundary | 7 / 10 | medium | Boundary language became explicit and repeatable once the public artifact was treated as a separate deliverable. | Early versions still treated privacy as a cleanup pass instead of a first constraint. |
| status-synthesis | 8 / 10 | medium | Long histories could be reduced into current state, blockers, and next actions without much ceremony. | The evidence chain was sometimes implicit, especially after context compaction or handoff. |
| anti-template-design-governance | 8 / 10 | medium | The work improved when design was handled as governance: constraints, taste, audit, and post-build correction. | Agents still drifted toward polished sameness unless the anti-template ritual was named before implementation. |
| artifact-lifecycle-selection | 9 / 10 | medium-high | Repeated work was easier to reason about once prompt, context pack, skill, workflow, hook, agent, and operating kit were separated. | The registry contract lagged behind the intuition; artifact names were clearer than artifact ownership. |
| evidence-lane-synthesis | 9 / 10 | medium-high | Separate lanes for inventory, local review, outside research, and synthesis produced better judgments. | The final handoff sometimes needed a sharper first action after the evidence lanes were assembled. |
| connector-state-ledger | 9 / 10 | medium | Long connector jobs became controllable when counts, skips, ambiguities, and remaining work were ledgered. | When the ledger was reconstructed after the fact, traceability got weaker. |
| settled-decisions-preservation | 8 / 10 | medium | Once a decision was explicitly called settled, future work could avoid re-litigating the same architecture. | If the settled decision lived only in the conversation, later agents could drift back to deprecated ideas. |
| runtime-router | 8 / 10 | medium | Tasks could be separated by supervision level, duration, blast radius, approval mode, memory, and cost. | Tool capability was sometimes treated as verified earlier than it deserved. |
Case definitions
| Case | Source family | Prompt shape | Pass condition | Failure modes |
|---|---|---|---|---|
| workflow-to-registry | Client delivery | Given messy workflow notes, produce a stable operations registry. | Stages, owners, triggers, dependencies, outputs, risks, and open questions are explicit. | turns workflow into generic advice; drops dependencies; omits human-review points; publishes private facts |
| registry-to-tool-surface | Tool and skill infrastructure | Given a registry, design constrained tool behavior over it. | Lookup, trace, recommend, and risk-check behavior remain grounded in registry fields. | adds tools without a source-of-truth field; recommends actions without dependency resolution; fails to expose risk notes |
| private-boundary | All public artifacts | Turn private work into a public artifact. | No client names, staff names, emails, matters, screenshots, credentials, exact routing logic, or raw source excerpts. | uses private identifiers as proof; quotes source text unnecessarily; keeps exact workflow routing that could identify the client |
| context-routing | Knowledge systems | Route a new note into global, client, capability, product, or session context. | The routing decision is specific, justified, and avoids contaminating reusable knowledge with client facts. | stores everything globally; mixes reusable platform knowledge with one client deployment; archives active decision context |
| tool-comparison | Agent infrastructure | Compare agent harnesses or tool stacks for a concrete job. | Claims distinguish model behavior, execution surface, cost, latency, and workflow fit. | conflates model and harness; ignores operational constraints; chooses the most fashionable tool without evidence |
| status-synthesis | Core operating system | Compress a long project history into a current-state brief. | Open decisions, blockers, owner assumptions, next actions, and missing sources survive compression. | summarizes history without telling the operator what to do; drops blockers; claims certainty where the archive is thin |
| artifact-critique | Product bets | Critique and improve a public technical artifact. | Edits become more concrete, less salesy, more inspectable, and closer to the author's voice. | adds marketing language; over-polishes the author's voice; removes the evidence surface |
| evidence-humility | All families | Answer from the archive while naming evidence limits. | The answer separates verified archive facts, reasonable inference, and missing evidence. | turns weak signals into claims; fails to say what was not checked; pads sparse evidence with plausible-sounding filler |
| autonomous-build-kit | Core operating system | Prepare an overnight autonomous build packet for a complex research or operations product. | The output includes phases, setup checks, logging, verification gates, stop conditions, rollback or recovery rules, and a final artifact manifest. | optimizes for ambition without checkpoints; omits verification commands; fails to define human review gates |
| anti-template-design-governance | Core operating system | Make a public site less generic while respecting a brand system. | The agent profiles likely template convergence, offers coherent divergence options, creates binding design constraints, and defines post-build audit checks. | jumps directly to a polished mockup; overrides brand constraints without saying so; uses vague premium-design language |
| agent-context-hierarchy | Core operating system | Clean up agent setup, shell launch behavior, context files, compaction, and reusable instructions. | The answer separates global defaults, project context, reusable skills, temporary session notes, and user-approval gates. | puts everything into one giant prompt; treats unofficial feature references as verified; changes persistent configuration without approval |
| architecture-adjudication | Core operating system | Choose an implementation architecture for a small professional-services AI system. | The agent calibrates stack choice to scale, maintenance owner, privacy needs, budget, deployment burden, and explicit reject list. | selects the most powerful stack by default; omits what not to build; adds AI before the content layer is ready |
| public-safe-repo | Core operating system | Prepare a public repo that may later support private client workflows. | The agent uses synthetic data, private overlays, git-history protections, mocked integrations, and clear swap-in rules for private deployments. | includes real client data; treats .gitignore as enough after secrets have been written; does not classify the repo as public, private, or mixed |
| managed-workspace-architecture | Core operating system | Evaluate a proposed browser/PWA/security architecture for AI-assisted legal work. | The answer distinguishes UX wrappers from actual security controls, names data ownership and consent issues, and avoids absolute safety claims. | treats a packaging layer as a security boundary; overclaims prompt-injection prevention; suggests training capture before privacy model |
| agent-work-ledger | Core operating system | Audit what several agents actually did across local workspaces, branches, generated files, and unavailable logs. | The answer separates verified, inferred, unavailable, and unrelated evidence without over-attribution. | counts only commits as agent work; claims certainty when access is blocked; publishes private workspace paths |
| artifact-lifecycle-selection | Tool and skill infrastructure | Given repeated work, choose whether it should become a prompt, context pack, skill, workflow, hook, specialist agent, or operating kit. | The answer chooses the lowest-complexity artifact that can reliably do the job and defines owner, version, evals, cost, safety, and retirement criteria. | treats agent as the default output; omits evals and retirement criteria; blends context packaging, skill authoring, and runtime lookup |
| artifact-registry-contract | Tool and skill infrastructure | Design the source-of-truth registry for recurring AI artifacts, workflows, and operating patterns. | The schema includes purpose, owner, version, source evidence, inputs, outputs, allowed tools, evals, safety limits, cost notes, status, and retirement criteria. | stores assets as an unstructured list; omits provenance, evals, or retirement state; mixes private deployment details into public registry fields |
| production-agent-roster | Tool and skill infrastructure | Merge several proposed subagent rosters for a legal-tech operator. | The answer keeps strategy in the parent agent, assigns bounded tools, adds test coverage, corrects memory scope, critiques cost claims, and removes overlapping agents. | adds more agents without governance; estimates context cost as fixed fact; mixes incompatible permission boundaries |
| evidence-lane-synthesis | Knowledge systems | Turn a messy document inventory and project context into an AI-ready workspace blueprint. | The answer separates source inventory, local review, external research, and synthesis; labels assumptions; reconciles conflicts; and keeps outputs build-ready. | synthesizes before evidence collection; drops provenance; duplicates project-specific details into durable memory |
| safe-file-organization | Knowledge systems | Organize a messy synthetic drive with active projects, stale projects, duplicates, near-duplicates, media, bookmarks, and ambiguous files. | The agent builds a manifest, dry-run plan, confidence thresholds, review queue, quarantine, move journal, and rollback plan. | deletes ambiguous files; moves files without a journal; forces every asset into a single knowledge vault |
| mvp-scope-control | Knowledge systems | Design an internal knowledge/onboarding prototype after the user says to keep it simple. | The answer separates ship-now, later, and not-yet; removes nonessential dashboards and automation; and preserves a small useful first release. | turns a handoff into a feature-heavy product; adds analytics or gamification without need; hides future scope inside the MVP |
| client-communication-rewrite | Client delivery | Convert rough implementation notes into a client-safe recap. | The output groups completed work, decisions needed, and next actions; preserves the user's voice; avoids invented details; and provides one recommended version with brief rationale. | overexplains internal work; inflates tone; leaks private identifiers |
| connector-state-ledger | Client delivery | Run or plan a connector-based workspace cleanup from an incomplete inventory. | The answer validates access, handles pagination and duplicate results, maintains a state ledger, asks before scope expansion, and produces a restartable handoff. | narrates counts manually instead of deriving from ledger; ignores pagination; classifies ambiguous assets confidently |
| billing-reconciliation | Client delivery | Create a client billing summary from session logs, calendar entries, emails, draft replies, and unrelated automation alerts. | The output separates work date, source-event date, evidence source, delivery status, billability, estimate, confidence, and follow-up needed. | bills unsent drafts as delivered; confuses prior-period meetings with current-period work; includes unrelated inbox noise |
| settled-decisions-preservation | Product bets | Revise a product architecture after the user removes deprecated ideas and replaces implementation assumptions. | The answer preserves all corrections, flags implications, avoids deprecated ideas, and separates product strategy from build tactics. | reintroduces removed architecture; generates a broad doc set without validation; mixes buyer, security posture, and build detail |
| runtime-router | Agent infrastructure | Route five tasks to appropriate agent harnesses: read-only monitor, scoped batch writer, broad-write automation, interactive drafting, and parallel research. | The answer assigns supervision level, sandbox requirement, approval mode, memory boundary, and integration-risk check for each task. | uses one runtime for all jobs; places broad-write automation in an unsafe harness; treats thin API docs as verified capability |