Research artifact
Session Benchmark v0
A benchmark shell built from aggregate patterns across 1,299 agent sessions. It tests whether an agent can keep evidence, constraints, privacy boundaries, and useful next actions intact when the source context is messy.
Public boundary: this project publishes scrubbed aggregates, benchmark definitions, mocked candidate fixtures, subjective pilot scores, and synthetic registry data. The private source archive remains local. It does not publish raw source excerpts, source identifiers, client details, credentials, screenshots, or production routing.
- 1,299sessions in the source archive
- 869AI-relevant sessions
- 18sessions reviewed qualitatively
- 26benchmark cases
From deployment artifact to session benchmark
The project began as a synthetic registry-first deployment example. The more useful anchor was the archive itself: a long trail of sessions where recurring work becomes visible in aggregate.
The benchmark asks a narrower question than whether an agent can write a polished answer. It asks whether the agent can preserve constraints while crossing between rough notes, tools, private context, and public artifacts. The current release contains 26 cases, three mocked public-safe candidate profiles, and an eight-case observed-archive pilot used as calibration.
This is not a live leaderboard. The fixture proves the scoring harness and public artifact shape before any provider-backed candidate run is claimed.
Corpus shape
| Task family | Sessions | Public note |
|---|---|---|
| Core operating system | 674 | Identity, operating principles, reusable delivery context, and routing shells. |
| Tool and skill infrastructure | 298 | Repeatable workflows, execution surfaces, research helpers, and packaging. |
| Knowledge systems | 279 | Project memory, long-term knowledge bases, file organization, and context layers. |
| Client delivery | 258 | Scrubbed legal-operations delivery work and implementation planning. |
| Product bets | 179 | Product strategy, UX, public positioning, and experimental applications. |
| Agent infrastructure | 149 | Agent harnesses, runtime patterns, orchestration, and comparison work. |
What the qualitative review added
The 18-session proportional review added cases where generic benchmark prompts usually miss operational detail.
- Artifact choice before agent choiceRepeated work should become the smallest useful artifact: prompt, context pack, skill, workflow, hook, specialist agent, or full operating kit.
- Autonomy needs a ledgerLong-running agents need explicit state: discovered, classified, moved, skipped, ambiguous, verified, remaining, and blocked.
- Public-safe by defaultIf a repo or page may become public, the data model must be synthetic from the start. Git history is part of the privacy surface.
- Evidence before synthesisComplex answers improve when source inventory, filesystem review, outside research, and final synthesis are separate lanes.
- Settled decisions must bind future workThe agent should not reintroduce deprecated architecture after the user has corrected it.
- Runtime selection follows riskDifferent jobs need different harnesses: read-only monitoring, scoped batch writing, broad-write automation, drafting, and parallel research are not the same agent problem.
- Design is a governance problemAvoiding generic AI design requires a pre-build ritual: diagnose convergence, choose a direction, bind the design spec, and audit the result.
- Scope discipline is a quality signalA strong agent preserves ambition while shipping the smallest useful version first.
- Communication work is operational workUseful agents turn rough notes into concise client-safe recaps, decision requests, and next actions without inflating the tone.
- Billing requires reconciliationChat/session history alone is not enough for client-facing billing; calendars, emails, drafts, deliverables, and unrelated noise must be reconciled.
Publication boundary
Published: aggregate counts, scrubbed task-family labels, benchmark case definitions, mocked candidate fixture results, observed-archive pilot scores, failure-mode labels, synthetic registry fixture.
Withheld: client names, staff names, matters, emails, screenshots, credentials, private field names, raw source excerpts, production automations, exact routing logic, local source UUIDs.