Benchmarks
How well frontier models read and maintain a Creed through its tools.
Personal context use by cost
Pass@1 vs average cost per task · Creed Bench v1.0.1
How Creed Bench works
Creed Bench measures whether a model can use Creed the way a connected agent should: through the real MCP tool surface, not by stuffing context into a prompt and hoping for the best.
Try a model with one medium-effort pass over the full suite:
npm run bench -- openai/gpt-5.6-solEach trial starts from a fresh synthetic Creed. The model gets the same 17 tools production agents get, one user request, and a live write policy. It can read, search, update, append, create, rename, recolor, reorder, delete, propose, or decide nothing durable happened and leave the file alone.
Eight tasks are single-tool, ten are realistic multi-tool workflows, and six are adversarial policy or restraint cases. Together they cover every Creed MCP tool, including when to write, when not to, formatting, maintenance, recovery from bad arguments, and locked or proposal-only sections.
Single-tool tasks check that the model can find and use the right Creed tool for a narrow request:
| Task | Tools covered |
|---|---|
| Read and apply constraints | read_creed |
| Identify the connected Creed | list_creeds |
| Inspect section structure | list_sections |
| Read one section surgically | creed_get_section |
| Search for a constraint | creed_search |
| Inspect recent activity | creed_get_recent_activity |
| Find the weakest section | creed_get_quality_report |
| Check write policy | get_write_policy |
Multi-tool and adversarial tasks check realistic maintenance workflows, plus when the model must refuse, recover, or obey proposal-only and read-only policy:
| Task | Type | Tools covered |
|---|---|---|
| Append a durable preference | Multi-tool | read_creed, creed_append_to_section |
| Correct an existing fact | Multi-tool | read_creed, creed_update_section |
| Create a justified section | Multi-tool | read_creed, creed_create_section |
| Rename a section | Multi-tool | list_sections, creed_rename_section |
| Recolor a section | Multi-tool | list_sections, creed_recolor_section |
| Reorder a section | Multi-tool | list_sections, creed_reorder_section |
| Delete confirmed obsolete context | Multi-tool | creed_search, creed_delete_section |
| Target maintenance from quality | Multi-tool | creed_get_quality_report, creed_get_section, creed_update_section |
| Locate and append context | Multi-tool | creed_search, creed_get_section, creed_append_to_section |
| Exercise legacy direct adapter | Multi-tool | get_write_policy, direct_edit_creed |
| Reject transient context | Adversarial | read_creed |
| Obey proposal-only policy | Adversarial | read_creed, get_write_policy, creed_append_to_section |
| Respect section permissions | Adversarial | read_creed, creed_update_section |
| Recover from a stale identifier | Adversarial | list_sections, creed_get_section, creed_append_to_section |
| Avoid duplicate maintenance | Adversarial | creed_get_recent_activity, creed_get_section |
| Exercise legacy proposal adapter | Adversarial | list_sections, propose_creed_update |
Inspect the same list from the CLI:
npm run bench -- --listWhen the model finishes, deterministic graders check the outcome: final Creed state, proposal versus direct mode, required contract steps, forbidden mutations, and whether the answer is grounded. Equivalent correct paths can pass. Lucky one-shot trajectories are not enough on their own.
Iterate on one task while building or debugging a grader:
npm run bench -- openai/gpt-5.6-sol --task append-durable-preferenceThe graph plots Pass@1 against average cost per task attempt. Pass@1 is whether the first trial for each task succeeds. Hover a model square in the legend for average cost, total cost, and total tokens. Official runs measure Pass^3, the share of tasks that succeed in all three repeats.
Publish a complete official result: all 24 tasks, three trials each, at low, medium, and high effort.
npm run bench -- openai/gpt-5.6-sol --official --yesEvery run keeps the full tool transcript, before and after state, verifier checks, model route, tokens, cost, benchmark version, and runner commit. Only complete official results appear in the graph above.