antiloki · technical report · fourth-edition revision 2026-10-01 · third 2026-10-01 · first 2026-09-23, second 2026-09-29
Compiled, Not Trained
Heimdall, the merge and the X-ray adapter: how antiloki builds a specialised coding model out of measured parts.
Second edition, measured 28–29 September 2026 on the five subscription seats — Claude Code, Codex, Gemini, Grok and Cursor. The first edition (22–23 September) measured Chinese models through OpenCode Zen; those arms are withdrawn and kept on disk as history. §4.2, §4.3, §7.1, §8 and §9 are rewritten; the depth runs still to come are listed in §10.
Third edition, 1 October 2026 — adds §11: the one tolerance (a seat wins if it keeps 80 % of the best measured quality, then price, then time), the solo path for asks of size L and smaller, a screening call of its own, obligatory QA, the QA judges' false-accept rates on two defect sets, the cost against a raw Claude session cell by cell (the 3× the site carried at the time of the third edition), and three X-ray experiments. It supersedes the 0.10 absolute band of §7.1, the planner and screening rows of Table 1, the “on request” QA of §5, the judge paragraph of §8.1 and the open items of §10 that are now done. Measured 2026-09-28 to 10-01; no Claude arm was re-run in this edition, and the Claude references date from 2026-09-22 to 28.
Fourth-edition revision, 1 October 2026 — the site’s headline reference is now raw Claude Opus and not the cheapest Claude arm inside the tolerance, because Opus is the Claude model a subscriber on the top plan reaches for. The headline is 10× on the eight-module task: the best measured cell, 11.3× rounded down, with a range of 6×–11× over the days Opus was measured (§11.7). The seven-cell analysis of §11.4, which gave the 3×, stays as the Sonnet-and-Haiku-reference result and as context; it is no longer the headline. The weekly-limit figure is restated as a paired task result (§11.4, §11.7).
Erratum, 2026-10-08. The cost figures in this report were measured in September with the models of that month. Re-measured on 2026-10-07 with today's models (same eight-module task, same 37 hidden checks), raw Claude Opus 5.5 costs $0.443 and 108 s a run (3 runs), not $1.151 and 278 s; raw Sonnet $0.166 (3 runs); raw Codex $0.149 (3 runs); Composer 2.5, the seat the router picks for this task, $0.065 and 72 s at 36.3 of 37 checks (12 runs); the full Sonnet-plans / Composer-writes pipeline with its QA gate about $0.26 (3 runs). So the headline figures below (10×, 7.4×, +112 %) no longer reproduce: the defensible statement today is "up to about 7× cheaper than running Opus, by routing the job to the cheapest seat that passes", and a merge is not cheaper than a cheap raw seat at this size. Costs are list-price equivalents. On the XXL task raw Opus 5.5 costs $1.83 (434 s, 1 run) and raw Sonnet 5.5 $0.96 (473 s, 1 run); the split pipeline costs $0.29–0.47 at 0.99 of the checks but takes 760–880 s (2–3 runs), so on big tasks splitting is about 4–6× cheaper than Opus and about twice as slow. The September tables are kept unchanged below as the record of what was measured then.
A coding agent's behaviour is decided by three things: which model answers each kind of question, what that model is shown about the codebase, and how the whole is judged. antiloki fixes all three without training a weight. A merge assigns every job of a coding run — planning, editing, verifying, judging, reading, reporting — to one side of a brain × hands pair; a deterministic adapter conditions each call — X-ray's facts computed from the repository, plus the skills, specialists and tools the role carries — rather than on what the model remembers; and a benchmark re-weights every combination from measured outcomes, so the assignment learns. The result behaves like a new model, with its own score, price and speed. The central result of this edition is about where the money goes. At small size the writer decides quality: every pairing writing with Cursor Composer scored 1.00 on a calibrated sixteen-case task whatever planned it, and a frontier planner only multiplied the price, by 1.6–2.7× (raw Claude Haiku, writing alone, scored 0.87 on the same checks — the grader separates writers). At large size one cheap seat planning and writing its own work stayed within the quality band of the best pairing in all six domains, at $0.08–0.12 a run. The cheap seats are also the better screeners — Gemini and Haiku caught 87.5 % of deliberately under-specified prompts, Claude Sonnet 68.8 % — and the better verifier: Composer judged every must-pass and mutant case correctly. What a merge sells is a checked result at a cheap model's price; the expensive model is paid for only where a measurement says it earns it. We give every rule, formula, price and number the system uses, the results, and where the analogy to a trained merge breaks.
1The claim, stated carefully
"Model merging" in machine learning means arithmetic on the weights of checkpoints that share a base (task vectors, TIES, DARE; mergekit is the tool). antiloki does none of that: the models are frozen, hosted by their vendors, reached through the vendors' own CLIs on your own subscription. What antiloki merges is behaviour: the answer to "which model does which job, shown what, judged how" — and that answer is learned from outcomes.
The analogy we defend is narrower and, we think, exact in practice. A LoRA adapter specialises a frozen model for a task by adding a small learned component in front of it. X-ray is a small deterministic component in front of every call that supplies what a coding fine-tune would otherwise have to learn — the structure, laws, risks and history of this repository — and it is recomputed at each commit instead of going stale. The merge table plays the role of the training loop: an outcome-weighted selection over roles, updated by every benchmark row and every audited use. The difference from a trained merge is the ceiling: no role can exceed what its base model can do when shown the right evidence. For code, where the missing knowledge is almost always the codebase and not the craft, that ceiling has not been the binding constraint in any measurement below.
the merge
brain × hands. The brain plans, verifies, judges, reads and screens; the hands edit, write tests and report. Each side is one engine and, for OpenCode, one model.
the adapter
X-ray: deterministic code intelligence — reviews scored axis by axis, the laws, the graph, blast radius, coverage gaps — packed into every call's context.
the training loop
the benchmark: every run and every audited call becomes a row per job × size × planner × worker × QA, re-weighted by quality, cost and time.
2Related work
Query routing between a strong and a weak model is well studied. RouteLLM (LMSYS, 2024) trains routers on Chatbot Arena preferences and reports cost reductions of over 85% on MT Bench at 95% of GPT-4's quality [1]. Commercial routers — Martian [2], Not Diamond [3], and the OpenRouter aggregator now processing 25 trillion tokens a week [4] — sell the same lever per request. Heimdall differs in what it routes over: not a request, but a job inside a run, with the job's own acceptance and its own measured winner.
The brain × hands split is aider's architect/editor mode [5]: a reasoning model describes the change, a cheaper model writes the edit. aider reported a new state of the art on its polyglot benchmark with DeepSeek R1 as architect and Claude Sonnet as editor at 14× less cost than the previous o1 result [6]. Cline and Roo Code let users bind different models to Plan and Act modes by hand [7]. antiloki generalises the split from two roles to ten jobs and makes the binding measured rather than chosen.
That a composition of models can behave as a new, better model was shown by Mixture-of-Agents (Together, 2024): layered open models reached 65.1% on AlpacaEval 2.0 against GPT-4o's 57.5% [8]. The "compound AI systems" position from Berkeley [9] argues the unit of capability is now the system. We take both as the frame: the merged agent is the model; its card is its benchmark row.
3The jobs
Every call antiloki makes on a coding run belongs to one of ten jobs. A job has a side (brain or hands), a prompt template, and — the point of the system — its own winner and fallback per engine and globally.
| job | side | what the call does | fires when |
|---|---|---|---|
| planner | brain | screens the ask and writes the plan: tasks that own disjoint files, each with acceptance lines | a goal of two or more enumerated tasks or over 600 characters; an ask of size L or smaller skips it (§11.2) |
| screening | brain | the blocking/assumable questions before a plan | before the planner, as a call of its own on the best screening seat (§11.2) |
| editor-s | hands | a change to one file, an ask under 400 characters | a task that owns one file |
| editor-l | hands | a change across files or a long ask | every other task |
| verifier | brain | checks a task's diff against its acceptance; may send it back once | after every task |
| test-writer | hands | writes the tests a contract's acceptance lines imply | blueprint mode |
| qa-gate | brain | checks each acceptance claim against the final diff; the isolated judge | every run — QA is obligatory (§11.2) |
| report | hands | consolidates the tasks' reports into one | a multi-task run (a single task's report stands) |
| lookup | brain | a factual question about the code: where, which, who | a chat turn the classifier reads as a lookup |
| synthesis | brain | why, how, compare, in which order — an answer that reasons | a chat turn the classifier reads as synthesis |
The classifier is deterministic (cli::classify_ask): a question mark or an interrogative first word makes a read; among reads, the words why, how, explain, describe, compare, difference, trade-off, should, in which order make it synthesis, else lookup. A write that names at most one file and is under 400 characters is editor-s, else editor-l. Sizes for the benchmark follow the plan, not the ask: two or more tasks is XXL; one task owning one file with a goal under 400 characters is S; else L.
4Heimdall, the router
4.1 Resolving a binding
For a job on a run, Heimdall produces a binding (engine, model) and a fallback, in this order:
- The merge. If the run carries a pair, the job's side decides the engine and, when the side names one, the model.
job_binding_paired. - The learned table. For a solo run, if the merge table holds a row for this job and size with at least three samples, its best row outranks the static table.
job_binding_learned,merges::best_for(job, size, 3). - The static table. The winner and fallback measured for the job (Table 2), taken globally if the winner's CLI is installed (
which), else the run engine's own cell.job_binding_any. - The effort dial. CHEAPER moves every role one rung down its engine's price ladder; SMARTER one rung up; BALANCED leaves the bound models. AUTO lets steps 1–3 stand; ON obeys the dial; OFF routes nothing.
- The price rule. A worker is never dearer than its planner:
worker_under_plannerwalks the worker down its ladder until its list price is under the planner's, and logs why. - Failure. A binding whose engine fails is retried once on the fallback, at the fill for workers and in
run_jobfor every other job.
| job | winner | fallback | the measurement behind it |
|---|---|---|---|
| lookup | opencode · deepseek-v4-pro | claude-code · haiku | readers /50: Sonnet 42, Pro 41 at $0.13, Qwen 38, Haiku 36 (strong on lookup, fabricates on synthesis) |
| synthesis | claude-code · sonnet | opencode · qwen3.6-plus | synthesis /20: Sonnet 16, Qwen 14, Pro 13 — outside the band, Sonnet keeps it |
| screening | opencode · deepseek-v4-flash | claude-code · haiku | part of the planner call; cheap by design |
| planner | opencode · deepseek-v4-pro | claude-code · sonnet | the planner's model did not move L quality (Opus 37, 36 · Sonnet 35, 36); Pro is within the band and 5× cheaper |
| editor-s | opencode · qwen3.6-plus | claude-code · sonnet | S ×5 raw: Qwen 5/5 · 34 s · $0.011; Sonnet 5/5 · $0.093; Pro 4/5; Flash 2/5 |
| editor-l | opencode · deepseek-v4-flash | opencode · deepseek-v4-pro | L one-worker: Flash under a planner 36–37/37 at $0.02 of worker cost |
| verifier | claude-code · sonnet | opencode · deepseek-v4-pro | both L rejections it made were real (missing JSDoc; a wrong money format) |
| test-writer | claude-code · sonnet | opencode · deepseek-v4-pro | blueprint runs — being measured |
| qa-gate | claude-code · opus | opencode · deepseek-v4-pro | the judge is the one place the dearest model is bound by default |
| report | opencode · deepseek-v4-flash | claude-code · haiku | a consolidation; cheap by design |
The three-point rule. Our rule for every table above: when two candidates are within three points of each other on the job's score, the cheaper one is the winner. Three points of fifty is 0.06 on the unit scale, and that number reappears as the band in §7.
4.2 Prices and the deterministic cost
Every cost antiloki reports is computed, never read from a vendor's estimate: input and output tokens times the model's list price per million, with cache reads at one tenth of input and cache writes at 1.25× (cli::price_tokens). The same table gives the ladder each engine's effort dial walks.
| model | input $ | output $ | model | input $ | output $ |
|---|---|---|---|---|---|
| claude opus | 5.00 | 25.00 | gemini pro | 1.25 | 10.00 |
| claude sonnet | 3.00 | 15.00 | gemini flash | 0.14 | 0.28 |
| claude haiku | 1.00 | 5.00 | grok 4.6 | 3.00 | 15.00 |
| codex / gpt-5 | 1.25 | 10.00 | grok code fast | 0.20 | 1.50 |
| cursor composer 2.5 | 0.50 | 1.50 |
Each call is also timed on the serve's own clock and its tokens counted from the engine's stream; the triple (cost, ms, tokens) lands on the pane's timeline as an audit line and in the ai_calls table, and — through merges::record_use — as a benchmark row of size call.
What "saved" is measured against (since 2026-09-28): the same tokens priced at the frontier model's list rate — Claude Opus, or Codex's frontier model on a Codex seat (cli::frontier_of) — against what the work cost at the seats that actually ran it, each task priced at its own engine and model. Before that the yardstick was the seat's own top rung, which for four of the five seats is the same rate as its cheap rung and printed "saved $0.00" on work five times cheaper than Opus. A row nobody priced is priced from list rates (merges::estimated_usd); a replayed plan is never counted as a free planner.
4.3 Modes and the merge in Studio
A merged agent in Studio is a recipe with a pair: a name, a brain side, a hands side, a theme, a scope, its specialists and skills. Since 2026-09-27 every side is one of the five subscription seats — Claude Code, Codex, Gemini, Grok and Cursor; the OpenCode/Zen pairs of the first edition (Claude × DeepSeek, Claude × Kimi, Codex × DeepSeek …) are withdrawn and kept on disk only as history. A seat may sit on both sides (Composer plans and writes), which is measured as a pairing like any other. Heimdall's chip on a pane shows the mode and the money the router saved on that pane, computed as API-equivalent dollars against one high-tier session doing the same tokens.
5The pipeline a goal runs through
A goal run is the unit the benchmark measures. Its stages, and the model that answers each, are what the merge assigns.
The projection. Before a plan fans out, antiloki estimates whether a wave beats one worker. A single session doing k parts is projected at SERIAL_FACTOR = 1.5 times the parallel per-part time; the wave's concurrency is bounded by the depth of the plan's dependency chain (effective_concurrency = ceil(fills / chain), clamped to the configured cap); and a plan whose single-session projection is under the parallel floor — 900 s by default, ANTILOKI_PARALLEL_FLOOR_S, 0 = off — collapses to one worker. With no history the medians assumed are 120 s per fill and 150 s of orchestration. This is why every S and L row below is a one-worker run: the wave only pays above fifteen minutes of projected serial work.
The verifier's one send-back. A task the verifier rejects goes back to its worker once with the rejection as the brief; a second rejection fails the task. Both rejections observed on L were real defects a compile gate would not catch.
The QA gate. An isolated judge checks each acceptance claim against the final diff and answers met / unmet / unproven; on a one-worker run the always-branch kept a placeholder claim, so the gate reported unproven. That gap is closed in the third edition: QA is obligatory, runs the deterministic checks (§11.2), sends unmet claims back to their workers a bounded number of times and, when those are spent, turns the run into a question for the person.
6The adapter
A LoRA specialises a frozen model by adding a small learned component in front of it. antiloki's adapter is the whole conditioning layer around frozen models, and it is larger than the router: which model answers each job, the brain × hands split, the skills materialised into the agent's box, the specialists it may consult, the tools it may drive, and the deterministic intel it boots with. A Designer is not a different model from a Backender — it is a different adapter.
the models
a merge binds every job to a side: the brain plans, verifies, judges, reads; the hands edit, test, report. Per job, per domain, measured.
what it knows
skills materialised into its box, specialists it may consult, and the X-ray packs it reads before it explores.
what it drives
the panes it may open — Frames and the design system for a Designer, X-ray and the constitution for a Reviewer, the runner for a Tester.
6.1 X-ray — the deterministic half
X-ray is antiloki's analysis of the repository. Nothing in it is generated by a model; all of it is computed from the files, the ASTs, the tests and the git history, and it is what every call is shown before it explores.
- reviews
- every file scored, and why not 100 axis by axis — coupling, structure, cohesion, defects, tests, change — with the comment at the line that carries the move.
- laws
- the rules the codebase lives by, checked at the line (for TypeScript: no
any, novar, no loose equality, nodebugger, no@ts-ignore, no swallowed errors), with what compliance is worth. - graph
- every file a node, imports the edges: hubs, orphans, cycles, the most central and the most risky, per folder.
- blast radius
- what a change to a file would touch, before it lands.
- coverage gaps
- what the tests never reach, by file; the score's history over snapshots.
- context pack
- the bundle a pane boots with:
laws · folder · file · graph · tests · blast · docitems, derived from the pane's scope by default, under a character budget (200 000). - the retriever
- for a chat turn,
chat_auto::compose_forpicks from the workspace's intel and prior runs — the chat's own log excluded — under an 8 000-character cap. - MIN mode
- exploration off: the agent navigates by the pack instead of reading the tree, and touches the disk only to edit inside its scope.
This is why the analogy in §1 holds for code specifically. What a repository-specific fine-tune would teach — where things are, what the conventions are, which files are dangerous — is the content of the pack, and the pack is true at the commit the agent is working on.
6.2 The role — the adapter as a unit a person hires
A job is a stage inside one run; nobody hires a verifier. A role is what a person hires — a domain of the codebase — and inside it every one of the ten jobs still happens. So a role is not an eleventh job; it is a profile over all ten, scoped to a domain, carrying its own adapter.
| role | domain | consults | drives |
|---|---|---|---|
| Backender — Database, API under it | backend | correctness · errors, types · architecture · dependencies · security · injection | code graph, code, diff |
| Frontender — Designer, Architect under it | frontend | framework · idioms · style · suppressions | components, code, diff |
| Frontend designer | design | category · style · axis · cohesion | Frames, design system, components, canvas |
| Tester | tests | testing · isolation · axis · tests | run, X-ray, diff |
| Reviewer | mixed | axis · defects, compliance, change · book · warden | X-ray, constitution, diff — no editor |
| Product owner | docs | book · definer, questioneer, synthesizer | intent, todo, kanban, context |
| Infra | infra | security · secrets · book · warden | terminal, run, git graph |
The domain is derived from the paths a task owns, deterministically (roles::domain_of): a test of the
API is test work; the styles folder beats the extension; a change touching six backend files and one stylesheet is
still backend, and an even split is honestly mixed. So every run tags itself, and runs that happened
before the dimension existed can be attributed without re-running them.
A role asks the table for the merge that won its kind of work. When its domain has no samples it reaches in order — its nearest neighbour (design looks at frontend; infra and docs at backend), then the shared pool, then everything pooled — so a role always has an answer and the sample count says how far it had to reach.
6.3 Three of each
The same rows, three questions. Cheaper counts the bill above the score (0.35 / 0.55 / 0.10); Best is the table's own weight (0.70 / 0.20 / 0.10); Smarter counts only quality and allows no gap at all below the best. Each returns a different merge, and the three-point band still holds for the first two: a row ten points behind cannot be bought back at any price. They are the same three stops as the effort dial on the pane.
What "training" a merge means here. No weights change, so a new pair — Codex × Gemini, say — cannot be fine-tuned. Three things do adapt: it inherits its sides' measured priors rather than starting blank; it learns from its own use, since every audited call and run files a row and the learned table overrides the static one at three samples in its domain; and the handoff between the sides — the brief the brain writes, how much diff the verifier is shown, the acceptance lines it is judged on — is a set of deterministic parameters that can be measured and tuned per pair. That last one is prompt-level adaptation with frozen weights: the only honest sense of training available, and the one this system has not yet done.
7The benchmark, and how it learns
The merge table (~/.antiloki/merges.sqlite, merge_scores) has one row per job × size × planner × worker × qa. Sizes are s · l · xxl for runs and call for audited single calls. A row accumulates:
where audit is the external behavioural score a bench posts after the run (L: 37 checks of the eight modules; S: a five-case check of the function the ask names) and source says who measured it: bench (antiloki's own runs, shipped as the seed), run (the user's goal runs), use (audited calls). A user's table is therefore the shipped base plus their own rows, weighted together; nothing is collected back.
7.1 The weight
Every record re-weights all rows of that job and size against each other (merges::reweigh, merges::rank):
Price first, speed last, quality as a gate. Third edition: the gate is now relative — a seat or pair wins if it keeps at least 80 % of the best measured quality (Objective::band() = 0.20), and measured quality outranks claimed; there is no second tolerance anywhere in the product (§11.1). The band disqualifies a materially worse answer outright; inside it, the bill decides. The first edition weighted 0.7 / 0.2 / 0.1 with a 0.06 band, which was narrower than the instrument — a ten-instance bench resolves quality in steps of 0.10 — and would pay 6.7× for one instance in ten. Speed was reduced from 0.30 to 0.10 on 2026-09-28, after the fastest frontier planner won a simple frontend task that a model a tenth of its price had measured the same on. A pairing measured on several suites is pooled by samples, never overwritten; a row nobody priced is priced from list rates; a planner row is never cheaper than its own list price.
7.2 Winners
A job's winner is its best sized row; a job the benches never sized falls back to its best audited-call row with at least two samples; else it has no winner yet, and the Studio says so. Each winner is named by composition and role — the brain's brand and the hands' suffix as one word, then the job's title: CodexComposer Builder, Composer Tinker, ClaudeComposer Architect — and can be made an agent in one click, carrying its pair. For solo runs the learned table outranks Table 2 once a row has three samples, which is how the static table is meant to retire.
8Results
Measured 28–29 September 2026 on the five subscription seats, one machine, a pinned fixture (ministore at 3de696c), every arm in a fresh clone with its own plan (plan replay cleared per arm). Quality is the deterministic grader's score for that task — calibrated before any arm ran, never a model's opinion; dollars are the API-equivalent of every call the serve made for the run (planner, writer, verifier, QA), priced from the tokens each seat reported (§4.2). The tables are generated from the record by bench/campaign/make-results.py; rows under three samples are marked and never ranked.
| planner × writer | n | quality | $/run | s/run | priced |
|---|---|---|---|---|---|
| Codex medium × Composer 2.5 * | 2 | 1.00 | 0.1299 | 256 | before QA · 09-26/28 |
| Gemini × Composer 2.5 * | 2 | 1.00 | 0.1384 | 132 | before QA · 09-26/28 |
| Claude Sonnet × Composer 2.5 * | 2 | 1.00 | 0.1500 | 118 | before QA · 09-26/28 |
| Composer 2.5 × Composer 2.5 | 3 | 0.97 | 0.1017 | 95 | with QA · 09-29 |
* fewer than three samples — shown, never ranked. Prices from different columns are not comparable: the later ones include the QA check every run now pays.
| planner × writer | n | quality | $/run | s/run | priced |
|---|---|---|---|---|---|
| Codex terra × Gemini Flash * | 2 | 1.00 | 0.0363 | 271 | before QA · 09-26/28 |
| — none (raw) × Grok 4.6 medium * | 2 | 1.00 | 0.0837 | 260 | before QA · 09-26/28 |
| Composer 2.5 × Composer 2.5 | 3 | 1.00 | 0.1133 | 157 | with QA · 09-29 |
| Claude Sonnet × Claude Haiku * | 2 | 0.98 | — | 144 | before QA · 09-26/28 |
| Grok 4.6 high × Composer 2.5 * | 2 | 0.98 | 0.0377 | 156 | before QA · 09-26/28 |
| Codex terra × Composer 2.5 * | 2 | 0.98 | 0.0378 | 168 | before QA · 09-26/28 |
| Claude Opus × Grok 4.6 medium * | 2 | 0.98 | 0.2766 | 147 | before QA · 09-26/28 |
| — none (raw) × Composer 2.5 * | 2 | 0.93 | 0.0319 | 107 | before QA · 09-26/28 |
| Claude Sonnet × Composer 2.5 * | 2 | 0.80 | 0.0318 | 137 | before QA · 09-26/28 |
* fewer than three samples — shown, never ranked. Prices from different columns are not comparable: the later ones include the QA check every run now pays.
| planner × writer | n | quality | $/run | s/run | priced |
|---|---|---|---|---|---|
| Codex terra × Gemini Flash * | 2 | 1.00 | 0.0143 | 103 | before QA · 09-26/28 |
| Codex terra × Composer 2.5 * | 2 | 1.00 | 0.0157 | 46 | before QA · 09-26/28 |
| Grok 4.6 high × Composer 2.5 * | 2 | 1.00 | 0.0158 | 87 | before QA · 09-26/28 |
| — none (raw) × Composer 2.5 * | 2 | 1.00 | 0.0167 | 37 | before QA · 09-26/28 |
| Claude Sonnet × Composer 2.5 * | 2 | 1.00 | 0.0211 | 74 | before QA · 09-26/28 |
| — none (raw) × Grok 4.6 medium * | 2 | 1.00 | 0.0264 | 78 | before QA · 09-26/28 |
| Claude Sonnet × Claude Haiku * | 2 | 1.00 | 0.0740 | 95 | before QA · 09-26/28 |
| Composer 2.5 × Composer 2.5 | 3 | 1.00 | 0.1173 | 93 | with QA · 09-29 |
| Claude Opus × Grok 4.6 medium * | 2 | 1.00 | 0.1248 | 57 | before QA · 09-26/28 |
* fewer than three samples — shown, never ranked. Prices from different columns are not comparable: the later ones include the QA check every run now pays.
| planner × writer | n | quality | $/run | s/run | priced |
|---|---|---|---|---|---|
| Codex terra × Gemini Flash * | 2 | 1.00 | 0.0209 | 57 | before QA · 09-26/28 |
| Codex terra × Composer 2.5 * | 2 | 0.98 | 0.0131 | 42 | before QA · 09-26/28 |
| Claude Opus × Grok 4.6 medium * | 2 | 0.98 | 0.1022 | 46 | before QA · 09-26/28 |
| Grok 4.6 high × Composer 2.5 * | 2 | 0.96 | 0.0124 | 54 | before QA · 09-26/28 |
| Claude Sonnet × Composer 2.5 * | 2 | 0.96 | 0.0135 | 55 | before QA · 09-26/28 |
| — none (raw) × Grok 4.6 medium * | 2 | 0.96 | 0.0305 | 65 | before QA · 09-26/28 |
| Claude Sonnet × Claude Haiku * | 2 | 0.96 | 0.0570 | 105 | before QA · 09-26/28 |
| Composer 2.5 × Composer 2.5 | 3 | 0.96 | 0.0810 | 83 | with QA · 09-29 |
| — none (raw) × Composer 2.5 * | 2 | 0.94 | 0.0112 | 27 | before QA · 09-26/28 |
* fewer than three samples — shown, never ranked. Prices from different columns are not comparable: the later ones include the QA check every run now pays.
| planner × writer | n | quality | $/run | s/run | priced |
|---|---|---|---|---|---|
| — none (raw) × Composer 2.5 * | 2 | 1.00 | 0.0070 | 47 | before QA · 09-26/28 |
| Claude Sonnet × Composer 2.5 * | 2 | 1.00 | 0.0076 | 44 | before QA · 09-26/28 |
| Codex terra × Composer 2.5 * | 2 | 1.00 | 0.0076 | 36 | before QA · 09-26/28 |
| Grok 4.6 high × Composer 2.5 * | 2 | 1.00 | 0.0081 | 51 | before QA · 09-26/28 |
| — none (raw) × Grok 4.6 medium * | 2 | 1.00 | 0.0129 | 32 | before QA · 09-26/28 |
| Codex terra × Gemini Flash * | 2 | 1.00 | 0.0206 | 50 | before QA · 09-26/28 |
| Claude Sonnet × Claude Haiku * | 2 | 1.00 | 0.0500 | 79 | before QA · 09-26/28 |
| Claude Opus × Grok 4.6 medium * | 2 | 1.00 | 0.0606 | 43 | before QA · 09-26/28 |
| Composer 2.5 × Composer 2.5 | 3 | 1.00 | 0.0800 | 68 | with QA · 09-29 |
* fewer than three samples — shown, never ranked. Prices from different columns are not comparable: the later ones include the QA check every run now pays.
| planner × writer | n | quality | $/run | s/run | priced |
|---|---|---|---|---|---|
| Claude Sonnet × Claude Haiku * | 2 | 0.94 | 0.0580 | 75 | before QA · 09-26/28 |
| — none (raw) × Composer 2.5 * | 2 | 0.88 | 0.0122 | 37 | before QA · 09-26/28 |
| Codex terra × Composer 2.5 * | 2 | 0.88 | 0.0136 | 49 | before QA · 09-26/28 |
| — none (raw) × Grok 4.6 medium * | 2 | 0.88 | 0.0180 | 50 | before QA · 09-26/28 |
| Grok 4.6 high × Composer 2.5 * | 2 | 0.88 | 0.0189 | 100 | before QA · 09-26/28 |
| Codex terra × Gemini Flash * | 2 | 0.88 | 0.0248 | 85 | before QA · 09-26/28 |
| Composer 2.5 × Composer 2.5 | 3 | 0.88 | 0.0830 | 105 | with QA · 09-29 |
| Claude Sonnet × Composer 2.5 * | 2 | 0.88 | 0.1030 | 63 | before QA · 09-26/28 |
| Claude Opus × Grok 4.6 medium * | 2 | 0.88 | 0.2668 | 71 | before QA · 09-26/28 |
* fewer than three samples — shown, never ranked. Prices from different columns are not comparable: the later ones include the QA check every run now pays.
| planner × writer | n | quality | $/run | s/run |
|---|---|---|---|---|
| — none (raw) × Gemini Flash | 4 | 1.00 | 0.0445 | 230 |
| — none (raw) × Composer 2.5 | 4 | 1.00 | 0.0488 | 25 |
| Composer 2.5 × Composer 2.5 | 8 | 1.00 | 0.0860 | 66 |
| Codex terra × Composer 2.5 | 8 | 1.00 | 0.1260 | 54 |
| Claude Sonnet × Composer 2.5 | 16 | 1.00 | 0.1400 | 48 |
| Grok 4.6 high × Composer 2.5 | 8 | 1.00 | 0.2320 | 104 |
| — none (raw) × Grok 4.6 medium | 4 | 1.00 | 0.2489 | 30 |
| — none (raw) × Claude Haiku | 4 | 0.87 | 0.0437 | 25 |
Calibrated before any arm ran: a correct reference 16/16, rounding the discount 14/16, case-sensitive codes 14/16, naive floats 7/16, no change 0/16.
| seat | screens | balanced accuracy | $/screen | s/screen |
|---|---|---|---|---|
| Claude Haiku | 36 | 0.94 | 0.0142 | 9 |
| Gemini | 36 | 0.94 | 0.0061 | 25 |
| Codex low | 36 | 0.91 | 0.0188 | 4 |
| Grok 4.6 medium | 35 | 0.87 | 0.0590 | 21 |
| Composer 2.5 | 34 | 0.86 | 0.0145 | 15 |
| Claude Sonnet | 36 | 0.84 | 0.0412 | 5 |
Balanced accuracy = (recall on damaged prompts + precision on clear ones) / 2. The per-seat recall and precision are in bench/screen-solo/out.
| job · domain · seat | cases | accuracy | $/call |
|---|---|---|---|
| qa-gate · backend · Composer 2.5 | 6 | 0.83 | 0.0457 |
| qa-gate · backend · Claude Haiku | 5 | 0.80 | 0.0461 |
| qa-gate · backend · Codex low | 6 | 0.50 | 0.0899 |
| qa-gate · docs · Codex low | 6 | 0.67 | 0.0823 |
| qa-gate · docs · Composer 2.5 | 6 | 0.67 | 0.1483 |
| qa-gate · infra · Claude Haiku | 4 | 1.00 | 0.0296 |
| qa-gate · infra · Composer 2.5 | 6 | 1.00 | 0.0474 |
| qa-gate · infra · Codex low | 6 | 0.50 | 0.0765 |
| verifier · backend · Composer 2.5 | 6 | 1.00 | 0.0140 |
| verifier · backend · Codex low | 6 | 0.67 | 0.0340 |
| verifier · backend · Claude Haiku | 6 | 0.50 | 0.0455 |
| verifier · docs · Codex low | 6 | 1.00 | 0.0266 |
| verifier · docs · Composer 2.5 | 6 | 1.00 | 0.0110 |
| verifier · docs · Claude Haiku | 6 | 0.67 | 0.0389 |
| verifier · infra · Claude Haiku | 6 | 1.00 | 0.0283 |
| verifier · infra · Codex low | 6 | 1.00 | 0.0238 |
| verifier · infra · Composer 2.5 | 6 | 1.00 | 0.0093 |
Cases: 3 must-pass artefacts and 3 mutants per domain (files the ask names cut in half, re-graded below 0.90). Frontend, design and mixed have no honest must-fail case yet — see §10.
| planner × writer | n | resolved | $/run | s/run |
|---|---|---|---|---|
| — none (raw) × Claude Sonnet | 10 | 0.90 | 0.3737 | 111 |
| Claude Sonnet × Composer 2.5 | 10 | 0.80 | 0.1187 | 161 |
| — none (raw) × Composer 2.5 | 14 | 0.79 | 0.2018 | 85 |
8.1 What the numbers say
- At S, the writer decides and the planner buys nothing. Every pairing writing with Composer scored 1.00 on S2 whatever planned it — Claude Sonnet, Codex, Grok, or Composer itself — so the cell ranks on price: Composer planning and writing alone costs $0.086 a run with its QA check, Sonnet × Composer $0.140. The grader is not blind: raw Haiku scored 0.87 on the same checks (raw Composer 1.00, $0.049, with no pipeline and no QA).
- At L, one seat planning and writing is within the band everywhere. Composer × Composer scored 0.88–1.00 across the six domains at $0.080–$0.117 a run, inside 0.10 of the best pairing in each. The "writer plans alone" hypothesis holds; whether a frontier brain earns its price at L is the question the deeper runs (§10) are for.
- Screening is a cheap seat's job, done separately. One screening call per ask on Gemini or Haiku reached 0.94 balanced accuracy — it caught 87.5 % of the damaged prompts and left every clear one alone — for $0.006 and $0.014 a screen; Claude Sonnet caught 68.8 % of the damaged prompts at $0.041. The screening built into today's planner call caught 43.8 % — HumanEvalComm reports models asking on under 37 %.
- Which seat should judge. (Second edition. Superseded by §11.3, which scores the judges on false accepts over a harder defect set.) On must-pass artefacts and re-graded mutants of the same work, qa-gate · Claude Haiku 89 % over 9 cases in 2 domain(s); qa-gate · Composer 83 % over 18 cases in 3 domain(s); qa-gate · Codex low 56 % over 18 cases in 3 domain(s); verifier · Composer 100 % over 18 cases in 3 domain(s); verifier · Codex low 89 % over 18 cases in 3 domain(s); verifier · Claude Haiku 72 % over 18 cases in 3 domain(s). Composer is the verifier to bind: right on every case, the cheapest. As the QA judge no seat is clean — Codex rejects correct work (every one of its errors a false reject), Haiku lets truncated TypeScript through as a verifier and answers the QA prompt in a shape that cannot always be read. A false accept lets a broken change through; the per-case log separates the two error kinds.
- A frontier planner moved the price, not the quality, where it was measured on equal footing. On S2, Sonnet planning cost 1.6× and Grok planning 2.7× what Composer planning for itself did, for the same 1.00. At L the pairings' rows predate the QA-inclusive pricing, so that comparison waits for the deeper runs (§10).
9Limitations and threats to validity
- n is small, and most comparisons are ties. An L cell holds two or three runs per pairing per domain; S2 holds eight. Inside the 0.10 quality band the ranking is decided by price, so in most cells the honest statement is "the cheapest of the pairings that did equally well", not "the best model". §8 marks every row under three samples. The depth that would turn ties into rankings (n ≥ 20 per routed cell) is scheduled, not done; real use now feeds the record continuously (below), which is where most of that depth is expected to come from.
- One repository. Every number is ministore, a small TypeScript store. The merge's advantage on a large, old codebase is the claim we most want to test and have not.
- Where the tie is, the writer decides. On S2 every pairing writing with Composer scored 1.00 whatever planned it, and raw Composer alone did too; the grader still separates writers (raw Haiku 0.87). A pairing's brain buys nothing measurable at this size, which is why the cheapest configuration wins there.
- Two recording faults, found and fixed on 2026-09-29, bound the record written the day before. (1) Once QA became obligatory, the QA gate's own record counted every run twice; bench rows were re-copied from their run counters, real-use rows from that window keep the inflated count. (2) A guard read Antigravity's
defaultmodel as a third-party one and silently re-seated a Gemini planner on Claude Sonnet; those runs are filed as Sonnet (which is what planned them), Gemini as an S planner is unmeasured, and a bench arm whose pair would be re-seated is now refused instead. - Grading. Every bench row is graded by an executed check written before any arm ran and calibrated at both ends (a correct reference scores 100 %, a no-op 0 %); no row's quality is a model's opinion. The judges (§8) are scored against those same checks on must-pass artefacts and re-graded mutants, on three domains only.
- Screening was measured on HumanEvalComm — Python function specifications with deliberate ambiguity — not on this repository's own asks. It shows which seat notices a missing requirement; it does not show how often real asks have one.
- Cost is an API-equivalent yardstick. Every figure is list price × the tokens the seat reported (§4.2). The five seats are flat subscriptions; what a run really consumes is subscription allowance, which no figure here measures.
- Same-day variance is large. Wall-clock time varies by more than 2× between identical runs; time carries a weight of 0.10 for that reason.
- Real use feeds the record. Since 2026-09-29 every goal run a pane starts files its row, its price and its QA verdict, and a turn the pane answers itself is filed as a same-seat pairing. Those rows are marked observed (
run,use) and never averaged with benched ones: observation moves a benched score by at most a third, it never sets one.
10What follows
- Depth where it can change a decision. Third edition: the 108-arm L campaign finished on 2026-09-29 (108 of 108); the deeper repeats below are still to come. Every pairing again at L in all six domains under the QA-inclusive pricing (108 arms), then the top two per domain to five more samples, then leader vs runner-up vs raw to twelve more in backend and frontend — 25–45 hours of machine time, scheduled rather than done, because in most cells the evidence is a tie that more repeats would confirm.
- The judges on every domain and seat, and SWT-bench challengers for the test writer. Third edition: the judge benches ran on five defect classes (§11.3); still open — Codex on the harder set and the Gemini lanes.
- Real use as the long-run source. Every goal run a pane starts now files its QA-checked row; the benches set a score, observation moves it (§7).
- A dedicated screening call before the planner, on the cheap seat that screened best (§8.1). Third edition: shipped (§11.2).
- The reading jobs (lookup, synthesis) need an instrument of their own; until then they are shown as unmeasured.
- Larger corpora. The same asks on a codebase with history, laws violated and cycles, where X-ray has something to say.
11Third edition: what changed between 29 September and 1 October
Everything here was measured or decided after the second edition's rows were written. The record is the same merge table (§7); a row is only counted once its quality has been audited by an executed check.
11.1 One tolerance
The rule, 2026-09-30: a seat or a pair wins if it keeps at least 80 % of the best measured quality in its job and size; among those, the cheapest wins; among equally cheap ones, the fastest. Measured quality outranks claimed quality. The earlier 0.10 absolute band (§7.1) is retired, and with it the per-job and per-role pickers in the app: the choice of AI for a job is automatic, and Settings keeps only the providers, the main provider and a switch for each seat. A seat that is switched off, signed out, or at 95 % or more of its quota window is skipped before the job starts and the work moves to the next seat in the measured ladder.
11.2 The solo path, a screening call, and QA on every run
- Solo for asks of size L or smaller. The editor runs alone: no planning call, $0.00 for the plan, filed in the record as
raw × editor. A planner is still used for an ask with two or more enumerated tasks or over 600 characters. At S and L the router compares measured solo cells against planner × worker pairs under the one tolerance. On a probe over a copy of the live record, solo Composer ateditor-skept 98 % of the best measured quality at $0.016 a run against $0.086 for the cheapest pair inside the band; documentation stays a pair. Only the backend domain is measured ateditor-s, and no end-to-end run was watched binding a solo seat. - Screening is its own call, on the seat that screened best (§8.1), with a fixed set of question kinds and three to seven answer chips. A deterministic front reads closed replies (a yes, a number after a proposal) in six languages without a model.
- QA is obligatory. The checks are deterministic where they can be: expect steps that drive the real page in Chrome, a coverage plan that uses the controls the ask and the added lines are about (not every control on the page: at most 6 buttons, 5 options of a dropdown and 48 steps), and an HTTP test for servers. Pictures are always taken and kept in the QA film; the judge is shown them only when a check failed, none decided, or the styles changed. Unmet claims go back to their workers; once the send-backs are spent the run becomes a question for the person, and a run the person then undoes counts as a false accept against the judge that passed it.
11.3 The QA judges, scored on false accepts
A false accept lets a broken change through, so the QA gate is chosen on it, not on price. Two defect sets, each a broken run or a good run, graded by an executed check. The first (10 broken and 10 good per class, five classes): Opus 0 of 50 false accepts, Grok-4.6-high 0, Composer 2.5 0, Gemini 0, Codex 0 (but it rejects 39–41 of 50 good runs), Sonnet 8 of 50, Haiku 6 of 50. The first set is easy — five seats scored zero — so it does not separate them, and Gemini’s zero there does not carry over: on the second set it is the worst seat measured (below). The second (15 broken and 15 good per class, five classes) does:
| seat | false accepts, all classes | without the frontend class | false rejects |
|---|---|---|---|
| Claude Opus | 14 / 75 (19 %) | 0 / 60 | 26 / 75 (35 %) |
| Grok-4.6-high | 18 / 75 (24 %) | 4 / 60 | 8 / 71 (11 %) |
| Cursor Composer 2.5 | 20 / 75 (27 %) | 6 / 60 | 14 / 73 (19 %) |
| Claude Sonnet | 26 / 74 (35 %) | 12 / 59 | 18 / 75 (24 %) |
| Claude Haiku | 27 / 75 (36 %) | 19 / 60 | 21 / 74 (28 %) |
| Gemini flash (medium) · incomplete | 27 / 45 readable (60 %): backend 14 / 15, frontend 13 / 15 | not measured | 2 / 50 · 55 unread (docs 25, infra 30) |
| Gemini default · incomplete | 15 / 15 on backend (100 %); 113 unread | not measured | 0 / 22 |
Paired sign tests on the cases where two seats disagreed: Opus against Grok-high 0 : 4 (p = 0.125), Opus against Composer 0 : 6 (p = 0.031), Grok-high against Composer 4 : 6 (p = 0.75), Grok-high against Sonnet 2 : 10 (p = 0.039), Opus against Sonnet 0 : 12 (p = 0.0005); no correction for multiple tests. An earlier ranking in tiers and an unpaired test were withdrawn after this paired recount.
- The frontend class is the finding. Every seat except Haiku accepts 14 of 15 broken frontend runs. A judge that reads code cannot see that a page is broken; that verdict belongs to the browser checks of §11.2. A deterministic screenshot lifted every QA seat to 8 of 8 on visual defects in the vision bench.
- The product's QA order is a ladder for when seats are unavailable — Cursor Grok-4.6-high, Composer 2.5, Opus, Sonnet, Gemini flash, Codex — and is not the false-accept order above. Read the table as an order, not as tiers. Gemini was measured only on backend and frontend (the lanes for docs, infra and, for the default arm, design stopped with no verdict, and the cause was not recorded), and on what was read it is the worst seat in the table: flash accepted 27 of 45 readable broken runs, the default arm all 15 broken backend runs, so its place late in the ladder is by evidence and not only by caution. The five rows above it are exact; Gemini’s are not comparable with them, because most of its cells are missing. Codex was not run on the second set.
11.4 The cost against a raw Claude session, cell by cell
Revised the same day, after a fact-check. The first version of this section printed 10.7× and the site said “10×”. It compared each winner with the Claude arm that made the ratio largest, it counted a 95 % interval as if the four S cells were independent, and it set raw single-seat sessions, which carry no QA, against raw Claude while the product ends every run in QA. This is the corrected section; the site said 3× on this basis until the fourth-edition revision (§11.7).
A cell is one job × size × domain × suite that holds a raw Claude reference with audited quality and at least one non-Claude arm with at least two audited samples. The winner of each cell is picked by the router’s own rule (§11.1), five subscription seats only. The reference is the cheapest raw Claude arm (Haiku, Sonnet or Opus) that keeps at least 80 % of the cell’s best measured quality, so that the ratio is not made larger by choosing a dearer Claude; the ratio is the reference’s dollars per run divided by the winner’s, API-equivalent list prices.
| cell | reference | winner | quality | × cheaper |
|---|---|---|---|---|
| S backend · plain | Haiku $0.0364 (n=3) | Composer alone $0.0048 (n=3) | 1.00 vs 1.00 | 7.6 |
| S backend · edges | Haiku $0.0509 (n=3) | Composer alone $0.0057 (n=3) | 1.00 vs 1.00 | 8.9 |
| S backend · branches | Sonnet $0.0941 (n=3; Haiku scores 0.73, below 0.8 of 1.00) | Composer alone $0.0062 (n=3) | 0.87 vs 0.87 | 15.3 |
| S backend · contract | Haiku $0.0464 (n=3) | Composer alone $0.0090 (n=3) | 1.00 vs 1.00 | 5.2 |
| L backend · eight modules | Sonnet $0.330 (n=1) | Codex low alone $0.1215 (n=1, 2 audits) | 0.97 vs 0.97 | 2.7 |
| L backend · fix | Haiku $0.0577 (n=3) | Composer alone $0.0150 (n=3) | 0.93 vs 0.90 | 3.8 |
| L test-writer · SWT-bench | Sonnet $0.3737 (n=10) | Gemini × Composer $0.1041 (n=10) | 0.80 vs 0.90 | 3.6 |
| geometric mean, before the QA check | 5.7 | |||
| the same with the QA step added (estimate) | 3.2 |
What the number is. The median is 5.2× and the cells run from 2.7× to 15.3×; dropping any one cell moves the geometric mean between 4.9× and 6.5×. No interval is printed: the four S cells are one synthetic small-ask family (twelve instances) where every arm scores about 1.0, they are not independent, and a bootstrap that treats them as seven cells is narrower than the evidence (the first version printed 7.0–15.2×). Without the family there are three cells, 2.7×, 3.8× and 3.6×, geometric mean 3.3×. Weighted by what a bill would be (dollars summed over the runs of each cell) the saving is 3.9× over the six cells with n of at least 3.
What the number leaves out: QA. Six of the seven winners are one seat running alone, filed as raw with no QA check, and the Claude arm they are set against has none either; only SWT-bench uses an antiloki pair (Gemini × Composer, 3.6×). The product ends every run in QA, and QA costs money: on the S2 suite raw Composer is $0.0488 a run and Composer × Composer with its QA check $0.086, 1.76× more; on the eight-module task Composer alone is $0.0671 and Composer × Composer × QA $0.1017, 1.5×. Dividing the 5.7× by 1.76 gives 3.2×. That division is an estimate, not a measurement — no end-to-end run was watched binding a solo seat (§11.2, §11.6), and the QA overhead of the other cells was not measured. The one cell where the whole pipeline was priced is the eight-module task: $0.1017 against raw Sonnet at $0.330, 3.2×. The 3× the site carried on this basis was that estimate rounded down; it sat under the 5.7× of the QA-free comparison, and under the 3.3× of the three cells outside the synthetic family.
Limits. n is 3 per arm in five cells and 10 in SWT-bench; the eight-module cell is one Codex run (two audits) against one Sonnet run (the first version used Opus, mean of 2, $1.151, which is inside the tolerance but not the cheapest Claude arm that is). The same-quality swap on SWT-bench (Codex × Composer, 9 of 10 like raw Sonnet, $0.157) is 2.4×; the 3.6× above is Gemini × Composer, which scored 8 of 10 (merges::separated is false at n=10, so “same quality” means inside the tolerance and not separated at these samples, not identical). The 6.7× of the first headline was raw Qwen3.6-plus ($0.0558) against raw Sonnet ($0.3737) on that same SWT cell — not an antiloki arm, a metered model that is off by default — and is withdrawn; the five-seat equivalent is the 3.6× above. Composer, Gemini and Codex dollars are estimated from reported tokens at list rate (Cursor and Antigravity report tokens, not dollars); the Claude references ran 2026-09-22 to 28. The eight-module row prices Codex low alone ($0.1215, 2.7×), the winner when the table was derived; the five-seat re-run of the same task later measured Composer alone at $0.0671 (4.9× under raw Sonnet) and the QA pipeline at $0.1017, and with Composer alone in that row the geometric mean before QA would be 6.2×. S2 and the S cells also disagree: raw Haiku $0.0437, Composer alone $0.0488 and Composer × Composer with QA $0.086 on S2, while Composer alone is priced about ten times lower in the S cells for asks of similar size (cause not found; Composer dollars are token estimates at list price). Time is not a headline axis: matched cells are about equal, except the eight-module cell, where Composer alone took 61 s against Opus’s 278 s (the Opus time moves between 112 s and 303 s from day to day).
The weekly limit. The site states +112 %, and restates what it is (fourth-edition revision). It is the paired task result of §11.7 — on the eight-module task, $0.1554 for the arm where Claude plans and judges against $0.3297 for raw Sonnet, 2.12× — and not a measured percentage of anyone’s weekly limit. The earlier derivation, 1 / (1 − 0.528) for the share of one machine’s tokens that ran off the frontier seat, is withdrawn: an independent audit could not reproduce the share, and the cuts of the record differed (+471 % over the whole mix, +106 %, +118 % and +122 % over three others). By dollars the gain over the recorded mix is +33 % to +45 %, which is the range to quote for a mix of work and not for a task.
The eight-module task, re-run on the five seats (37 checks, 2026-09-28/29; the page’s table): Composer alone $0.067, 61 s, 36.3 of 37 (n=3); Composer × Composer with a Composer QA check $0.102, 95 s, 36.0 (n=3); Sonnet × Composer $0.155, 106 s, 36.8 (n=5); Codex × Composer $0.165, 137 s, 36.7 (n=3); Gemini × Composer $0.184, 134 s, 36.3 (n=5); Grok-high × Composer $0.292, 199 s, 36.3 (n=3); raw Opus $1.151, 278 s, 37 (n=2, the 2026-09-22/23 reference); raw Sonnet $0.330 (n=1, 0.973).
11.5 Does the X-ray map help an agent? Three experiments
Each experiment crossed three agents (Cursor Composer 2.5, Codex, Gemini via Antigravity) with a small and a wide change and three scenarios — the repository only, the X-ray intel only with the agent told not to explore, or both. One run per cell; graded statically, nothing compiled; Gemini reports no tool-call telemetry.
- A wide TypeScript rename (React Compiler, 30 files, 75 places, real baked intel): intel alone (the agent told not to explore) against the repository alone was 31 % faster (75 s to 52 s on average) and used 48 % fewer input tokens (135k to 70k) at the same pass rate (3 of 3 each); both together was about 15 % faster and 5 % lighter.
- A small change: no measurable difference in quality or time on a 13,000-line TypeScript repository with hidden tests (45, 46 and 44 of 48 tests; 178, 180 and 182 s), and in that repository the intel cost more tokens on the wide change for Codex (472k to 1.15M), because the pack is sent again each turn. In the React Compiler experiment's small change the intel-only agent passed 0 of 3 (the repository alone passed 3 of 3; the two together failed once, Gemini), because the index had excluded the file that needed the change — an agent told not to explore inherits the index's blind spots. So the 31 % and 48 % hold for the wide rename only, and the same arm was the worst one on the small change. That is why the intel now says which files it could not read (excluded, unsupported language, parse failed, changed since the index) and never certifies completeness.
- Flow-typed JavaScript was unreadable to the indexer during the second experiment, so that run used hand-built intel and is not reported as product evidence. The reader was fixed on 2026-10-01 (on a copy of facebook/react, files skipped went from 1,160 to 0 and import edges from 1,689 to 6,510); the experiment has not been re-run since.
11.6 What the third edition did not run
- No Claude arm was re-measured in this edition; the Claude references date from 2026-09-22 to 28.
- Codex on the second QA defect set, the Gemini lanes for docs and infra, and any larger-than-L (XXL) run on the five seats. The 6.4× at XXL in the first edition used a Claude brain with DeepSeek hands and is not carried over.
- No new fresh reference arm on the eight-module task, and no end-to-end run watching the router bind a solo seat. The cost cells of §11.4 carry no QA step; its overhead was measured on S2 and on the eight-module task only, and the QA-inclusive 3.2× is an estimate from it.
- A weekly-limit figure per task. The router's own saved-money ledger over 82 settled runs (3.4× on task tokens, 1.9× once planner, screening and QA costs are added) is not used: its baseline is the same tokens priced at Opus's list rate, not a measured raw run, and 99.5 % of its task tokens are Gemini's.
11.7 Fourth-edition revision: the headline reference is Opus
How the headline is defined: the best measured antiloki result against raw Claude Opus on the same task and the same checks, not against the cheapest Claude model. The reasoning is that Opus is the Claude model a subscriber on the top plan reaches for. All figures below come from the merge record (merge_scores) and are the eight-module task of §11.4 (37 checks, ministore); the row ids are in the merge record.
| arm | n | checks | $/run | s/run | × under Opus | × under Sonnet |
|---|---|---|---|---|---|---|
| raw Claude Opus · 2026-09-22/23 | 2 | 37.0 / 37 | 1.151 | 278 | — | — |
| raw Claude Sonnet · 2026-09-23 | 1 | 36 / 37 | 0.3297 | 121 | 3.5 | — |
| Composer alone, no QA step | 3 | 36.3 / 37 | 0.0671 | 61 | 17.2 | 4.9 |
| Composer × Composer, with QA — the headline arm | 3 | 36.0 / 37 | 0.1017 | 95 | 11.3 | 3.2 |
| Sonnet plans, Composer writes, Sonnet judges | 5 | 36.8 / 37 | 0.1554 | 106 | 7.4 | 2.1 |
The headline is the pipeline with its QA step: 11.3×, rounded down to 10×. Its quality is 0.973 of 1.0, inside the one tolerance of §11.1 and not equal to raw Opus; it passes 36.0 of the same 37 checks on average. Opus is not stable from day to day: the ladder run of 2026-09-22 recorded $0.61 for the same task, and against that run the pipeline is about 6.0×, so the honest range is 6×–11× and the headline is the best measured cell. The weekly-limit figure, +112 %, is the Sonnet-planned arm against raw Sonnet, $0.1554 against $0.3297 = 2.12×; with n=5 against n=1 and list-price dollars standing in for an allowance the vendors do not publish, it is a paired task result and not a share of a quota window.
The headline arm is the nearest measured arm to the shipped path, not the shipped path itself: for an ask of size L the shipped path is solo (no planning call) plus QA, and that exact path has not been run end to end. What the headline does not include: the separate screening call the shipped path makes before the plan ($0.006 on Gemini, $0.014 on Haiku), and any end-to-end run of the solo path in real use (§11.2). On the small-ask suite S2 the picture is different: raw Haiku $0.0437, Composer alone $0.0488, Composer × Composer with QA $0.086, so on a small ask there is no saving against Haiku. That is why the S family is not the evidence for the headline.
12Material moved from the landing page (7 October 2026)
The landing page was cut down to what the product does and what it can achieve. Everything it used to say that carries a limit, a number or a source now lives here, so this paper stays the one place with all the details. Nothing below is a new measurement, except where §12.6 says so; it is the former page text, kept whole.
12.1 The task behind the showcase
Eight TypeScript modules in a real React store; tsc -b && vite build must pass; 37 behavioural checks, written as executable code before any arm ran. The same task and the same checks for every arm, scored as cost per run. The pipeline arms were measured 2026-09-28/29 and the Claude references 2026-09-22/23. Composer is Cursor Composer 2.5 (it needs a Cursor seat).
The eight: fuzzy product search, a stable sort, two localStorage-backed React hooks, money formatting with discount codes, a quantity-stepper component, and checkout validation with a Luhn card check. Strict types, no any, JSDoc on every export, no new dependencies, no edits to existing files.
12.2 Every arm, including the single-provider ones
The landing page now shows only the arms where different providers work together, as a showcase of what a merge can do. The two single-provider arms it dropped, and the full ratios, are here.
| arm | $ / run | checks passed | time | n | vs raw Opus | vs raw Sonnet |
|---|---|---|---|---|---|---|
| Composer alone, no planner, no QA step (an ask of size L) | 0.0671 | 36.3 / 37 | 61 s | 3 | 17.2× | 4.9× |
| Composer plans, writes and judges, QA included (the former headline arm) | 0.1017 | 36.0 / 37 | 95 s | 3 | 11.3× | 3.2× |
| Sonnet plans and judges, Composer writes | 0.1554 | 36.8 / 37 | 106 s | 5 | 7.4× | 2.1× |
| Codex plans and judges, Composer writes | 0.165 | 36.7 / 37 | 137 s | 3 | 7.0× | 2.0× |
| Gemini plans and judges, Composer writes | 0.184 | 36.3 / 37 | 134 s | 5 | 6.3× | 1.8× |
| Grok-4.6-high plans and judges, Composer writes | 0.292 | 36.3 / 37 | 199 s | 3 | 3.9× | 1.1× |
| raw Claude Sonnet, one session (2026-09-23) | 0.330 | 36 / 37 | 121 s | 1 | — | — |
| raw Claude Opus, one session (2026-09-22/23) | 1.151 | 37 / 37 | 278 s | 2 | — | — |
The ratios for the Codex, Gemini and Grok rows are computed from the rounded costs shown, so they are good to about one decimal. The Sonnet raw reference is $0.3297 before rounding.
12.3 Reading the 10× and the +112 %
10× is the best measured result against raw Claude Opus on the eight-module task. Raw Opus, one session: $1.151 a run (n=2, 37 of 37 checks, 278 s). The antiloki pipeline, Composer planning, writing and judging with its QA step: $0.1017 a run (n=3, 36.0 of 37, 95 s). That is 11.3×, and the headline rounds it down to 10×. This arm is the nearest measured arm to the shipped path, not the shipped path itself: for an ask of size L the shipped path is solo (no planning call) plus QA, and that exact path has not been run end to end. The reference is Opus because it is the Claude model a subscriber on the top plan reaches for; against raw Sonnet ($0.330, n=1) the same pipeline is 3.2× and the Sonnet-planned run 2.1×.
The honest range is 6×–11×. Raw Opus does not cost the same every day: the ladder run of 2026-09-22 recorded $0.61 for this same task, and against that run the pipeline is about 6.0×. 11.3× is the best measured cell, not the typical one. The 17.2× row has no QA step; the shipped path is solo plus QA, and it was not measured end to end.
The same code quality here means code that passes the same 37 checks: the pipeline averaged 36.0 of 37 against 37 for raw Opus, which is 0.973 of 1.0 — inside the product's 80 % tolerance, not identical. The checks are executed code written before any arm ran, not a model's opinion; quality is the mean over the audited runs (4 of 5 in the Sonnet and Gemini arms, so their means are 36.75 and 36.25, printed 36.8 and 36.3).
+112 % on your weekly limit is a paired task result and not a measured percentage of anyone's weekly limit. On this task the arm where Claude still plans and judges while Composer writes cost $0.1554 against $0.3297 for raw Sonnet: 2.12×, so about 2.1× further on that task (n=5 against n=1, list-price dollars standing in for an allowance the vendors do not publish). What it does for your limit depends on your mix of work.
Limits. One task in one repository, n of 1 to 5 per arm. Dollars are API list prices; Cursor, Codex and Gemini report tokens, not dollars, so theirs are estimated from the tokens. The shipped path adds a separate screening call before the plan ($0.006 on Gemini, $0.014 on Haiku) that is not in these figures, and the solo path (no planner, for an ask of size L or smaller) has not been watched end to end in a real run. The small-ask suite cuts the other way: on S2, raw Haiku cost $0.0437, Composer alone $0.0488 and the Composer pipeline with QA $0.086, so on a small ask there is no saving against Haiku — and in the report's small-ask cells Composer is priced about ten times lower than on S2 for asks of similar size, cause not found. That family is not the evidence for this headline. Across seven cells against the cheapest Claude arm inside the tolerance the geometric mean was 5.7× before QA and about 3.2× with QA added by estimate (§11.4).
12.4 What sits around the work
The five points on the landing page are short forms of what follows.
Routing. Every job of an ask goes to the seat measured best for it, under one rule: keep at least 80 % of the best measured quality, then the cheapest, then the fastest. The frontier seat is asked only for the parts that need it. A seat at 95 % of its window, or out of quota, is skipped before the job starts, not after it fails. There are no per-job pickers to set. An ask of size L or smaller goes to one seat, alone, with no planning call; a bigger one is split into jobs, each bound to the seat the benchmark measured best on that work — usually not the expensive one.
Control. You decide which folders may be written. For every seat the fence is enforced where the work lands: a change beyond it is shown, can be undone in one click, and stops the merge or the delivery. On Claude panes a hook at the tool call, outside the model, also refuses the write before it happens; the other seats have no such hook.
Verification. Every turn is priced and timed, and on Claude panes a hook outside the model logs each tool call. Then QA opens the real page in Chrome and uses the controls the change is about, or sends requests to the server, with checks written as plain expectations. The judge is shown a picture only when a check failed, none could decide, or the styles changed. Unmet claims go back to the worker; when the send-backs are spent, the run becomes a question to you. The judge is chosen on how often it let a broken run through (§11.3). The QA test is filmed: a browser frame with the live picture, a cursor gliding to each target, a subtitle for every step, and the kept film to replay afterwards; for a server, a terminal of requests, answers and checks. Before and after screenshots sit in the run story, linked at full size.
Learning. Every run is scored on your machine — which seat, which job, how long, what it cost, whether it passed first time — and filed in the benchmark that ships with the app. The shipped scores are the starting point; your runs adjust them by at most a third, and a pairing that was only watched working never displaces one that was benched. The scores stay on the machine.
Agents and roles. You do not staff a roster. A setup wizard finds the AIs on your machine — Claude, Codex, Cursor, Gemini and Grok, each listed as ready, not signed in or not installed, with the command that fixes it — lets you switch off the ones you do not have, and asks for one main provider. It offers a provider only when the seat is real, never merely because a binary exists. Then you ask, and the agent takes the role the ask needs (Backender, Frontend designer, Tester, Reviewer and others), arriving with its scope, skills and specialists. The router chooses the seat for every job from the seats you are signed in to and that still have room in their window. New workspaces open in a proper window with a folder browser.
Queue and run controls. Each agent keeps a queue of asks (done, doing, next) you can reorder, cancel and re-prompt, with a checkpoint per response to undo to. A run can be paused, resumed or stopped, with buttons in the run story. While a finished run is still being judged the button reads “stop the QA”: the work stands, is marked not verified, and nothing is sent back. A long conversation closes into episodes that consolidate into a summary. Every turn closes with what it did, which seat did it, what it cost, and the QA verdict.
Seats and limits. Settings has a switch per seat, and a seat you turn off is not called — not by a chat, a terminal, a hand-off, QA, a merge or the advice. Nothing is hard-coded to one vendor: a new chat starts on the first usable seat, and every role has a measured fallback when its first choice is off or out of quota. Claude and Codex show their 5-hour and weekly windows. Cursor, Gemini and Grok publish no quota, so they show the calls and tokens sent in the last 5 hours and 7 days — a count, never dressed up as a percentage, and covering only what went through antiloki. The saving on screen counts only the current workspace's runs and drops the old number the moment you switch; the windows and the counts are the account's and this machine's, whatever workspace made the call.
Shipping. Before it opens a pull request, the button tells you what will happen; for a repo with no online remote it offers, with your confirmation, to create a private GitHub repo first. A GitHub token can be added in Settings: verified before it is saved, kept owner-only, never shown again, removable.
The plain parts are not asked of a model. A yes, or a number, after a proposal is read by a deterministic front in English, Portuguese, Spanish, French, German and Italian. A gate rejects a run that changed nothing, wiped the tree or only deleted files before any model reads it. Gauges decide only when they can point at a fact.
12.5 X-ray: what it reads, and what it measured
Every file gets a verdict — a score and why it is not 100, axis by axis — computed from your ASTs and git history, so the same commit scores the same twice and no model is asked for the number. Around it: file reviews with comments at the lines, the rules your codebase lives by checked at the line, a map of the codebase (every file a box; hubs, orphans, cycles), the coverage gaps, and the blast radius of a change before it lands.
It reads TypeScript and, since 2026-10-01, Flow-typed JavaScript: on a copy of React the Flow files it had to skip went from 1,160 to none, and 11 of 1,177 are still unreadable. It says what it did not index — excluded, an unsupported language, a parse that failed, changed since the index. The same facts reach the agent as its context pack. Measured on a wide TypeScript rename (three agents, one run per cell), the map alone was 31 % faster on 48 % fewer input tokens at the same pass rate. On a small change the map alone passed 0 of 3, because the index had left out the file that needed the change; that is why the pack now says what it could not read. On a second repository the map made Codex use more tokens (472k to 1.15M). The cells are in §11.5.
12.6 Does the agent's own law check help? A pilot
The landing page's comparison says the agent can check its own edits against your project's laws before it finishes. This is the measurement behind it, and it is a pilot: three seats (Cursor, Codex, Gemini), one run per cell, a fixture repository whose legacy module itself breaks the laws. It was run on 2026-10-06, after the §11.5 experiments, and is new in this edition.
Four conditions: L0 task only; L1 the 17 laws as text in the prompt; L2 the laws as text plus a lawcheck command that runs the product's own detectors, with the instruction to run it on every edited file; L3 the command alone, no laws text. Violations were counted by independent graders (Biome rules, regexes and suppression detection), not by the product's own detectors, and hidden tests checked that the work still worked.
| seat | L0 task only | L1 laws as text | L2 laws + tool | L3 tool only |
|---|---|---|---|---|
| Cursor | 30 | 19 | 0 | 5 |
| Codex | 19 | 0 | not run | 0 |
| Gemini | 54 | 0 | not run | 0 |
Without the laws, the three seats introduced 103 violations; with the tool alone, 5. The laws as text were enough for Codex and Gemini (19 and 54 to 0) and not for Cursor (30 to 19, because it copied the legacy module's style anyway). The tool took Cursor from 19 to 0 at +29 % time (56 s to 72 s) and flat input tokens (19k), and the tool alone was not slower than the text for Codex (116 s against 147 s) or Gemini (190 s against 250 s). On the first, smaller tasks the laws as text already removed every violation, so there was nothing left for the tool to improve.
What it does not show. The pre-registered rule (the tool helps if violations fall by at least half against the text, tests do not drop, time and tokens grow by no more than 25 %, same direction on two seats) could not be met: only Cursor had headroom, and its time rose by 29 %, one 104 s outlier included. It is one seat, three cells, n=1. It supports the tool as a real gain where the text fails; it is not proof. The five violations left under the tool alone exposed a defect in the check, a cap of five reported hits per law per file; it now counts every hit, and a result it cannot tell is reported as “cannot tell”. Over the 30 pilot cells the product's own count matched a literal count of what each agent added in 119 of 120 (cell, law) comparisons; the one difference is a law that skips test files by design.
12.7 Compared with the tools alone
antiloki runs Claude Code, Cursor, Codex, Gemini and Grok; it does not replace them. Each does some of what is below by itself, so the difference is what you set up and keep up by hand, and what is on by default. The vendor side is taken from their documentation as it stood in October 2026 [11–13]; it is a snapshot, and these products change quickly.
| what you want | the tools alone | with antiloki |
|---|---|---|
| Whose models do the work | Claude Code uses Anthropic's models. Cursor uses the models in its plan, picked by you or by its own router, which chooses per request. | Every subscription you already pay for, as one agent. Each job goes to the seat measured best for it, and to the next one when a seat runs out. |
| Keeping it inside a folder | Claude Code has permissions, hooks and an OS-level sandbox you configure; the sandbox covers shell commands and their children. Other tools have their own settings. | One fence for every seat, enforced where the work lands. A change beyond it is shown, can be undone in one click, and stops the merge. A pre-write hook exists on Claude panes only. |
| Testing the result in a browser | Claude Code, with its Chrome extension, and Cursor's browser tool can drive a page when you ask them to. | On every run. QA opens the page and uses the controls the change is about, with checks written as plain expectations. The test is filmed. |
| A verdict on the code itself | Review tools such as Cursor's Bugbot read a pull request and give an opinion. | X-ray scores every file from your ASTs and git history; no model gives the number. |
| Following your rules | Rules live in prompt files and settings. | The agent can check its own edits against your project's laws before it finishes (§12.6, a pilot). |
| What it costs, and where you stand | Each tool shows its own usage. | Every turn priced and timed, per seat, with each seat's window or call count (§12.4). |
What you give up: one more app. The Windows build is current; the macOS and Linux builds are older. A merge needs two signed-in seats, the browser QA needs Chrome, and paid plans are not open yet. When the tool alone is the better choice: a quick edit in one file, or one subscription and nothing to compare (§12.3: on a small ask there is no saving against Haiku).
12.8 The setting: adoption without control
This was the landing page's opening argument, moved here with its sources. The figures are as the cited pages report them; we did not re-run those surveys, and two of the three links are secondary summaries.
Adoption is finished and control never arrived: 90 % of professional developers use coding agents weekly and 68 % daily, but only 29 % trust what comes out — down from 40 %, with 3 % who highly trust it [14, 15]. And the failure mode is not theoretical: an agent quoted the rule that forbade it and deleted a production database and its backups, and the deletion call took nine seconds — 93 % of organisations have had an AI-caused incident, 19 % have governance for the next one [16]. Rules in a prompt are not enforcement.
AReproducing a row
bench/merges/run.sh L claude-deepseek false # one arm: pair · size · QA — clones ministore, drives /cockpit/goals/plan,
# audits with bench/ladder/audit-dir.sh, POSTs the audit to /settings/merges/record
bench/merges/matrix.sh # the built-in pairs × S · L, own serve on :41941, its own home
bench/merges/jobs.sh · jobs2.sh · jobs3.sh · jobs4.sh # QA-on runs · blueprint (tests) · a wave of 3 (report) · raw singles + six pairs
GET /api/v1/settings/merges # the table, weights included
POST /api/v1/settings/merges/record # an Outcome; audit_only joins an existing row
GET /api/v1/settings/audit # every call: model · ms · tokens · deterministic cost
RReferences
- [1]LMSYS, RouteLLM: an open-source framework for cost-effective LLM routing, July 2024.
- [2]Martian's tool automatically switches between LLMs to reduce costs, TechCrunch, Nov 2023.
- [3]Not Diamond — model routing for coding agents.
- [4]OpenRouter more than doubles valuation to $1.3B in a year, TechCrunch, May 2026; Sacra for volume.
- [5]aider, Separating code reasoning and editing, Sept 2024.
- [6]aider, R1 + Sonnet set SOTA on aider's polyglot benchmark, Jan 2025; R1 + V3 benchmark.
- [7]Cline: Plan & Act; Roo Code: using modes.
- [8]Wang et al., Mixture-of-Agents enhances large language model capabilities, 2024.
- [9]Zaharia et al., The shift from models to compound AI systems, BAIR, Feb 2024.
- [10]mergekit — what "model merging" means in ML, and what this paper does not do.
- [11]Anthropic, Claude Code documentation: permissions and sandboxing; hooks; subagents; accessed October 2026.
- [12]Anthropic, Claude in Chrome for Claude Code, driving a browser to test an app, accessed October 2026 (what's new).
- [13]Cursor documentation: agent overview (Auto router, browser tool, checkpoints, cloud agents); Bugbot; accessed October 2026.
- [14]JetBrains Research, AI coding agent adoption 2026, August 2026.
- [15]Stack Overflow developer survey 2026, as summarised by byteiota (a secondary source).
- [16]Vibe coding incident guardrails, bex.co, July 2026, citing the PocketOS incident and Spacelift's 2026 survey (a secondary source).