Compiled, Not Trained

Heimdall, the merge and the X-ray adapter: how antiloki builds a specialised coding model out of measured parts.

Second edition, measured 28–29 September 2026 on the five subscription seats — Claude Code, Codex, Gemini, Grok and Cursor. The first edition (22–23 September) measured Chinese models through OpenCode Zen; those arms are withdrawn and kept on disk as history. §4.2, §4.3, §7.1, §8 and §9 are rewritten; the depth runs still to come are listed in §10.

Third edition, 1 October 2026 — adds §11: the one tolerance (a seat wins if it keeps 80 % of the best measured quality, then price, then time), the solo path for asks of size L and smaller, a screening call of its own, obligatory QA, the QA judges' false-accept rates on two defect sets, the cost against a raw Claude session cell by cell (the 3× the site carried at the time of the third edition), and three X-ray experiments. It supersedes the 0.10 absolute band of §7.1, the planner and screening rows of Table 1, the “on request” QA of §5, the judge paragraph of §8.1 and the open items of §10 that are now done. Measured 2026-09-28 to 10-01; no Claude arm was re-run in this edition, and the Claude references date from 2026-09-22 to 28.

Fourth-edition revision, 1 October 2026 — the site’s headline reference is now raw Claude Opus and not the cheapest Claude arm inside the tolerance, because Opus is the Claude model a subscriber on the top plan reaches for. The headline is 10× on the eight-module task: the best measured cell, 11.3× rounded down, with a range of 6×–11× over the days Opus was measured (§11.7). The seven-cell analysis of §11.4, which gave the 3×, stays as the Sonnet-and-Haiku-reference result and as context; it is no longer the headline. The weekly-limit figure is restated as a paired task result (§11.4, §11.7).

1The claim, stated carefully

"Model merging" in machine learning means arithmetic on the weights of checkpoints that share a base (task vectors, TIES, DARE; mergekit is the tool). antiloki does none of that: the models are frozen, hosted by their vendors, reached through the vendors' own CLIs on your own subscription. What antiloki merges is behaviour: the answer to "which model does which job, shown what, judged how" — and that answer is learned from outcomes.

The analogy we defend is narrower and, we think, exact in practice. A LoRA adapter specialises a frozen model for a task by adding a small learned component in front of it. X-ray is a small deterministic component in front of every call that supplies what a coding fine-tune would otherwise have to learn — the structure, laws, risks and history of this repository — and it is recomputed at each commit instead of going stale. The merge table plays the role of the training loop: an outcome-weighted selection over roles, updated by every benchmark row and every audited use. The difference from a trained merge is the ceiling: no role can exceed what its base model can do when shown the right evidence. For code, where the missing knowledge is almost always the codebase and not the craft, that ceiling has not been the binding constraint in any measurement below.

2Related work

Query routing between a strong and a weak model is well studied. RouteLLM (LMSYS, 2024) trains routers on Chatbot Arena preferences and reports cost reductions of over 85% on MT Bench at 95% of GPT-4's quality [1]. Commercial routers — Martian [2], Not Diamond [3], and the OpenRouter aggregator now processing 25 trillion tokens a week [4] — sell the same lever per request. Heimdall differs in what it routes over: not a request, but a job inside a run, with the job's own acceptance and its own measured winner.

The brain × hands split is aider's architect/editor mode [5]: a reasoning model describes the change, a cheaper model writes the edit. aider reported a new state of the art on its polyglot benchmark with DeepSeek R1 as architect and Claude Sonnet as editor at 14× less cost than the previous o1 result [6]. Cline and Roo Code let users bind different models to Plan and Act modes by hand [7]. antiloki generalises the split from two roles to ten jobs and makes the binding measured rather than chosen.

That a composition of models can behave as a new, better model was shown by Mixture-of-Agents (Together, 2024): layered open models reached 65.1% on AlpacaEval 2.0 against GPT-4o's 57.5% [8]. The "compound AI systems" position from Berkeley [9] argues the unit of capability is now the system. We take both as the frame: the merged agent is the model; its card is its benchmark row.

3The jobs

Every call antiloki makes on a coding run belongs to one of ten jobs. A job has a side (brain or hands), a prompt template, and — the point of the system — its own winner and fallback per engine and globally.

The classifier is deterministic (cli::classify_ask): a question mark or an interrogative first word makes a read; among reads, the words why, how, explain, describe, compare, difference, trade-off, should, in which order make it synthesis, else lookup. A write that names at most one file and is under 400 characters is editor-s, else editor-l. Sizes for the benchmark follow the plan, not the ask: two or more tasks is XXL; one task owning one file with a goal under 400 characters is S; else L.

4Heimdall, the router

4.1 Resolving a binding

For a job on a run, Heimdall produces a binding (engine, model) and a fallback, in this order:

1. The merge. If the run carries a pair, the job's side decides the engine and, when the side names one, the model. job_binding_paired.
2. The learned table. For a solo run, if the merge table holds a row for this job and size with at least three samples, its best row outranks the static table. job_binding_learned, merges::best_for(job, size, 3).
3. The static table. The winner and fallback measured for the job (Table 2), taken globally if the winner's CLI is installed (which), else the run engine's own cell. job_binding_any.
4. The effort dial. CHEAPER moves every role one rung down its engine's price ladder; SMARTER one rung up; BALANCED leaves the bound models. AUTO lets steps 1–3 stand; ON obeys the dial; OFF routes nothing.
5. The price rule. A worker is never dearer than its planner: worker_under_planner walks the worker down its ladder until its list price is under the planner's, and logs why.
6. Failure. A binding whose engine fails is retried once on the fallback, at the fill for workers and in run_job for every other job.

4.2 Prices and the deterministic cost

Every cost antiloki reports is computed, never read from a vendor's estimate: input and output tokens times the model's list price per million, with cache reads at one tenth of input and cache writes at 1.25× (cli::price_tokens). The same table gives the ladder each engine's effort dial walks.

Each call is also timed on the serve's own clock and its tokens counted from the engine's stream; the triple (cost, ms, tokens) lands on the pane's timeline as an audit line and in the ai_calls table, and — through merges::record_use — as a benchmark row of size call.

What "saved" is measured against (since 2026-09-28): the same tokens priced at the frontier model's list rate — Claude Opus, or Codex's frontier model on a Codex seat (cli::frontier_of) — against what the work cost at the seats that actually ran it, each task priced at its own engine and model. Before that the yardstick was the seat's own top rung, which for four of the five seats is the same rate as its cheap rung and printed "saved $0.00" on work five times cheaper than Opus. A row nobody priced is priced from list rates (merges::estimated_usd); a replayed plan is never counted as a free planner.

4.3 Modes and the merge in Studio

A merged agent in Studio is a recipe with a pair: a name, a brain side, a hands side, a theme, a scope, its specialists and skills. Since 2026-09-27 every side is one of the five subscription seats — Claude Code, Codex, Gemini, Grok and Cursor; the OpenCode/Zen pairs of the first edition (Claude × DeepSeek, Claude × Kimi, Codex × DeepSeek …) are withdrawn and kept on disk only as history. A seat may sit on both sides (Composer plans and writes), which is measured as a pairing like any other. Heimdall's chip on a pane shows the mode and the money the router saved on that pane, computed as API-equivalent dollars against one high-tier session doing the same tokens.

5The pipeline a goal runs through

A goal run is the unit the benchmark measures. Its stages, and the model that answers each, are what the merge assigns.

The projection. Before a plan fans out, antiloki estimates whether a wave beats one worker. A single session doing k parts is projected at SERIAL_FACTOR = 1.5 times the parallel per-part time; the wave's concurrency is bounded by the depth of the plan's dependency chain (effective_concurrency = ceil(fills / chain), clamped to the configured cap); and a plan whose single-session projection is under the parallel floor — 900 s by default, ANTILOKI_PARALLEL_FLOOR_S, 0 = off — collapses to one worker. With no history the medians assumed are 120 s per fill and 150 s of orchestration. This is why every S and L row below is a one-worker run: the wave only pays above fifteen minutes of projected serial work.

The verifier's one send-back. A task the verifier rejects goes back to its worker once with the rejection as the brief; a second rejection fails the task. Both rejections observed on L were real defects a compile gate would not catch.

The QA gate. An isolated judge checks each acceptance claim against the final diff and answers met / unmet / unproven; on a one-worker run the always-branch kept a placeholder claim, so the gate reported unproven. That gap is closed in the third edition: QA is obligatory, runs the deterministic checks (§11.2), sends unmet claims back to their workers a bounded number of times and, when those are spent, turns the run into a question for the person.

6The adapter

A LoRA specialises a frozen model by adding a small learned component in front of it. antiloki's adapter is the whole conditioning layer around frozen models, and it is larger than the router: which model answers each job, the brain × hands split, the skills materialised into the agent's box, the specialists it may consult, the tools it may drive, and the deterministic intel it boots with. A Designer is not a different model from a Backender — it is a different adapter.

6.1 X-ray — the deterministic half

X-ray is antiloki's analysis of the repository. Nothing in it is generated by a model; all of it is computed from the files, the ASTs, the tests and the git history, and it is what every call is shown before it explores.

reviews

every file scored, and why not 100 axis by axis — coupling, structure, cohesion, defects, tests, change — with the comment at the line that carries the move.

laws

the rules the codebase lives by, checked at the line (for TypeScript: no any, no var, no loose equality, no debugger, no @ts-ignore, no swallowed errors), with what compliance is worth.

graph

every file a node, imports the edges: hubs, orphans, cycles, the most central and the most risky, per folder.

blast radius

what a change to a file would touch, before it lands.

coverage gaps

what the tests never reach, by file; the score's history over snapshots.

context pack

the bundle a pane boots with: laws · folder · file · graph · tests · blast · doc items, derived from the pane's scope by default, under a character budget (200 000).

the retriever

for a chat turn, chat_auto::compose_for picks from the workspace's intel and prior runs — the chat's own log excluded — under an 8 000-character cap.

MIN mode

exploration off: the agent navigates by the pack instead of reading the tree, and touches the disk only to edit inside its scope.

This is why the analogy in §1 holds for code specifically. What a repository-specific fine-tune would teach — where things are, what the conventions are, which files are dangerous — is the content of the pack, and the pack is true at the commit the agent is working on.

6.2 The role — the adapter as a unit a person hires

A job is a stage inside one run; nobody hires a verifier. A role is what a person hires — a domain of the codebase — and inside it every one of the ten jobs still happens. So a role is not an eleventh job; it is a profile over all ten, scoped to a domain, carrying its own adapter.

The domain is derived from the paths a task owns, deterministically (roles::domain_of): a test of the API is test work; the styles folder beats the extension; a change touching six backend files and one stylesheet is still backend, and an even split is honestly mixed. So every run tags itself, and runs that happened before the dimension existed can be attributed without re-running them.

A role asks the table for the merge that won its kind of work. When its domain has no samples it reaches in order — its nearest neighbour (design looks at frontend; infra and docs at backend), then the shared pool, then everything pooled — so a role always has an answer and the sample count says how far it had to reach.

6.3 Three of each

The same rows, three questions. Cheaper counts the bill above the score (0.35 / 0.55 / 0.10); Best is the table's own weight (0.70 / 0.20 / 0.10); Smarter counts only quality and allows no gap at all below the best. Each returns a different merge, and the three-point band still holds for the first two: a row ten points behind cannot be bought back at any price. They are the same three stops as the effort dial on the pane.

7The benchmark, and how it learns

The merge table (~/.antiloki/merges.sqlite, merge_scores) has one row per job × size × planner × worker × qa. Sizes are s · l · xxl for runs and call for audited single calls. A row accumulates:

where audit is the external behavioural score a bench posts after the run (L: 37 checks of the eight modules; S: a five-case check of the function the ask names) and source says who measured it: bench (antiloki's own runs, shipped as the seed), run (the user's goal runs), use (audited calls). A user's table is therefore the shipped base plus their own rows, weighted together; nothing is collected back.

7.1 The weight

Every record re-weights all rows of that job and size against each other (merges::reweigh, merges::rank):

Price first, speed last, quality as a gate. Third edition: the gate is now relative — a seat or pair wins if it keeps at least 80 % of the best measured quality (Objective::band() = 0.20), and measured quality outranks claimed; there is no second tolerance anywhere in the product (§11.1). The band disqualifies a materially worse answer outright; inside it, the bill decides. The first edition weighted 0.7 / 0.2 / 0.1 with a 0.06 band, which was narrower than the instrument — a ten-instance bench resolves quality in steps of 0.10 — and would pay 6.7× for one instance in ten. Speed was reduced from 0.30 to 0.10 on 2026-09-28, after the fastest frontier planner won a simple frontend task that a model a tenth of its price had measured the same on. A pairing measured on several suites is pooled by samples, never overwritten; a row nobody priced is priced from list rates; a planner row is never cheaper than its own list price.

7.2 Winners

A job's winner is its best sized row; a job the benches never sized falls back to its best audited-call row with at least two samples; else it has no winner yet, and the Studio says so. Each winner is named by composition and role — the brain's brand and the hands' suffix as one word, then the job's title: CodexComposer Builder, Composer Tinker, ClaudeComposer Architect — and can be made an agent in one click, carrying its pair. For solo runs the learned table outranks Table 2 once a row has three samples, which is how the static table is meant to retire.

8Results

Measured 28–29 September 2026 on the five subscription seats, one machine, a pinned fixture (ministore at 3de696c), every arm in a fresh clone with its own plan (plan replay cleared per arm). Quality is the deterministic grader's score for that task — calibrated before any arm ran, never a model's opinion; dollars are the API-equivalent of every call the serve made for the run (planner, writer, verifier, QA), priced from the tokens each seat reported (§4.2). The tables are generated from the record by bench/campaign/make-results.py; rows under three samples are marked and never ranked.

* fewer than three samples — shown, never ranked. Prices from different columns are not comparable: the later ones include the QA check every run now pays.

* fewer than three samples — shown, never ranked. Prices from different columns are not comparable: the later ones include the QA check every run now pays.

* fewer than three samples — shown, never ranked. Prices from different columns are not comparable: the later ones include the QA check every run now pays.

* fewer than three samples — shown, never ranked. Prices from different columns are not comparable: the later ones include the QA check every run now pays.

* fewer than three samples — shown, never ranked. Prices from different columns are not comparable: the later ones include the QA check every run now pays.

* fewer than three samples — shown, never ranked. Prices from different columns are not comparable: the later ones include the QA check every run now pays.

Calibrated before any arm ran: a correct reference 16/16, rounding the discount 14/16, case-sensitive codes 14/16, naive floats 7/16, no change 0/16.

Balanced accuracy = (recall on damaged prompts + precision on clear ones) / 2. The per-seat recall and precision are in bench/screen-solo/out.

Cases: 3 must-pass artefacts and 3 mutants per domain (files the ask names cut in half, re-graded below 0.90). Frontend, design and mixed have no honest must-fail case yet — see §10.

8.1 What the numbers say

• At S, the writer decides and the planner buys nothing. Every pairing writing with Composer scored 1.00 on S2 whatever planned it — Claude Sonnet, Codex, Grok, or Composer itself — so the cell ranks on price: Composer planning and writing alone costs $0.086 a run with its QA check, Sonnet × Composer $0.140. The grader is not blind: raw Haiku scored 0.87 on the same checks (raw Composer 1.00, $0.049, with no pipeline and no QA).
• At L, one seat planning and writing is within the band everywhere. Composer × Composer scored 0.88–1.00 across the six domains at $0.080–$0.117 a run, inside 0.10 of the best pairing in each. The "writer plans alone" hypothesis holds; whether a frontier brain earns its price at L is the question the deeper runs (§10) are for.
• Screening is a cheap seat's job, done separately. One screening call per ask on Gemini or Haiku reached 0.94 balanced accuracy — it caught 87.5 % of the damaged prompts and left every clear one alone — for $0.006 and $0.014 a screen; Claude Sonnet caught 68.8 % of the damaged prompts at $0.041. The screening built into today's planner call caught 43.8 % — HumanEvalComm reports models asking on under 37 %.
• Which seat should judge. (Second edition. Superseded by §11.3, which scores the judges on false accepts over a harder defect set.) On must-pass artefacts and re-graded mutants of the same work, qa-gate · Claude Haiku 89 % over 9 cases in 2 domain(s); qa-gate · Composer 83 % over 18 cases in 3 domain(s); qa-gate · Codex low 56 % over 18 cases in 3 domain(s); verifier · Composer 100 % over 18 cases in 3 domain(s); verifier · Codex low 89 % over 18 cases in 3 domain(s); verifier · Claude Haiku 72 % over 18 cases in 3 domain(s). Composer is the verifier to bind: right on every case, the cheapest. As the QA judge no seat is clean — Codex rejects correct work (every one of its errors a false reject), Haiku lets truncated TypeScript through as a verifier and answers the QA prompt in a shape that cannot always be read. A false accept lets a broken change through; the per-case log separates the two error kinds.
• A frontier planner moved the price, not the quality, where it was measured on equal footing. On S2, Sonnet planning cost 1.6× and Grok planning 2.7× what Composer planning for itself did, for the same 1.00. At L the pairings' rows predate the QA-inclusive pricing, so that comparison waits for the deeper runs (§10).

9Limitations and threats to validity

• n is small, and most comparisons are ties. An L cell holds two or three runs per pairing per domain; S2 holds eight. Inside the 0.10 quality band the ranking is decided by price, so in most cells the honest statement is "the cheapest of the pairings that did equally well", not "the best model". §8 marks every row under three samples. The depth that would turn ties into rankings (n ≥ 20 per routed cell) is scheduled, not done; real use now feeds the record continuously (below), which is where most of that depth is expected to come from.
• One repository. Every number is ministore, a small TypeScript store. The merge's advantage on a large, old codebase is the claim we most want to test and have not.
• Where the tie is, the writer decides. On S2 every pairing writing with Composer scored 1.00 whatever planned it, and raw Composer alone did too; the grader still separates writers (raw Haiku 0.87). A pairing's brain buys nothing measurable at this size, which is why the cheapest configuration wins there.
• Two recording faults, found and fixed on 2026-09-29, bound the record written the day before. (1) Once QA became obligatory, the QA gate's own record counted every run twice; bench rows were re-copied from their run counters, real-use rows from that window keep the inflated count. (2) A guard read Antigravity's default model as a third-party one and silently re-seated a Gemini planner on Claude Sonnet; those runs are filed as Sonnet (which is what planned them), Gemini as an S planner is unmeasured, and a bench arm whose pair would be re-seated is now refused instead.
• Grading. Every bench row is graded by an executed check written before any arm ran and calibrated at both ends (a correct reference scores 100 %, a no-op 0 %); no row's quality is a model's opinion. The judges (§8) are scored against those same checks on must-pass artefacts and re-graded mutants, on three domains only.
• Screening was measured on HumanEvalComm — Python function specifications with deliberate ambiguity — not on this repository's own asks. It shows which seat notices a missing requirement; it does not show how often real asks have one.
• Cost is an API-equivalent yardstick. Every figure is list price × the tokens the seat reported (§4.2). The five seats are flat subscriptions; what a run really consumes is subscription allowance, which no figure here measures.
• Same-day variance is large. Wall-clock time varies by more than 2× between identical runs; time carries a weight of 0.10 for that reason.
• Real use feeds the record. Since 2026-09-29 every goal run a pane starts files its row, its price and its QA verdict, and a turn the pane answers itself is filed as a same-seat pairing. Those rows are marked observed (run, use) and never averaged with benched ones: observation moves a benched score by at most a third, it never sets one.

10What follows

• Depth where it can change a decision. Third edition: the 108-arm L campaign finished on 2026-09-29 (108 of 108); the deeper repeats below are still to come. Every pairing again at L in all six domains under the QA-inclusive pricing (108 arms), then the top two per domain to five more samples, then leader vs runner-up vs raw to twelve more in backend and frontend — 25–45 hours of machine time, scheduled rather than done, because in most cells the evidence is a tie that more repeats would confirm.
• The judges on every domain and seat, and SWT-bench challengers for the test writer. Third edition: the judge benches ran on five defect classes (§11.3); still open — Codex on the harder set and the Gemini lanes.
• Real use as the long-run source. Every goal run a pane starts now files its QA-checked row; the benches set a score, observation moves it (§7).
• A dedicated screening call before the planner, on the cheap seat that screened best (§8.1). Third edition: shipped (§11.2).
• The reading jobs (lookup, synthesis) need an instrument of their own; until then they are shown as unmeasured.
• Larger corpora. The same asks on a codebase with history, laws violated and cycles, where X-ray has something to say.

11Third edition: what changed between 29 September and 1 October

Everything here was measured or decided after the second edition's rows were written. The record is the same merge table (§7); a row is only counted once its quality has been audited by an executed check.

11.1 One tolerance

The rule, 2026-09-30: a seat or a pair wins if it keeps at least 80 % of the best measured quality in its job and size; among those, the cheapest wins; among equally cheap ones, the fastest. Measured quality outranks claimed quality. The earlier 0.10 absolute band (§7.1) is retired, and with it the per-job and per-role pickers in the app: the choice of AI for a job is automatic, and Settings keeps only the providers, the main provider and a switch for each seat. A seat that is switched off, signed out, or at 95 % or more of its quota window is skipped before the job starts and the work moves to the next seat in the measured ladder.

11.2 The solo path, a screening call, and QA on every run

• Solo for asks of size L or smaller. The editor runs alone: no planning call, $0.00 for the plan, filed in the record as raw × editor. A planner is still used for an ask with two or more enumerated tasks or over 600 characters. At S and L the router compares measured solo cells against planner × worker pairs under the one tolerance. On a probe over a copy of the live record, solo Composer at editor-s kept 98 % of the best measured quality at $0.016 a run against $0.086 for the cheapest pair inside the band; documentation stays a pair. Only the backend domain is measured at editor-s, and no end-to-end run was watched binding a solo seat.
• Screening is its own call, on the seat that screened best (§8.1), with a fixed set of question kinds and three to seven answer chips. A deterministic front reads closed replies (a yes, a number after a proposal) in six languages without a model.
• QA is obligatory. The checks are deterministic where they can be: expect steps that drive the real page in Chrome, a coverage plan that uses the controls the ask and the added lines are about (not every control on the page: at most 6 buttons, 5 options of a dropdown and 48 steps), and an HTTP test for servers. Pictures are always taken and kept in the QA film; the judge is shown them only when a check failed, none decided, or the styles changed. Unmet claims go back to their workers; once the send-backs are spent the run becomes a question for the person, and a run the person then undoes counts as a false accept against the judge that passed it.

11.3 The QA judges, scored on false accepts

A false accept lets a broken change through, so the QA gate is chosen on it, not on price. Two defect sets, each a broken run or a good run, graded by an executed check. The first (10 broken and 10 good per class, five classes): Opus 0 of 50 false accepts, Grok-4.6-high 0, Composer 2.5 0, Gemini 0, Codex 0 (but it rejects 39–41 of 50 good runs), Sonnet 8 of 50, Haiku 6 of 50. The first set is easy — five seats scored zero — so it does not separate them, and Gemini’s zero there does not carry over: on the second set it is the worst seat measured (below). The second (15 broken and 15 good per class, five classes) does:

Paired sign tests on the cases where two seats disagreed: Opus against Grok-high 0 : 4 (p = 0.125), Opus against Composer 0 : 6 (p = 0.031), Grok-high against Composer 4 : 6 (p = 0.75), Grok-high against Sonnet 2 : 10 (p = 0.039), Opus against Sonnet 0 : 12 (p = 0.0005); no correction for multiple tests. An earlier ranking in tiers and an unpaired test were withdrawn after this paired recount.

• The frontend class is the finding. Every seat except Haiku accepts 14 of 15 broken frontend runs. A judge that reads code cannot see that a page is broken; that verdict belongs to the browser checks of §11.2. A deterministic screenshot lifted every QA seat to 8 of 8 on visual defects in the vision bench.
• The product's QA order is a ladder for when seats are unavailable — Cursor Grok-4.6-high, Composer 2.5, Opus, Sonnet, Gemini flash, Codex — and is not the false-accept order above. Read the table as an order, not as tiers. Gemini was measured only on backend and frontend (the lanes for docs, infra and, for the default arm, design stopped with no verdict, and the cause was not recorded), and on what was read it is the worst seat in the table: flash accepted 27 of 45 readable broken runs, the default arm all 15 broken backend runs, so its place late in the ladder is by evidence and not only by caution. The five rows above it are exact; Gemini’s are not comparable with them, because most of its cells are missing. Codex was not run on the second set.

11.4 The cost against a raw Claude session, cell by cell

Revised the same day, after a fact-check. The first version of this section printed 10.7× and the site said “10×”. It compared each winner with the Claude arm that made the ratio largest, it counted a 95 % interval as if the four S cells were independent, and it set raw single-seat sessions, which carry no QA, against raw Claude while the product ends every run in QA. This is the corrected section; the site said 3× on this basis until the fourth-edition revision (§11.7).

A cell is one job × size × domain × suite that holds a raw Claude reference with audited quality and at least one non-Claude arm with at least two audited samples. The winner of each cell is picked by the router’s own rule (§11.1), five subscription seats only. The reference is the cheapest raw Claude arm (Haiku, Sonnet or Opus) that keeps at least 80 % of the cell’s best measured quality, so that the ratio is not made larger by choosing a dearer Claude; the ratio is the reference’s dollars per run divided by the winner’s, API-equivalent list prices.

What the number is. The median is 5.2× and the cells run from 2.7× to 15.3×; dropping any one cell moves the geometric mean between 4.9× and 6.5×. No interval is printed: the four S cells are one synthetic small-ask family (twelve instances) where every arm scores about 1.0, they are not independent, and a bootstrap that treats them as seven cells is narrower than the evidence (the first version printed 7.0–15.2×). Without the family there are three cells, 2.7×, 3.8× and 3.6×, geometric mean 3.3×. Weighted by what a bill would be (dollars summed over the runs of each cell) the saving is 3.9× over the six cells with n of at least 3.

What the number leaves out: QA. Six of the seven winners are one seat running alone, filed as raw with no QA check, and the Claude arm they are set against has none either; only SWT-bench uses an antiloki pair (Gemini × Composer, 3.6×). The product ends every run in QA, and QA costs money: on the S2 suite raw Composer is $0.0488 a run and Composer × Composer with its QA check $0.086, 1.76× more; on the eight-module task Composer alone is $0.0671 and Composer × Composer × QA $0.1017, 1.5×. Dividing the 5.7× by 1.76 gives 3.2×. That division is an estimate, not a measurement — no end-to-end run was watched binding a solo seat (§11.2, §11.6), and the QA overhead of the other cells was not measured. The one cell where the whole pipeline was priced is the eight-module task: $0.1017 against raw Sonnet at $0.330, 3.2×. The 3× the site carried on this basis was that estimate rounded down; it sat under the 5.7× of the QA-free comparison, and under the 3.3× of the three cells outside the synthetic family.

Limits. n is 3 per arm in five cells and 10 in SWT-bench; the eight-module cell is one Codex run (two audits) against one Sonnet run (the first version used Opus, mean of 2, $1.151, which is inside the tolerance but not the cheapest Claude arm that is). The same-quality swap on SWT-bench (Codex × Composer, 9 of 10 like raw Sonnet, $0.157) is 2.4×; the 3.6× above is Gemini × Composer, which scored 8 of 10 (merges::separated is false at n=10, so “same quality” means inside the tolerance and not separated at these samples, not identical). The 6.7× of the first headline was raw Qwen3.6-plus ($0.0558) against raw Sonnet ($0.3737) on that same SWT cell — not an antiloki arm, a metered model that is off by default — and is withdrawn; the five-seat equivalent is the 3.6× above. Composer, Gemini and Codex dollars are estimated from reported tokens at list rate (Cursor and Antigravity report tokens, not dollars); the Claude references ran 2026-09-22 to 28. The eight-module row prices Codex low alone ($0.1215, 2.7×), the winner when the table was derived; the five-seat re-run of the same task later measured Composer alone at $0.0671 (4.9× under raw Sonnet) and the QA pipeline at $0.1017, and with Composer alone in that row the geometric mean before QA would be 6.2×. S2 and the S cells also disagree: raw Haiku $0.0437, Composer alone $0.0488 and Composer × Composer with QA $0.086 on S2, while Composer alone is priced about ten times lower in the S cells for asks of similar size (cause not found; Composer dollars are token estimates at list price). Time is not a headline axis: matched cells are about equal, except the eight-module cell, where Composer alone took 61 s against Opus’s 278 s (the Opus time moves between 112 s and 303 s from day to day).

The weekly limit. The site states +112 %, and restates what it is (fourth-edition revision). It is the paired task result of §11.7 — on the eight-module task, $0.1554 for the arm where Claude plans and judges against $0.3297 for raw Sonnet, 2.12× — and not a measured percentage of anyone’s weekly limit. The earlier derivation, 1 / (1 − 0.528) for the share of one machine’s tokens that ran off the frontier seat, is withdrawn: an independent audit could not reproduce the share, and the cuts of the record differed (+471 % over the whole mix, +106 %, +118 % and +122 % over three others). By dollars the gain over the recorded mix is +33 % to +45 %, which is the range to quote for a mix of work and not for a task.

The eight-module task, re-run on the five seats (37 checks, 2026-09-28/29; the page’s table): Composer alone $0.067, 61 s, 36.3 of 37 (n=3); Composer × Composer with a Composer QA check $0.102, 95 s, 36.0 (n=3); Sonnet × Composer $0.155, 106 s, 36.8 (n=5); Codex × Composer $0.165, 137 s, 36.7 (n=3); Gemini × Composer $0.184, 134 s, 36.3 (n=5); Grok-high × Composer $0.292, 199 s, 36.3 (n=3); raw Opus $1.151, 278 s, 37 (n=2, the 2026-09-22/23 reference); raw Sonnet $0.330 (n=1, 0.973).

11.5 Does the X-ray map help an agent? Three experiments

Each experiment crossed three agents (Cursor Composer 2.5, Codex, Gemini via Antigravity) with a small and a wide change and three scenarios — the repository only, the X-ray intel only with the agent told not to explore, or both. One run per cell; graded statically, nothing compiled; Gemini reports no tool-call telemetry.

• A wide TypeScript rename (React Compiler, 30 files, 75 places, real baked intel): intel alone (the agent told not to explore) against the repository alone was 31 % faster (75 s to 52 s on average) and used 48 % fewer input tokens (135k to 70k) at the same pass rate (3 of 3 each); both together was about 15 % faster and 5 % lighter.
• A small change: no measurable difference in quality or time on a 13,000-line TypeScript repository with hidden tests (45, 46 and 44 of 48 tests; 178, 180 and 182 s), and in that repository the intel cost more tokens on the wide change for Codex (472k to 1.15M), because the pack is sent again each turn. In the React Compiler experiment's small change the intel-only agent passed 0 of 3 (the repository alone passed 3 of 3; the two together failed once, Gemini), because the index had excluded the file that needed the change — an agent told not to explore inherits the index's blind spots. So the 31 % and 48 % hold for the wide rename only, and the same arm was the worst one on the small change. That is why the intel now says which files it could not read (excluded, unsupported language, parse failed, changed since the index) and never certifies completeness.
• Flow-typed JavaScript was unreadable to the indexer during the second experiment, so that run used hand-built intel and is not reported as product evidence. The reader was fixed on 2026-10-01 (on a copy of facebook/react, files skipped went from 1,160 to 0 and import edges from 1,689 to 6,510); the experiment has not been re-run since.

11.6 What the third edition did not run

• No Claude arm was re-measured in this edition; the Claude references date from 2026-09-22 to 28.
• Codex on the second QA defect set, the Gemini lanes for docs and infra, and any larger-than-L (XXL) run on the five seats. The 6.4× at XXL in the first edition used a Claude brain with DeepSeek hands and is not carried over.
• No new fresh reference arm on the eight-module task, and no end-to-end run watching the router bind a solo seat. The cost cells of §11.4 carry no QA step; its overhead was measured on S2 and on the eight-module task only, and the QA-inclusive 3.2× is an estimate from it.
• A weekly-limit figure per task. The router's own saved-money ledger over 82 settled runs (3.4× on task tokens, 1.9× once planner, screening and QA costs are added) is not used: its baseline is the same tokens priced at Opus's list rate, not a measured raw run, and 99.5 % of its task tokens are Gemini's.

11.7 Fourth-edition revision: the headline reference is Opus

How the headline is defined: the best measured antiloki result against raw Claude Opus on the same task and the same checks, not against the cheapest Claude model. The reasoning is that Opus is the Claude model a subscriber on the top plan reaches for. All figures below come from the merge record (merge_scores) and are the eight-module task of §11.4 (37 checks, ministore); the row ids are in the merge record.

The headline is the pipeline with its QA step: 11.3×, rounded down to 10×. Its quality is 0.973 of 1.0, inside the one tolerance of §11.1 and not equal to raw Opus; it passes 36.0 of the same 37 checks on average. Opus is not stable from day to day: the ladder run of 2026-09-22 recorded $0.61 for the same task, and against that run the pipeline is about 6.0×, so the honest range is 6×–11× and the headline is the best measured cell. The weekly-limit figure, +112 %, is the Sonnet-planned arm against raw Sonnet, $0.1554 against $0.3297 = 2.12×; with n=5 against n=1 and list-price dollars standing in for an allowance the vendors do not publish, it is a paired task result and not a share of a quota window.

The headline arm is the nearest measured arm to the shipped path, not the shipped path itself: for an ask of size L the shipped path is solo (no planning call) plus QA, and that exact path has not been run end to end. What the headline does not include: the separate screening call the shipped path makes before the plan ($0.006 on Gemini, $0.014 on Haiku), and any end-to-end run of the solo path in real use (§11.2). On the small-ask suite S2 the picture is different: raw Haiku $0.0437, Composer alone $0.0488, Composer × Composer with QA $0.086, so on a small ask there is no saving against Haiku. That is why the S family is not the evidence for the headline.

AReproducing a row

bench/merges/run.sh L claude-deepseek false     # one arm: pair · size · QA — clones ministore, drives /cockpit/goals/plan,
                                                  # audits with bench/ladder/audit-dir.sh, POSTs the audit to /settings/merges/record
bench/merges/matrix.sh                            # the built-in pairs × S · L, own serve on :41941, its own home
bench/merges/jobs.sh · jobs2.sh · jobs3.sh · jobs4.sh   # QA-on runs · blueprint (tests) · a wave of 3 (report) · raw singles + six pairs
GET  /api/v1/settings/merges                      # the table, weights included
POST /api/v1/settings/merges/record               # an Outcome; audit_only joins an existing row
GET  /api/v1/settings/audit                       # every call: model · ms · tokens · deterministic cost

RReferences

1. [1]LMSYS, RouteLLM: an open-source framework for cost-effective LLM routing, July 2024.
2. [2]Martian's tool automatically switches between LLMs to reduce costs, TechCrunch, Nov 2023.
3. [3]Not Diamond — model routing for coding agents.
4. [4]OpenRouter more than doubles valuation to $1.3B in a year, TechCrunch, May 2026; Sacra for volume.
5. [5]aider, Separating code reasoning and editing, Sept 2024.
6. [6]aider, R1 + Sonnet set SOTA on aider's polyglot benchmark, Jan 2025; R1 + V3 benchmark.
7. [7]Cline: Plan & Act; Roo Code: using modes.
8. [8]Wang et al., Mixture-of-Agents enhances large language model capabilities, 2024.
9. [9]Zaharia et al., The shift from models to compound AI systems, BAIR, Feb 2024.
10. [10]mergekit — what "model merging" means in ML, and what this paper does not do.