Benchmark
97% of Opus's accuracy for 66% less, 22% faster. The same 320 public tasks, three ways, at the same time.
What we ran
320 tasks, a fixed and seeded sample of four public datasets, each sent the way a company app sends it: with its own system prompt, streamed, in Anthropic's Messages format, the one Claude Code uses.
| Dataset | Tasks | What it is | Checked by |
|---|---|---|---|
| MMLU-Pro | 120 | expert multiple choice in business, law, health, economics, engineering and computer science | the answer key |
| GSM8K | 60 | grade-school math word problems | the answer key |
| MATH-500 | 60 | competition math, every difficulty level | the answer key |
| HumanEval | 80 | Python functions | the dataset's unit tests, run |
Every task went three ways at once, in the same order:
- Always Claude Opus 5.5: straight to Opus, at its own default effort. What a team pays without Sovereign.
- Sovereign: through a new Sovereign gateway with nothing learned yet, asking for Claude Opus 5.5.
- Always Claude Haiku 5.5: straight to the cheapest model, to show what blind savings cost.
No model graded the benchmark: an answer is right only when it matches the key or passes the tests.
Results
| Accuracy | Cost per 1,000 tasks | Median response | First words | MMLU-Pro | GSM8K | MATH-500 | HumanEval | |
|---|---|---|---|---|---|---|---|---|
| Always Claude Opus 5.5 | 95.6% | $7.77 | 3.6 s | 2.8 s | 90% | 98% | 98% | 100% |
| Sovereign | 92.8% | $2.61 | 2.8 s | 1.6 s | 86% | 97% | 93% | 100% |
| Always Claude Haiku 5.5 | 90.6% | $0.25 | 2.3 s | 1.5 s | 81% | 98% | 90% | 100% |
Sovereign kept 97% of Opus's accuracy (-2.8 points) for 66% less, and answered 22% faster at the median, with first words 43% sooner. Its answers came from Claude Sonnet 5.5 (51%), Claude Haiku 5.5 (38%), Claude Opus 5.5 (7%), GPT-6 Luna (2%).
Reading it fairly
- Costs are the customer's bill: what each request cost at list prices, reasoning included.
- Sovereign started cold. A gateway that has seen a company's traffic already knows how each model does on that work; this run started with nothing learned.
- Public benchmarks aren't your work. They're a fair, repeatable test, but the result that matters is the one on your own traffic, which is on your Overview from the first day.
Run on October 21, 2026 through OpenRouter. Want to run it against your own traffic? Talk to us.