Replit Agent Benchmarks: 72% on DeepSWE at $2.11 a Task, and What That Number Hides
Disclosure: this post contains an affiliate link to Replit. If you sign up through it we may earn a commission at no extra cost to you. Every figure.
On 29 September 2026 Replit published Free the models: Harness design at the frontier, an engineering post about how Replit Agent decides when to hand work to subagents. Buried in it is something more useful than the architecture argument: actual cost-per-task numbers, plotted against benchmark scores.
Almost nobody publishes this. Vendors publish scores. Scores without cost are close to meaningless when you can buy a higher score by spending more, which is exactly what the data below shows.
The numbers
DeepSWE v1.1 — long-horizon changes to live open-source repos
| Configuration | Score | Cost per task |
|---|---|---|
| Replit Agent (Max mode, GPT-6 Astra core) | 72% | $2.11 |
| GPT-6 Astra alone (mini-swe-agent), low effort | 67% | $1.60 |
| GPT-6 Astra alone (mini-swe-agent), xhigh effort | 74% | $4.43 |
| “Sidekick” architecture (one long-lived worker) | 61% | $1.34 |
Terminal-Bench 4.0 — multi-step work done entirely from a shell
| Configuration | Score | Cost per task |
|---|---|---|
| Replit Agent (Max mode) | 49% | $2.53 |
| GPT-6 Astra alone, low effort | 42% | $2.25 |
| GPT-6 Astra alone, xhigh effort | 60% | $5.86 |
| “Sidekick” architecture | 33% | $1.84 |
DeepSWE figures are the mean of 4 repetitions over 113 tasks; Terminal-Bench figures the mean of 4 repetitions over 63 tasks with GPU tasks excluded. Astra baselines are the published leaderboard numbers for Astra alone in mini-swe-agent.
What this actually tells you
Read the two tables as a price list rather than a scoreboard and the picture is clear. On DeepSWE, buying the extra two points over Replit Agent costs you 110% more per task — $4.43 against $2.11. On Terminal-Bench the gap is wider in score terms, eleven points, but it costs 132% more. Going the other way, the cheap sidekick setup saves about 36% per task on DeepSWE and gives up eleven points doing it.
That is the real decision for anyone running an agent at volume: not which configuration is best, but how much a percentage point is worth to you. If you run a thousand tasks a month, the difference between $2.11 and $4.43 is $2,320. Whether two points of benchmark score is worth that depends entirely on what the tasks are.
The honest caveats
- These are Replit’s own benchmarks, run by Replit, published by Replit. No independent party has reproduced them.
- The “sidekick architecture” baseline is Replit’s own construction of a competing design, not a competitor’s product. It is a reasonable comparison to make, and also one where Replit chose both sides.
- Costs are per benchmark task, not per Replit credit. They do not map directly onto what you will be billed — check Replit’s current pricing for that.
- Replit does say it ran the Agent “exactly as it ships to users, with no changes to the prompting or harness”, which is a meaningful claim and more than most vendors offer.
The architecture argument, briefly
Replit’s thesis is that model routers are structurally limited: a router choosing between models can never be smarter than the model it picks. So instead of routing, Replit lets the main agent decide three things at every step — how hard to think, whether to hand work off, and who to hand it to.
Four harness primitives
- Domain-aware subagents — read-only explorers, browser testers, reviewers and a design subagent for slides and UI, each on whichever model is strongest at that job.
- Tiers and effort — small, standard and large, each with an effort level; a mechanical rename goes to small at low effort, a stubborn bug to large at high.
- Reusable subagents — the core loop can return to a subagent it already briefed rather than starting over.
- Dynamic effort tuning — effort changes mid-turn without a cache miss on the GPT-6 family, so the system escalates on work in progress rather than once on the request.
The table nobody is quoting
The most interesting data in the post is not the benchmarks at all. It is Replit’s production telemetry on how different models behave when you give them the freedom to delegate.
| Behaviour, at medium effort | Fable 5 | Fable 5.1 | GPT-6 Astra |
|---|---|---|---|
| Turns that dispatch a subagent | 32% | 21% | 36% |
| Turns that hand work to a general worker | 0.9% | 2.3% | 20% |
| Dispatches that return to an existing subagent | 17% | 29% | 42% |
The middle row is the striking one. Fable models at medium effort almost never hand implementation work to a general worker — they send out explorers and reviewers and keep the coding for themselves. Astra hands off roughly one turn in five. Replit describes it as the first model it has seen routinely delegate without being told to, and notes the return rate has risen with every model generation.
If you are building your own agent harness, that difference matters more than either benchmark. A model that delegates on its own needs a different harness from one that does not.
Worth reading if
Read the original if
- You are choosing between agent configurations and cost per task is a real constraint.
- You are building a harness and want a working argument against routers.
- You want to see what production delegation telemetry looks like — very few companies publish it.
Replit Agent in Max mode is what was benchmarked, so that is the setting these numbers describe. If you want to try it, it is at replit.com.
What we could not verify
Open questions
- Any independent reproduction of either benchmark result.
- How benchmark cost per task translates into Replit credits or a monthly bill.
- What the sidekick baseline would score if built by someone with an interest in it doing well.
- Whether the Fable / Astra telemetry holds outside Replit’s own user base and task mix.
Related on DailyAIPedia
- Replit, Softr and Bubble: what shipped in September 2026
- Claude Opus 5.5 pricing: what 40% cheaper really means
- BytePlus September 2026: Seedream 5.0 Flash and Seedance 2.5 Premium
- ChatLLM by Abacus.AI review: 100+ models for $10 a month
- 21st.dev review: 12,000 shadcn components, an MCP server and no monthly plan
- Replit’s October 2026 update: GPT-6.1 Sol, Claude Sonnet 5.5 and the Jev decision model