Chapter 2: Choose your models

Turn model choice into a repeatable engineering decision.

Chapter 2 of 6 · 5 sections

Learning objectives

Build an internal eval from your own pull requests, including the failures and the awkward cases.
Compare candidate models on cost per accepted task rather than on public benchmark position.
Write a routing rule an engineer can explain in one sentence, with a named escalation trigger and a retry limit.
Pick the smallest implementation your traffic justifies: task-level defaults, a worker with escalation, or a request-level router.

2.1

Model choice as a repeatable engineering decision

The most capable model is a useful tool. Making it the default for every task is a purchasing decision dressed up as an engineering preference. In Ramp’s June 2026 index of 70,000 businesses, the top 1% spent $7,449 per employee per month on AI while the median spent $11.38. A lot of that gap is the default model.

A frontier model sits at the high end of capability. An efficient model meets a specific task’s quality bar at a lower cost. The same model can be efficient for one kind of work and wasteful for another, which is why the label belongs to the task, not the model.

2.2

Build a small test from your own work

An evaluation, or eval, is a repeatable set of tasks with a defined pass condition. Build yours from recent pull requests that look like your real workload, and include the failures and the awkward cases. A collection of easy wins will flatter every candidate you test.

Start with a sample you can actually run. Give each candidate the same repository state, the same instructions, and the same tool access, and keep the expected answer out of the prompt. Record acceptance, cost, latency, and how long reviewers spent repairing the output. Then repeat enough tasks to know whether the result holds.

model-evaluation.md

Task class: bounded refactor
Starting state: pinned repository commit
Pass condition: intended change works; existing tests pass
Review bar: no unintended behavior or security regression
Compare: accepted-task cost, success rate, latency, repair time
Rollout: limited pilot with an explicit rollback condition

2.3

Sometimes the right upgrade is no upgrade

Sometimes the answer is no. Databricks reports that Stripe ran a new frontier release through its internal evals, found higher cost without a meaningful quality gain, and declined to roll it out.

Coinbase went the other direction and made open-weight models the default behind its gateway, at roughly a quarter of frontier input prices, with an escalation hatch to stronger models. Its bill nearly halved. These are examples of a decision process, not a shopping list.

✗ Public benchmark as the decision

New model tops the leaderboard
→ Roll it out org-wide
→ Cost and quality on your code: unknown

Benchmarks don’t predict your workload.

✓ Internal eval as the decision

New model ships
→ Run it on 200 tasks from your PRs
→ Compare cost per accepted task to incumbent
→ Adopt, decline, or adopt for a task subset

Sometimes the answer is no.

MAKE THE CALL · 2 / 5

Choose the model that earns the rollout

Each model attempts the same 100 tasks. Model A costs $80 and produces 80 accepted results. Model B costs $120 and produces 90. The minimum acceptable pass rate is 80%. Both meet the same security bar and require the same review time. What would you pilot for routine work?

Choose an answer. You can retry after reading the explanation.

2.4

Turn the result into a routing rule

Routing assigns a task to a model. Escalation moves it to a stronger model when a defined condition is met. Start with a rule your engineers can explain in a sentence. A clever router nobody understands is just another system to debug.

Treat the table below as a starting policy to test, not a standard to adopt. Set your own retry threshold. Escalating after two identical failures is usually smarter than letting an agent repeat the same mistake twenty times.

Task type	Start on	Escalate when
Docs, explanations, formatting	Efficient	Answer misses repo-specific context
Unit tests, summaries, bounded refactors	Efficient	Tests keep failing or change crosses system boundaries
Routine code review	Efficient	Change is security-sensitive or architecturally significant
Repo exploration	Efficient	Task needs broad cross-system reasoning. Start with focused retrieval.
Background automation	Efficient	Reliability drops below accepted threshold
Architecture, migrations, security-sensitive work	Frontier	Start strong; a wrong call costs more downstream than the tokens saved
Subtle debugging, ambiguous failures	Frontier	Keep investigation bounded with focused logs

MAKE THE CALL · 3 / 5

Do the routing math

Five routine tasks each get a $0.40 attempt. Four pass. One fails and succeeds after a $2.00 escalation. What is the model cost per accepted task?

Choose an answer. You can retry after reading the explanation.

Routed

$0.80

per accepted task. Five bounded refactors. Efficient attempt $0.40 with 80% pass rate; escalate to frontier at $2.00 on failure. $4.00 total.

All-frontier

$2.00

per accepted task. Same five refactors, every one on the frontier model. $10.00 total.

Five bounded refactors, total spend

Routed4 pass on efficient, 1 escalates$4.00

All-frontier5 tasks at $2.00$10.00

The arithmetic is illustrative, and Kilo walks through a similar example. It only holds if the cheap attempts did not create extra review or repair work, so count that cost before you declare the policy a win.

2.5

Choose the smallest implementation that works

Approach	Use it when	Watch for
Task-level defaults	You can classify work before it starts. Set one tier per task class.	Misclassified tasks need a visible escalation path.
Worker with escalation	Routine work usually succeeds cheaply. A stronger model can rescue failures.	Failed attempts and handoff context still cost money.
Request-level router	You have enough traffic and traces to justify per-call decisions.	Model switches can lose cache reuse. Validate full-session cost.

END OF CHAPTER 2 · PUT IT TO WORK

Write one routing rule

Pick one bounded class of tasks. Name its default model, its acceptance bar, its escalation trigger, and its retry limit. Pilot that rule with a small group before you touch the company default.

Key terms

Evaluation (eval): A repeatable set of tasks with a defined pass condition, built from work that looks like your real workload.
Frontier model: A model at the high end of capability, priced accordingly.
Efficient model: A model that meets a specific task’s quality bar at lower cost. The label belongs to the task, not the model.
Routing: Assigning a task to a model by rule.
Escalation: Moving a task to a stronger model when a defined condition is met, such as two identical failures.