Chapter 5: Remove the waste

Learning objectives

Order a request so the provider can reuse the prefix, then confirm hits in the cached-token fields.
Work out the break-even point for a cache write at your provider’s write and read multipliers.
Test every context change per task rather than per call, and read rereads as a signal that you cut something the agent needed.

5.1

Look inside the agent’s requests

Once routing and budgets are working, look inside the agent’s requests. Repeated instructions and verbose tool results can make every turn expensive, and the fix is not obvious from the outside. Start with evidence from real sessions. A shorter prompt is not automatically a cheaper task.

5.2

Reuse context before you remove it

Prompt caching lets a provider reuse the opening of a request when it matches one it has already seen. That reusable opening is called a prefix. Put your stable instructions and tool definitions first and the changing task content last, because any byte that changes above a breakpoint, even a timestamp, invalidates the cache.

Then check the actual cached-token fields in your usage records. A cache that never gets hit is just a setting. Coinbase moved its hit rate from 5% to 60%, and that was worth more than any model switch.

The rules differ by provider. OpenAI’s documentation describes automatic prefix matching with no write premium. Anthropic’s documentation explains explicit cache writes, retention windows, and model-specific read rates. Technspire’s comparison lays the two side by side. Do not assume the same discount across providers, or across models from the same provider.

5.3

Calculate the break-even point

Suppose a cache write costs 1.25 times a normal input token and a later read costs 0.1 times. Those are the standard five-minute write and the common read multiplier on one major provider, though some models differ.

cache-break-even

Two requests without caching: 1 + 1 = 2.00
One write and one cache hit:  1.25 + 0.10 = 1.35

Saving on that reusable prefix: 0.65 / 2.00 = 32.5%

For N later reads, caching pays when:
write_multiplier + N × read_multiplier < 1 + N

At those rates, a single later hit pays back the write premium. With a 2-times write and a 0.1-times read, you need two. This only prices the reusable prefix. New input and generated output still cost what they cost.

MID-LESSON CHECK

1. Where should stable instructions and tool definitions sit in a cacheable request?
2. A cache write costs more than a normal input token, but a later read is cheaper. When does caching pay off?
3. Why shouldn’t you assume the same caching discount across providers or models?

5.4

Trim the parts that do not help the task

Change to test	Reason	Evidence to watch
Start a new session when the subject changes	Unrelated history can make future turns heavier.	Lower full-task cost with the same acceptance rate.
Retrieve specific files and focused log excerpts	Generated folders and repeated logs can swamp useful context.	Fewer irrelevant tokens without extra retrieval loops.
Summarize long sessions at a deliberate boundary	Compaction replaces old context with a shorter summary.	No lost requirements or repeated investigation after the summary.
Set retry and step limits on unattended work	A repeating failure needs intervention.	Less runaway spend with a clear handoff when the limit is hit.

5.5

A smaller response can create a bigger bill

GitHub found that compressing every tool output could make tasks cost more, not less. When the removed detail mattered, the agent went back and fetched it again, added turns, and carried more context forward. The saving on one call created extra work later.

Their clean wins came from formatting nobody consumed. Stripping line-number prefixes from file reads cut inference cost about 3% in production, with no change in quality.

✗ Optimize per call

Compress every tool output
→ Tokens per call: down
→ Agent re-reads what it needs
→ Tokens per task: up

✓ Optimize per task

Preserve source-like output and diffs
Reorganize search results, drop nothing
Compress only install/build/test noise
Monitor the recovery path as a signal
→ Tokens per task: down

Measure the complete task after every change, in every workflow where it runs. Watch for rereads and repeated commands. That is the agent telling you an optimization removed something it needed.

END OF CHAPTER 5 · PUT IT TO WORK

Run one controlled change

Choose one noisy tool or one cache setting. Compare it with the current version on the same task set. Keep the change only if total cost falls without worse acceptance, longer repair time, or higher latency.

Key terms

Prompt caching: Provider reuse of a matching request beginning, so repeated instructions are billed at a lower read rate.
Prefix: The stable opening of a request, such as system instructions and tool definitions, placed before the changing task content.
Breakpoint: The position that ends a cacheable prefix. Any byte that changes above it, even a timestamp, invalidates the cache.
Compaction: Replacing older context with a shorter summary at a deliberate boundary.
Break-even: The number of later cache reads at which the write premium pays for itself.