Subscribe to Newsletter

Chapter 3: Set budgets that work

Catch accidents without making productive work wait.

Chapter 3 of 6 · 3 sections
Learning objectives
  • Separate an accident from sustained spend, and give each one its own control.
  • Design a daily guard with self-serve acknowledgment alongside a monthly limit in approved, expiring tiers.
  • Define the emergency path and the downshift rule before the first incident, and log every use of them.
3.1

Accidents versus sustained spend

A runaway script and a six-week migration can both blow past an engineer’s usual spend. The script needs an alarm. The migration needs an owner and a budget. If you give those two cases the same control, you will get the worst of both: accidents that run for hours and real work that waits for approval.

3.2

Separate accidents from sustained spend

A daily guard interrupts an unusual burst and asks a human to look at it. A monthly limit governs sustained usage. Either way, show engineers what they have spent across every tool before a limit turns into a surprise.

Databricks’ budget design came out of a failure. Under a single monthly cap, 500 to 1,000 engineers hit the limit every month and filed tickets to raise it, and every raise was permanent.

The replacement lets engineers clear daily increments themselves. Larger monthly allowances need a manager’s approval and expire with the project. Raising the monthly tier raises the daily increment with it. The pattern keeps ordinary work moving while making unusual spend deliberate. Uber’s $1,500 per employee per tool, with a dashboard and an exception process, is the blunter cousin of the same idea.

DecisionDaily guardMonthly limit
PurposeCatch an unexpected burst.Approve a sustained level of spend.
When reachedShow the recent spend. Ask a human to inspect it and resume intentionally.Ask the manager to approve a named project and an expiry date.
ScopeAggregate usage across tools where metering permits.Include every tool in the team’s overall envelope.
ResetReturn to the base daily allowance on a published schedule.Revert temporary increases when the project ends.
TWO BUDGETS, TWO KINDS OF WASTE DAILY · RUNAWAY GUARD 90% LIMIT self-serve · unlimited acks · resets nightly You’ve spent $X today, $Y headroom. One click raises the limit one increment. This is intentional effective_limit = min(spent + one daily increment, monthly max) Raise the monthly tier and the daily increment scales with it, so the runaway guard stays calibrated. MONTHLY · EXTRAORDINARY SPEND Base ~2x ~5x Effectively unlimited each step: manager approval · project-scoped · 1 / 3 / 6 months, then reverts BELOW SUSPENSION Downshift to cheaper model Suspend (last resort)
Scroll sideways →Source: Databricks.
3.3

Make the exception path part of the policy

Do not wait for the first production incident to decide who can override a limit. Define the emergency path now, log every use of it, and review it afterward. A security incident may justify an immediate suspension. Ordinary overspend deserves an explanation and a way back.

Downshifting means offering a cheaper model when a limit is reached. Use it wherever that model can still do the work safely, and tell the engineer what changed. Silently swapping model capability is the fastest way to turn a budget surprise into a debugging surprise.

MAKE THE CALL · 4 / 5

A runaway job hits its daily guard

An unattended job has burned through today’s allowance. A developer also has a legitimate migration to finish. Which policy catches the accident and permits intentional work?

Choose an answer. You can retry after reading the explanation.

budget-policy.md
Daily guard: [threshold and increment based on observed usage]
Acknowledgment: [where a human inspects spend and resumes]
Monthly base: [allowance] | temporary tiers: [allowances]
Override owner: [manager] | project: [name] | expires: [date]
Emergency path: [on-call owner and audit trail]
Downshift: [eligible tasks and available model]
Review: weekly exceptions; monthly policy calibration
END OF CHAPTER 3 · PUT IT TO WORK

Test the unblock path

Sit with one engineer and walk through three cases: a runaway job, a planned heavy project, and a production incident. Time how long each one takes to resolve. Then rewrite any step that depends on finding an approver nobody has written down.

Key terms

Daily guard
A threshold that interrupts an unusual burst and asks a human to inspect it before work resumes.
Monthly limit
The control that governs sustained usage, raised in coarse tiers with manager approval and an expiry date.
Downshifting
Offering a cheaper model when a limit is reached, with the change made visible to the engineer.
Break-glass path
The named emergency route around a limit, logged every time it is used and reviewed afterward.

Further reading

  1. DatabricksHow Databricks manages its own coding agent spend with Unity AI Gateway budgets

    The daily and monthly design this chapter follows, including what failed first.

  2. TechCrunchUber caps employee AI spending after blowing through budget in four months

    The blunter version of the same idea, with a dashboard and an exception process.