What an agent actually costs to run

Clients arrive with a number from a pricing page and a spreadsheet that says this will cost forty dollars a month. The real figure is usually five to fifteen times that, and the gap is entirely in the lines nobody quotes.

The parts of the bill

• Inference. The number on the pricing page. On a typical operations task with retrieved context, 3,000–12,000 tokens per item. Real, and rarely the largest line.
• Retries and fallbacks. Timeouts, malformed output, a second pass at higher effort when confidence is low. Budget 15–30% on top of nominal inference. Systems that do not retry are systems that fail silently.
• Retrieval. Embeddings on ingest, re-embedding when the catalogue changes, vector storage, and the query-time search. Modest per item, but it runs on the whole corpus, not just the queue.
• Evaluation. Regression runs on every change, plus a nightly sample scored against the current set. On mature deployments this is genuinely 10–20% of total compute, and cutting it is how you stop noticing drift.
• Infrastructure. Queue workers, database, logging and log retention. Logging is the one that surprises people: full audit trails on a high-volume task can outgrow the inference bill.
• Human review. The escalation queue is staff time and belongs in cost per task. At a 6% escalation rate and four minutes per escalation, that is often the largest line of all.

A worked example

Order-exception triage, 40,000 items a month, a mid-sized model, 8% escalation:

• Inference at ~7k tokens/item: roughly USD 340
• Retries and second passes: ~USD 70
• Retrieval and re-embedding: ~USD 45
• Evaluation and nightly sampling: ~USD 90
• Infrastructure and log retention: ~USD 120
• Human review, 3,200 escalations at 4 min: ~USD 640 in staff time

Total around USD 1,300 a month, against a manual baseline of roughly USD 9,400 for the same volume. The economics work comfortably — but they work at 1,300, not at 340, and a business case built on the pricing-page number will look like a failure the moment the real invoice lands.

Escalation rate is the main lever

Because human review usually dominates, the highest-leverage optimisation is almost never a cheaper model. Moving escalation from 8% to 4% on the example above saves more than eliminating inference entirely.

That is why we track escalation as a first-class metric from day one, and why most of our post-launch work is retrieval and guardrail tuning rather than prompt or model changes. It is also why "we switched to a cheaper model and costs went up" is a real thing we have seen: the cheaper model escalated more.

Where costs go wrong

Context stuffing. The easiest way to raise accuracy is to retrieve more, and it silently multiplies the token bill. We cap context per task and measure whether each additional chunk earns its place.

Unbounded retries. A retry loop without a ceiling turns one bad upstream day into a four-figure surprise. Every loop gets a hard limit and an alert.

Re-embedding the world. A daily full re-index of a 200,000-SKU catalogue when 400 SKUs changed. Incremental indexing is a two-day change that pays for itself in a fortnight.

Logging everything forever. Full payload retention at high volume becomes the largest line within a year. Tiered retention — full detail for 30 days, structured summary for two years — keeps the audit trail and drops the bill by most of it.

What we commit to

Cost per task appears in the monthly report next to accuracy and escalation rate, and it is the number we are measured on. If it rises without a corresponding rise in volume or accuracy, that is our problem to explain, in writing, before you ask.

We also state the floor honestly: below roughly 2,000 items a month, fixed infrastructure and evaluation costs dominate and per-task economics stop being flattering. At that scale the case has to be made on quality, consistency or turnaround rather than cost — and sometimes it genuinely can be, but we would rather say it upfront than let a spreadsheet say it later.