A Practical Guide, Part 9 of 15: Cost, Latency and Reliability in Production

Co-host: My favourite room. Quick recap first: last time, in Guardrails, Permissions and Safety, we stacked defences for an agent that can act. Least-privilege tools, deny-first rules, sandboxing, deterministic policy checks and a human signing off on the scary stuff.

Host: With indirect prompt injection and the lethal trifecta as the threat model. Today the threat is quieter. It is the invoice, the slow response and the agent that fails at step forty of fifty.

Host: A note on sourcing before we start. Our checked figures today all come from one provider's documentation, Anthropic's for Claude. Where the outline asks about other platforms, we will tell you the shape of the idea and leave the numbers to your own provider's docs.

Host: Picture a team that prototyped a support agent. The demo cost pennies. Then they turned it on for real traffic, and the bill looked like it belonged to a different company.

Host: Let us make that precise. The model is stateless between calls. So on each round-trip you send the system prompt, the tool definitions, every earlier message, every earlier tool result, and then the new tool result on the end.

Co-host: Which means input cost grows roughly with the square of the number of steps, not linearly. Step one sends a little. Step twenty sends everything from steps one to nineteen again. That is the part nobody budgets for.

Host: The steelman for big contexts is that more information means fewer mistakes. The test is whether the model actually uses it. Often you are paying to resend things it stopped looking at ten steps ago.

Host: In practice: model your cost per task as tokens per step times steps, and remember that every step resends the past, so trim the past before you trim the model.

Co-host: Okay, but if I'm resending the same prefix every call, surely the provider can just remember it?

Host: That is prompt caching, and it is the biggest lever we have. On Claude, a cache lives for five minutes by default, and every time you hit it, the clock resets for free.

Co-host: So a busy agent that calls every few seconds basically keeps its cache warm forever.

Host: Exactly, with one trap. The lifetime is counted from when the request starts, so time spent streaming a long response eats into it.

Co-host: Oh, that bites. If a slow tool or a long generation pushes the next call past the window, you pay to rebuild the cache. What if my agent waits on a human, or on a long-running job?

Host: You can ask for a one-hour lifetime instead, priced at twice the base input token rate.

Co-host: So that's a bet. Pay more up front to write, in exchange for not rewriting it if there's a long pause. Worth it if the gaps are long and the prefix is big; wasted if the conversation dies after one turn.

Host: The outline also asks about the price of cache writes and reads in general. Beyond that one-hour figure we do not have checked multipliers today, so look them up for whichever platform you run on before you model it.

Host: What we can describe is the multi-turn behaviour. With Claude's automatic caching, the earlier parts of a conversation are read from the cache and only the newest turns get written.

Co-host: Which only works because caches match on a prefix. Change one byte near the start and everything after it is a miss. So the golden rule: stable stuff first, volatile stuff last.

Host: Worked example in general terms. System prompt, then tool definitions, then fixed reference documents, then the conversation, then today's date and the user's latest message.

Co-host: And the classic mistake is putting a timestamp in the system prompt. Congratulations, you've built a very elegant machine for never hitting your cache. Same with shuffling tool order or tweaking tool descriptions between calls.

Host: What it means for you: order your prompt from most stable to least stable, keep the prefix byte-identical, and match the cache lifetime to the real gaps in your agent's rhythm.

Co-host: Providers usually limit you along a few axes: requests per minute, input tokens per minute and output tokens per minute. Agents with fat contexts hit the input limit long before the request limit.

Host: And on Claude, the limit is a token bucket, so capacity refills continuously rather than resetting on the minute.

Co-host: Which matters for how you retry. There's no magic top of the minute to wait for. Capacity trickles back, so smooth traffic beats bursts.

Host: When a Claude 429 comes back with a retry-after header, you wait that many seconds and then try again.

Co-host: And when there's no header, back off exponentially with jitter, so your thousand workers don't all retry in the same instant and knock each other over again. Is there a case where retrying is just pointless?

Host: Yes. If you hit a monthly spend cap, Claude returns a 429 with no retry-after header, and no amount of retrying works until access comes back.

Co-host: So your retry logic needs to tell those apart, or you'll have an agent politely hammering a locked door all weekend. Alert a human instead.

Host: Here is where caching comes back as a throughput trick. For most Claude models, cached reads do not count toward the input-tokens-per-minute limit. So with a two million limit and an eighty percent cache hit rate, you could push ten million input tokens a minute.

Co-host: That's a fivefold increase in effective throughput from the same limit. Caching isn't just a discount, it's capacity.

Host: And sometimes nobody is waiting at all. Nightly evaluations, bulk classification, backfills. For those there is batching.

Host: Claude's Message Batches API charges everything at half the standard price. A batch can hold up to a hundred thousand requests or two hundred and fifty-six megabytes, and it expires if it is not finished within twenty-four hours.

Co-host: The trade-off is obvious: you give up latency and interactivity. A batch request can't do a live agent loop where step two depends on step one, unless you run each step as its own batch round.

Host: The outline asks about a second platform's batch terms too. We do not have checked figures for it today, so check those docs rather than assume they match.

Co-host: Now the one that keeps me up at night. A coding agent is forty steps into a migration, the network blips, and it starts over from scratch.

Host: Agents fail differently from ordinary services. A web request either works or it does not. An agent carries state, so a small mistake early on gets built upon, and errors compound.

Co-host: Right, if each step is a little bit unreliable, a long chain of them is a lot unreliable. And a restart doesn't just cost tokens, it can redo side effects.

Host: So the fix is checkpoints. Save the agent's state after meaningful steps, so a failure resumes from the last good point rather than the beginning.

Co-host: And make tool calls idempotent where you can, so a retry after a timeout doesn't send the email twice. Retry the flaky API call at the tool level; don't make the whole agent start again.

Host: Then observability. You cannot read every conversation, and frankly you should not, for privacy as much as sanity.

Co-host: So you trace structure: which tools were called, how long each step took, token counts, cache hit rate, where loops happened, where it gave up. You look at patterns, then drill into a handful of examples.

Host: Last, deployments. A long-running agent may be mid-task when you ship new code. Swap everything at once and you strand it.

Co-host: That's the idea behind rainbow deployments: keep old and new versions running side by side, send new work to the new one, and let in-flight agents finish on the version they started with.

Host: I will be honest about the open problem. None of this tells you whether the agent's judgement was good, only whether it ran. Measuring quality at scale without reading everything is still unsolved, and the papers tend to be tidier than production.

Host: What it means for you: checkpoint state, make side effects safe to repeat, trace the shape of runs rather than every word, and never pull the rug out from under an agent mid-task.

Co-host: So if I could only remember one thing?

Host: An agent resends its past on every step, so the cheapest, fastest and most reliable agent is the one with a stable prefix, a tidy context and a saved checkpoint. Cache it, trim it, save it.

Host: That was part of our series Agentic AI: A Practical Guide. The show notes list every source, and this episode was made with AI voices, which do not get billed per token, but only just. Follow AI Shadow for the rest of the series. Goodbye for now.