Your Agent Loop Has a Compounding Cost Problem

July 14, 2026

agent-loops context-engineering llm-costs token-debt ai-infrastructure

A 30-step AI workflow doesn’t cost 30 times a single API call. It costs closer to 465 times, because the model re-reads every previous message on every single turn. That’s not a quirk you can optimize around at the margins. It’s baked into how stateless LLM APIs work, and if you’re building anything non-trivial with agents, it’s quietly eating your budget in a way that compound interest would recognize immediately.

I’ve been thinking about this as a form of debt, and I think the framing matters more than it might seem.

The Math Most People Miss

Here’s the actual shape of the problem. Each turn of an agent loop, the API doesn’t “remember” where it left off. It re-reads the entire conversation history from scratch. So if your workflow has 30 steps, and each step adds roughly 500 tokens of context, you’re not sending 30 × 500 = 15,000 tokens total. You’re sending 500 + 1,000 + 1,500 + … + 15,000, which is a triangular number series. That sum is 232,500 tokens — roughly 15.5x the naive estimate, or about 465x the cost of a single 500-token call.

At current rates on something like Claude Sonnet ($3.00 per million input tokens), a single 30-message agent loop runs around $0.70 in context re-reading alone. Not terrible in isolation. But if you’re an enterprise running 500 conversations a day, analysts estimate roughly 70% of your API spend is the model re-reading the same static context it already processed on the previous turn. You’re not paying for intelligence at that point. You’re paying for a very expensive re-read.

This is O(N²) scaling against what most people are treating as an O(N) problem. And the standard response — just expand the context window — makes it worse, not better. Longer contexts increase latency, cost, and what researchers are calling “context rot”: the gradual degradation in model recall accuracy as the window bloats. You can’t brute-force your way out of a quadratic.

Token Debt Is Real Debt

Technical debt is the idea that shortcuts taken now accumulate interest over time, slowing you down later. Token debt works the same way, except the interest compounds immediately, per-call, in dollars.

The discipline engineers apply to actual technical debt — tracking it, budgeting it, making explicit tradeoffs about when to take it on — is exactly what’s missing from how most teams think about agent context. People design workflows based on what works functionally, then get surprised when the cloud bill arrives. They’ve been treating tokens as approximately free, which they aren’t, and they’ve been treating context accumulation as a necessary feature of long-running agents, which it isn’t.

Some of the more thoughtful teams I’ve seen approach this are doing something they’re starting to call context engineering: treating the token budget as a hard constraint from the start, the same way you’d treat memory in an embedded system. That means deliberately resetting context windows at checkpoints, storing persistent state externally in databases or files, and spawning clean sub-agents instead of letting a single thread balloon. It also means using prompt caching aggressively — provider-level caching on static system prompts can cut prefix costs by up to 90%, which starts to look very important once you understand the triangular math.

The broader infrastructure reality makes this more urgent, not less. OpenAI is projected to lose $14 billion on a non-GAAP basis in 2026, with inference costs alone running $7 billion — a 3.5x increase from the prior year. The industry is spending something like $725 billion on AI infrastructure in 2026 against revenues that don’t yet justify it. That financial pressure flows downstream to pricing, and the teams that have built workflows assuming cheap-to-free inference are going to feel it.

The Analogy That Keeps Sticking

In my experience, the teams that manage technical debt well are the ones who made it visible. They have dashboards. They talk about it in planning. They know which parts of the system are carrying the most of it.

Token debt is almost entirely invisible right now. You see a monthly API bill but not a per-workflow breakdown. You see aggregate token usage but not which agent loops are the worst offenders. You definitely don’t see the 70% that’s going to re-reading context you already paid to process last turn.

The tooling to make this visible is starting to emerge — platforms that break down AI spend by team, by workflow, by efficiency — but it’s not standard yet. Until it is, most teams are flying blind, which is precisely when debt compounds fastest.

The question I keep returning to: if you knew, in real time, what percentage of your agent loop’s cost was re-reading versus generating new reasoning, would you build differently? I think most people would. The fact that we’re not seeing that number by default tells you something about how the industry has been pricing this, and probably how it will have to change.