There’s a failure mode in agent methods that by no means exhibits up as an error. The agent works nicely for twenty turns, then will get subtly worse — forgets a constraint from earlier, repeats a instrument name it already made, contradicts one thing it stated. No exception, no alert, no apparent trigger.
What occurred is that context crammed up and one thing needed to go. The query is whether or not you determined what went, or whether or not a default did.
The place the tokens really go
Earlier than optimizing, measure. An extended-running agent’s context is often 4 issues:
Instrument definitions. Each instrument the agent can see prices tokens for its identify, description, and JSON schema. A well-documented MCP instrument runs 150–400 tokens. Join twelve MCP servers averaging twenty instruments every and also you’re at 240 instruments — name it 30,000 tokens of schema, resident in each single request, earlier than the person has stated a phrase.
That is the commonest and most invisible price. It scales with what you related, not with what the agent makes use of.
Dialog historical past. Grows linearly with turns. Everybody expects this one.
Instrument outcomes. The variance killer. Most calls return a couple of hundred tokens. Then one question returns 400KB of JSON, and until one thing intervenes, that payload sits in context for the remainder of the run — re-sent on each subsequent flip.
System immediate and directions. Mounted and often the smallest piece, which is ironic given how a lot time groups spend modifying it.
The distribution issues strategically: two of the 4 are structural (instrument definitions, instrument outcomes) and may be fastened as soon as, architecturally. The opposite two are linear and might solely be managed. Repair the structural ones first — they’re the place the massive, low cost wins are.
4 methods
Deferred instrument loading
Don’t put all 240 instrument schemas within the immediate. Give the mannequin a compact index — server names and one-line summaries — and cargo full definitions just for the server it decides to make use of.
Typical saving is 60–80% of tool-definition tokens on runs that contact two or three servers, which is most runs.
The fee: an additional spherical journey when the agent picks a brand new server, and barely worse instrument discovery. The mannequin can’t motive a couple of instrument whose schema it hasn’t seen, so in case your agent genuinely wants to match capabilities throughout many servers, the index must be adequate to route on. Write these one-liners rigorously.
Giant instrument response offloading
Set a threshold — 5,000 tokens is an affordable default. Underneath it, outcomes go into context usually. Over it, write the payload to the sandbox filesystem and put a abstract plus a path into context as a substitute. The agent can then grep, slice, or parse the file with code if it wants element.
This converts a catastrophic price right into a small one, and it matches how the information is often used. An agent that fetched 10,000 rows virtually by no means wants all 10,000 in its reasoning; it wants an mixture or a handful of matches.
The fee: the agent wants sandbox entry, and it must be prompted to truly question the file moderately than guessing from the abstract. With out that nudge it’ll typically reply from the abstract alone — confidently and wrongly.
Code mode
As an alternative of the mannequin making 5 sequential instrument calls and receiving 5 outcomes into context, let it write one script that makes all 5 calls, joins the information, and returns solely the ultimate reply.
The intermediate payloads by no means enter context in any respect. For genuinely multi-step information work — cross-referencing two methods, aggregating throughout pages — that is the distinction between a run that matches and one which doesn’t.
The fee: tougher to debug, for the reason that reasoning is inside a script moderately than seen as discrete steps. It additionally requires the agent to be a reliable sufficient programmer for the duty, which is model-dependent. Use it for data-shaping work, not for choices you’ll must audit step-by-step.
Sub-agents
Delegate a bounded subtask to a recent agent with its personal clear context. It burns thirty turns exploring, and returns one paragraph. The father or mother’s context grows by that paragraph.
That is essentially the most highly effective approach obtainable and the one folks attain for final. It’s additionally the one which composes with all the pieces else: a sub-agent can use deferred loading, offloading, and code mode inside its personal run.
The fee: complete token spend often goes up even because the father or mother’s context stays small, since you’re paying for the sub-agent’s full transcript. You’re buying and selling tokens for high quality and run size. That’s usually the proper commerce, however watch the invoice, and watch sub-agent depend for delegation loops.
And compaction, as a flooring
When context approaches the restrict regardless of the entire above, summarize older turns moderately than dropping them. Truncation discards the choice that explains present state; summarization retains a lossy model of it.
Compaction is a security internet, not a technique. If it’s firing frequently, one of many 4 methods above isn’t doing its job. Deal with each compaction occasion as a sign value investigating moderately than a function working as meant.
The routing drawback beneath
All of this assumes you may see what instruments exist earlier than deciding what to load — which suggests instrument definitions want to return from someplace queryable, not from recordsdata dedicated subsequent to every agent.
That is the place MCP’s design pays off. In case your servers are registered centrally, an agent can enumerate obtainable functionality cheaply, pull schemas on demand, and choose instruments at runtime. TrueFoundry’s MCP Gateway is one implementation of that registry sample — servers register as soon as, authentication and per-user OAuth keep on the gateway, and brokers choose particular person instruments from {the catalogue} moderately than embedding endpoints and credentials. The aspect profit is that instruments flagged harmful as soon as, centrally, inherit an approval gate in each agent that makes use of them.
The associated level is that these methods are runtime habits, not software logic, which suggests they belong within the harness moderately than in your agent. TrueFoundry’s agent harness documentation enumerates them as configuration toggles — deferred instrument loading, giant instrument responses, code mode, subagents, compaction — which is a helpful guidelines no matter what you run on. If you happen to’re implementing these your self, that listing is roughly the scope.
Instrument earlier than you optimize
Monitor two issues per flip: energetic context measurement and context composition — how a lot is instrument definitions versus historical past versus outcomes.
Measurement tells you ways near the wall you’re and when compaction will hearth. Composition tells you which ones approach to use. An agent at 80% context that’s largely instrument definitions wants deferred loading. One which’s largely a single huge instrument end result wants offloading. One which’s largely historical past wants sub-agents. Similar symptom, three totally different fixes, and no manner to decide on with out the breakdown.
Most groups have neither quantity. Getting the primary one is often a day, and it’ll instantly clarify no less than one bug you’ve been carrying for a month.
Order of operations
- Measure composition. One afternoon. Do that first; it determines all the pieces after.
- Deferred instrument loading. Largest ratio of saving to effort you probably have many MCP servers.
- Giant response offloading. Kills the worst tail case.
- Sub-agents for something with a bounded, delegable subtask.
- Code mode for multi-step information work particularly.
- Compaction as a flooring, with alerting when it fires.
Skip straight to step 4 and also you’ll get an actual enchancment and nonetheless be paying 30,000 tokens a request for instruments no person calls.
