On this article, you’ll study what AI agent observability means, why conventional monitoring instruments fall brief for agentic techniques, and the right way to implement structured logging, distributed tracing, and sensible debugging workflows for AI brokers.
Subjects we’ll cowl embody:
- Why AI brokers fail in ways in which appear like success, and why that makes customary monitoring instruments inadequate.
- How you can implement structured logging and OpenTelemetry-based tracing for agent runs, instrument calls, and mannequin inference steps.
- How you can learn hint waterfalls, observe token prices with metrics, and use that knowledge to debug actual agent failures.
An agent dealing with buyer assist tickets closes one out with a clear, skilled, completely mistaken reply. It known as the refund-lookup instrument as soon as, then known as it once more with barely totally different arguments just a few seconds later, then answered confidently primarily based on the second consequence as an alternative of the primary. Nothing crashed. No error fired. The uptime dashboard exhibits inexperienced your complete time. The one motive anybody finds out is a buyer replying two days later, confused, and by then no one can reconstruct what truly occurred inside that run.
That failure is the entire motive this text exists. A conventional service both returns an error 200 or throws one thing you’ll be able to grep for. An agent can do neither and nonetheless be utterly mistaken, and the tooling constructed for the primary form of system is near blind to the second. This text walks via what truly wants to vary — logging, tracing, and debugging — one after the other, with actual code.
What AI Agent Observability Truly Means
AI agent observability is the observe of capturing each mannequin name, instrument execution, and reasoning step an agent makes as structured knowledge, in order that when one thing goes mistaken, you’ll be able to reconstruct precisely what occurred and why, slightly than guessing or re-running the identical immediate and hoping the issue repeats itself.
It borrows from the three pillars observability engineers already know — logs, metrics, and traces — however the motive it wants its personal identify and its personal self-discipline comes all the way down to how brokers truly fail. Aryan Kargwal, a researcher within the area, put it plainly in protection from Digital Utilized’s 2026 observability information: agentic techniques fail in ways in which appear like success — incorrect however well-formed outputs, pointless instrument calls, or actions which can be syntactically legitimate however semantically mistaken. None of that journeys an error handler. A well being verify reporting “up” tells you virtually nothing helpful about whether or not the agent truly did the suitable factor on any given run.
Why Brokers Break the Conventional Monitoring Mannequin
It’s value being particular concerning the mechanics right here, as a result of “brokers are unpredictable” undersells precisely what adjustments.
The identical enter doesn’t reliably produce the identical conduct anymore. Temperature settings, retrieval outcomes, and which instruments occur to be accessible can all shift the trail an agent takes, so the identical immediate can set off a genuinely totally different sequence of instrument calls on two consecutive runs. A single “it labored once I examined it” hint tells you virtually nothing about what the distribution of actual runs truly appears like.
Price and latency cease correlating with request rely and begin correlating with tokens as an alternative. A single “sluggish” request is perhaps consuming ten occasions the traditional token finances, and a monitoring setup constructed round requests-per-second is structurally blind to that. Multi-step chains compound the issue: one person request would possibly set off a number of mannequin calls, a handful of instrument calls, and a few retrieval lookups, and each is an impartial level of failure {that a} single combination error metric can’t distinguish between. And prompts themselves routinely carry actual private or confidential data, which implies naive logging that dumps full immediate textual content right into a backend creates a real compliance drawback earlier than it has created any debugging worth in any respect.
A side-by-side comparability of the 2 worlds makes the shift concrete:
| Sign | Conventional app | LLM / AI agent |
|---|---|---|
| Latency driver | CPU, I/O, community | Token rely, mannequin dimension, context window |
| Price unit | Requests per second | Tokens consumed |
| Failure mode | Exception, timeout | Hallucination, context overflow, instrument error |
| Debug artifact | Stack hint | Immediate, completion, and the reasoning chain between them |
Logging
Begin with probably the most acquainted pillar, as a result of it’s nonetheless the muse every thing else builds on, simply utilized otherwise. For an agent, the occasions value logging are particular: which instrument bought known as and with what arguments, what got here again, what number of tokens a given step consumed, how lengthy every hop took, and any error alongside the best way — and all of it structured slightly than written as free-text sentences a human has to parse later.
The element that really makes agent logging helpful is tying each log line again to the precise run it got here from. A log assertion that simply says “instrument name failed” is almost nugatory at 2 am when three totally different customers triggered three totally different runs in the identical minute. Attaching the present hint ID to each log line — one thing OpenTelemetry does robotically as soon as tracing is ready up — is what turns a pile of scattered log statements into one thing you’ll be able to filter all the way down to the precise run that broke.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
import logging from opentelemetry import hint
# Customary Python logging, nothing unique right here logger = logging.getLogger(“agent”) logging.basicConfig(stage=logging.INFO)
tracer = hint.get_tracer(“agent-service”)
def call_tool(tool_name: str, arguments: dict): # get_current_span() pulls no matter span is energetic proper now, # so the log line under may be tied again to the precise hint # and step it occurred inside span = hint.get_current_span() trace_id = format(span.get_span_context().trace_id, “032x”)
logger.information( “tool_call_started”, additional={ “trace_id”: trace_id, “tool_name”: tool_name, “arguments”: arguments, }, )
strive: consequence = execute_tool(tool_name, arguments) logger.information( “tool_call_succeeded”, additional={“trace_id”: trace_id, “tool_name”: tool_name, “result_length”: len(str(consequence))}, ) return consequence besides Exception as e: logger.error( “tool_call_failed”, additional={“trace_id”: trace_id, “tool_name”: tool_name, “error”: str(e)}, ) increase |
Just a few issues value noticing in that snippet. hint.get_current_span() doesn’t require you to manually move a hint ID down via each perform name — it reads no matter span is energetic within the present execution context, which is strictly what makes this sample sensible to sprinkle all through an actual codebase with out threading an ID parameter via each layer.
Logging the arguments and the consequence size, slightly than the total consequence content material, is a deliberate alternative, not an oversight; full instrument outputs may be massive and might carry delicate knowledge, and a size or a truncated preview is normally sufficient to identify an issue with out turning each log line right into a privateness legal responsibility. And logging each a begin and an finish occasion for a similar instrument name, slightly than simply the result, is what permits you to later measure precisely how lengthy that particular name took — which turns into the uncooked materials tracing formalizes correctly within the subsequent part.
Tracing
Logging tells you what occurred at particular person closing dates. Tracing is what stitches these factors right into a form — a full document of 1 agent run from the primary request to the ultimate reply, with each step nested contained in the step that triggered it. That nested form is the precise reply to “why did the agent try this,” as a result of it exhibits you not simply {that a} instrument was known as, however which reasoning step determined to name it and what occurred instantly earlier than and after.
The vocabulary right here comes from the OpenTelemetry GenAI semantic conventions, which outline a regular set of gen_ai.* span varieties and attributes particularly for this. Fairly than each group inventing their very own span names, the spec defines a handful of operation varieties value understanding: create_agent for when an agent is first outlined, invoke_agent for a single agent run, invoke_workflow for orchestration throughout a number of brokers handing off to one another, execute_tool for a person instrument name, and chat for the precise mannequin inference name itself. Each carries a regular set of attributes — gen_ai.request.mannequin, gen_ai.utilization.input_tokens, gen_ai.utilization.output_tokens, and gen_ai.response.finish_reasons amongst them — so a hint produced by one group’s agent appears structurally the identical as one produced by a totally totally different framework.
Right here’s what manually instrumenting a small tool-calling agent truly appears like:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
from opentelemetry import hint from opentelemetry.hint import Standing, StatusCode
tracer = hint.get_tracer(“agent-service”)
def run_agent(process: str) -> str: # The basis span for this whole run; each step under nests # inside it, which is what produces the parent-child tree with tracer.start_as_current_span(“invoke_agent”) as agent_span: agent_span.set_attributes({ “gen_ai.system”: “openai”, “agent.identify”: “support-agent”, “gen_ai.request.mannequin”: “gpt-4o”, })
messages = [ {“role”: “system”, “content”: “You are a support assistant.”}, {“role”: “user”, “content”: task}, ]
whereas True: # The mannequin name itself will get its personal youngster span with tracer.start_as_current_span(“chat”) as chat_span: response = model_client.chat.completions.create( mannequin=“gpt-4o”, messages=messages, instruments=AVAILABLE_TOOLS ) alternative = response.selections[0]
chat_span.set_attributes({ “gen_ai.response.mannequin”: response.mannequin, “gen_ai.utilization.input_tokens”: response.utilization.prompt_tokens, “gen_ai.utilization.output_tokens”: response.utilization.completion_tokens, })
if alternative.finish_reason != “tool_calls”: agent_span.set_status(Standing(StatusCode.OK)) return alternative.message.content material
# Every instrument name will get its personal youngster span, nested below # the agent run, not below the chat span, since a instrument # name is a sibling step, not a sub-step of inference for tool_call in alternative.message.tool_calls: with tracer.start_as_current_span(“execute_tool”) as tool_span: tool_span.set_attributes({ “gen_ai.instrument.identify”: tool_call.perform.identify, “gen_ai.instrument.name.id”: tool_call.id, }) strive: consequence = call_tool(tool_call.perform.identify, tool_call.perform.arguments) besides Exception as e: tool_span.record_exception(e) tool_span.set_status(Standing(StatusCode.ERROR, str(e))) increase
messages.append({ “position”: “instrument”, “content material”: str(consequence), “tool_call_id”: tool_call.id, }) |
The nesting is doing the actual work right here. Each chat span and each execute_tool span opens contained in the with tracer.start_as_current_span(…) block belonging to the run above it, which is strictly what OpenTelemetry makes use of to construct the parent-child relationship robotically — you by no means manually wire “this span belongs below that one,” it’s implicit in how the with blocks are structured in your code. record_exception plus set_status(StatusCode.ERROR, …) on the instrument span is what makes a failed instrument name present up clearly in a hint viewer slightly than silently vanishing into the returned string, which issues straight for the debugging part later on this article. And separating token utilization attributes onto the chat span particularly, slightly than the top-level invoke_agent span, is what lets a hint viewer later present you token price damaged down per mannequin name inside a single run, not only a single mixed complete for the entire thing.
A screenshot of a hint waterfall view in an observability dashboard (click on to enlarge)
Picture by Creator
Chain Visualization: Studying the Hint Waterfall
The spans from the final part don’t imply a lot as a uncooked listing. What truly makes them helpful is a waterfall view — spans stacked by their nesting depth and stretched horizontally by how lengthy each took, so a complete agent run turns into a single image you’ll be able to scan in seconds. It’s value studying to learn one in plain textual content earlier than ever opening an actual dashboard, because the form is similar both means:
|
invoke_agent (1850ms) ├── chat (920ms) ← decides to name two instruments │ ├── execute_tool: refund_lookup (310ms) │ └── execute_tool: refund_lookup (295ms) ← known as once more, similar instrument └── chat (530ms) ← closing reply |
That waterfall alone tells you greater than an hour of guessing would. The 2 refund_lookup calls sitting as siblings below the identical chat span is strictly the form of redundant instrument name that prompted the failure this text opened with, and it’s seen at a look slightly than buried in a wall of logs. In an actual dashboard, this similar construction renders as horizontal bars, and the habits value constructing are easy: search for a bar that’s unusually vast relative to its siblings, since that’s the place time and token finances are literally going; search for a instrument name that repeats when it shouldn’t; and search for a chat span the place the mannequin reached for a instrument in any respect when the duty didn’t clearly want one. None of that requires studying a single log line. It’s all seen within the form of the hint itself.
Token Monitoring and Price Metrics
Traces are glorious for understanding one particular run intimately. They’re the mistaken instrument for recognizing a development throughout 1000’s of runs, which is what metrics are for. The 2 genuinely load-bearing metrics for an agent, per the OpenTelemetry GenAI conventions, are gen_ai.consumer.token.utilization and gen_ai.consumer.operation.length, tracked as a counter and a histogram respectively.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
from opentelemetry import metrics import time
meter = metrics.get_meter(“agent-service”)
token_counter = meter.create_counter( “gen_ai.consumer.token.utilization”, unit=“token”, description=“Tokens consumed, damaged down by mannequin and enter/output”, )
duration_histogram = meter.create_histogram( “gen_ai.consumer.operation.length”, unit=“s”, description=“Period of every mannequin name, in seconds”, )
def tracked_chat_call(messages: listing, mannequin: str = “gpt-4o”) -> str: attrs = {“gen_ai.system”: “openai”, “gen_ai.request.mannequin”: mannequin} begin = time.time()
strive: response = model_client.chat.completions.create(mannequin=mannequin, messages=messages)
# Recording enter and output tokens as two separate calls, # not one mixed complete, is what preserves the precise # price construction, since enter and output tokens are # priced otherwise on almost each supplier token_counter.add(response.utilization.prompt_tokens, {**attrs, “gen_ai.token.sort”: “enter”}) token_counter.add(response.utilization.completion_tokens, {**attrs, “gen_ai.token.sort”: “output”})
return response.selections[0].message.content material lastly: duration_histogram.document(time.time() – begin, attrs) |
The selection to document enter and output tokens as two separate counter calls, slightly than one mixed total_tokens worth, issues greater than it appears. A single mixed counter hides precisely the knowledge you’d want to note — for example, {that a} system immediate has quietly grown so massive it’s dwarfing the precise person enter on each single name. Splitting them aside is what makes that seen in a dashboard slightly than buried inside a median.
With these two metrics flowing, a small set of alert situations covers most of what truly goes mistaken in manufacturing, per Uptrace’s suggestions:
| Metric | Alert situation | Why it issues |
|---|---|---|
| Token utilization fee | Greater than double the baseline over 10 minutes | Typically a runaway loop or a immediate injection try |
| Operation length, p99 | Above 30 seconds | The mannequin is overloaded, or the context window is simply too massive |
| Error fee | Above 2% over 5 minutes | Charge limiting or quota exhaustion, value catching earlier than customers do |
| Enter-to-output token ratio | Persistently above 10 to 1 | The system immediate has possible grown bloated and wishes trimming |
Telemetry Pipelines: Getting Traces from Code to a Backend
All the things up to now has assumed traces and metrics land someplace helpful, and that plumbing deserves its personal consideration slightly than being an afterthought. In a typical setup, your instrumented utility exports telemetry to an OpenTelemetry Collector — a separate course of that receives it, optionally transforms or filters it, and forwards it on to wherever you’re truly storing and viewing traces. That center layer is the place two sensible issues get dealt with with out touching a single line of utility code.
- The primary is sampling. Capturing each single hint at full quantity is affordable in growth, however LLM calls are sluggish and their spans are massive, so full seize in manufacturing will get costly quick with out including proportional worth. The sensible sample is to pattern otherwise relying on the state of affairs: full seize in growth, a modest share — typically 5 to 10% — of routine profitable manufacturing calls, and 100% seize for something genuinely useful: each error, each high-token request, and each full agent run, since these are precisely the instances you’ll truly need to look again at.
- The second is privateness, and it deserves to be handled as a first-class design resolution slightly than one thing bolted on later. Immediate and completion content material belongs in span occasions, not span attributes, since attributes are at all times listed and exported with no dimension restrict, whereas occasions may be filtered, truncated, or dropped completely on the Collector stage. A Collector configuration can strip or hash immediate content material from each span crossing the pipeline earlier than it ever reaches a storage backend, which implies a compliance requirement doesn’t have to show right into a code change scattered throughout each instrumented name web site in your utility.
|
processors: remodel: trace_statements: – context: spanevent statements: # Strips immediate and completion content material from each span # occasion that passes via the Collector, no utility # code adjustments required – delete_matching_keys(attributes, “gen_ai.immediate.content material”) – delete_matching_keys(attributes, “gen_ai.completion.content material”) |
MCP Tracing
It is a genuinely current addition value understanding about particularly, because it closes a niche most current protection of this matter hasn’t caught as much as but. Earlier than OpenTelemetry’s Mannequin Context Protocol semantic conventions, added in spec model 1.39, the precise protocol mechanics beneath an MCP instrument name — which technique bought invoked, which session it belonged to, and which protocol model was in use — have been successfully invisible in a hint. You could possibly see {that a} instrument ran and what it returned, however not the layer beneath that.
The element value understanding is how these new attributes get connected. Fairly than making a second, separate span for the protocol layer, MCP instrumentation enriches the prevailing execute_tool span with attributes like mcp.technique.identify, mcp.session.id, and mcp.protocol.model, layering the additional element onto the span you already had as an alternative of doubling the hint’s noise. For anybody constructing brokers that decision out to a number of MCP servers, that’s the distinction between a hint that stays readable and one which turns right into a wall of near-duplicate spans.
Debugging Workflows
That is the place every thing constructed up to now truly pays off. An actual debugging workflow, as soon as tracing is in place, tends to comply with the identical form whatever the platform behind it: begin from the dangerous output a person reported, pull the hint ID that produced it, open the waterfall, and scan for the span the place the run truly went mistaken — an unexpectedly vast chat span, a repeated execute_tool name, an error standing someplace within the tree. When you’ve discovered the step, the span’s attributes and occasions inform you precisely what arguments have been handed and what got here again, which is normally sufficient to grasp the error straight, no re-running required, no guessing.
Two extra superior methods are value understanding as this observe matures. Replay, typically known as time-travel debugging, permits you to re-run an agent session with point-in-time precision — successfully restoring the precise state the agent was in at a given step and persevering with from there, a functionality AgentOps is particularly identified for. And a more recent sample value watching is natural-language hint querying, the place as an alternative of manually scanning a waterfall, an engineer can straight ask a platform one thing like “why did the agent enter this loop” and get a solution generated from analyzing the hint knowledge itself — a functionality LangSmith has constructed straight into its product. Neither replaces the basics coated above. Each are what these fundamentals make doable as soon as a group has sufficient traces flowing to make asking that form of query worthwhile.
The Instruments High Groups Truly Use in 2026
Constructing this your self with uncooked OpenTelemetry, as proven all through this text, works and retains you vendor-neutral, however most groups ultimately attain for a platform to retailer, visualize, and question the traces they’re producing. The choice genuinely comes all the way down to deployment mannequin earlier than it comes all the way down to options, since that single alternative eliminates a lot of the area by itself, per Digital Utilized’s 2026 breakdown.
Self-hosted platforms — Langfuse and Arize Phoenix amongst them — swimsuit groups with actual knowledge residency necessities or a necessity for tight price management at scale, at the price of proudly owning the operational overhead your self. Managed SDKs — LangSmith and Braintrust — commerce that possession for pace: you add an SDK, the seller runs the backend and storage, and also you sometimes get analysis tooling bundled in from day one. Proxy gateways — Helicone being the clearest instance — sit your visitors behind a routing layer that logs price and utilization throughout lots of of fashions with near zero code change, with the tradeoff that the gateway itself turns into a single level of failure value planning actual uptime round.
| Platform | Deployment mannequin | Free tier | OpenTelemetry assist | 2025 to 2026 sign |
|---|---|---|---|---|
| Langfuse | Self-hosted or cloud | Free self-hosting | Sure | Acquired by ClickHouse, January 2026 |
| Arize Phoenix | Self-hosted | Open supply, free | Sure, OTLP-native | Actively rising open-source venture |
| LangSmith | Managed SDK | 5,000 traces monthly | Sure | Pure-language hint querying in-built |
| Braintrust | Managed SDK | 1 million spans monthly | Sure | $80M Collection B, February 2026 |
| Helicone | Proxy gateway | 10,000 requests monthly | By way of gateway | Price monitoring throughout 300+ fashions |
| AgentOps | SDK | Open supply, free | Sure | Identified for time-travel replay debugging |
| Datadog LLM Observability | Managed, extends current APM | 40,000 LLM spans monthly | Sure | Payments solely LLM spans, not instrument or retrieval spans |
Placing It Collectively
Right here’s how the entire items above mix into one working script, so the ideas on this article don’t keep disconnected. This wires up tracing, token metrics, and structured logging collectively round a single small agent.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 |
import logging import time from opentelemetry import hint, metrics from opentelemetry.hint import Standing, StatusCode
tracer = hint.get_tracer(“agent-service”) meter = metrics.get_meter(“agent-service”) logger = logging.getLogger(“agent”)
token_counter = meter.create_counter(“gen_ai.consumer.token.utilization”, unit=“token”) duration_histogram = meter.create_histogram(“gen_ai.consumer.operation.length”, unit=“s”)
def run_instrumented_agent(process: str) -> str: with tracer.start_as_current_span(“invoke_agent”) as agent_span: trace_id = format(agent_span.get_span_context().trace_id, “032x”) agent_span.set_attributes({“agent.identify”: “support-agent”, “gen_ai.request.mannequin”: “gpt-4o”}) logger.information(“agent_run_started”, additional={“trace_id”: trace_id, “process”: process[:100]})
messages = [{“role”: “user”, “content”: task}]
whereas True: with tracer.start_as_current_span(“chat”) as chat_span: begin = time.time() response = model_client.chat.completions.create( mannequin=“gpt-4o”, messages=messages, instruments=AVAILABLE_TOOLS ) duration_histogram.document(time.time() – begin, {“gen_ai.request.mannequin”: “gpt-4o”})
utilization = response.utilization token_counter.add(utilization.prompt_tokens, {“gen_ai.token.sort”: “enter”}) token_counter.add(utilization.completion_tokens, {“gen_ai.token.sort”: “output”}) chat_span.set_attributes({ “gen_ai.utilization.input_tokens”: utilization.prompt_tokens, “gen_ai.utilization.output_tokens”: utilization.completion_tokens, })
alternative = response.selections[0]
if alternative.finish_reason != “tool_calls”: agent_span.set_status(Standing(StatusCode.OK)) logger.information(“agent_run_completed”, additional={“trace_id”: trace_id}) return alternative.message.content material
for tool_call in alternative.message.tool_calls: with tracer.start_as_current_span(“execute_tool”) as tool_span: tool_span.set_attribute(“gen_ai.instrument.identify”, tool_call.perform.identify) logger.information( “tool_call_started”, additional={“trace_id”: trace_id, “tool_name”: tool_call.perform.identify}, ) strive: consequence = call_tool(tool_call.perform.identify, tool_call.perform.arguments) besides Exception as e: tool_span.record_exception(e) tool_span.set_status(Standing(StatusCode.ERROR, str(e))) logger.error( “tool_call_failed”, additional={“trace_id”: trace_id, “tool_name”: tool_call.perform.identify}, ) increase messages.append({“position”: “instrument”, “content material”: str(consequence), “tool_call_id”: tool_call.id}) |
Run this with a Collector configured to export to whichever backend you’ve picked from the desk above, and a single name to run_instrumented_agent produces a full hint with nested spans for each instrument name, token metrics recorded per mannequin name, and structured log traces carrying the hint ID that ties every thing again collectively — precisely the setup this whole article has been constructing towards.
Conclusion
An unmanaged agent isn’t a smaller danger than an unsupervised internet service; it’s a bigger one, exactly as a result of its failures are constructed to look tremendous till somebody checks carefully. The refund lookup known as twice, the assured reply constructed on stale knowledge — none of that journeys an alarm by itself. Logging tells you what occurred at every step. Tracing exhibits you the way these steps truly related. Debugging is what turns that document into a solution as an alternative of a guess. Construct all three in earlier than an agent is dealing with something that really issues, not after the primary buyer notices one thing went mistaken.
![11 social media tendencies each marketer ought to watch in 2026 [new data]](https://blog.aimactgrow.com/wp-content/uploads/2026/09/social20media20trends-1-120x86.png)
