TL;DR: I spent the last two months instrumenting a production multi-agent system (a fleet of customer-facing support and research agents) with the OpenTelemetry GenAI semantic conventions. This is the build log: what the span schema actually looks like in code, the v1.42.0 migration that broke half our dashboards, how to wire LLM call → tool call → retry into one readable waterfall, how W3C traceparent finally crosses the MCP boundary via SEP-414, and which vendor backends actually work. If you instrument agents against a 2025 blog post today, you are emitting deprecated attributes.

1. The $10/hr Silent Tool Loop

It started with a billing alert, not a pager. One of our support agents — a ReAct-style loop over a ticket-search tool — had been quietly burning tokens at an alarming rate for six hours before anyone looked 6. The agent was stuck in a loop: reason → call search_tickets → the tool returned a malformed cursor → the model retried with the same broken cursor → repeat. Every iteration cost a full reasoning-model round trip. The logs told us nothing. Each entry said tool call failed, retrying with a correlation ID. There was no shape to it — no loop count, no per-iteration cost, no parent-child relationship between the failing tool call and the retry that followed.

This is the fundamental problem with logging agentic systems: logs are events, but agent bugs are structures. A nondeterministic agent failure is a pattern of calls, retries, and partial state — it only becomes visible when you can see the whole execution as a tree. Traces are the only observability primitive that preserves that structure, which is exactly why the GenAI semantic conventions model the entire agent execution as a span tree rather than single LLM calls 5. The conventions now capture the causal chain that matters for debugging: which LLM call produced which tool call, which tool call’s output fed which retry, and which retry finally settled on a result 6. We could not have debugged that loop without it.

The spec even anticipates the retry case: if a transient issue occurred and the request was retried automatically, the span SHOULD cover the duration of the logical operation with all retries 4. More on that in section 4.

2. Schema Anatomy: Manual Spans That Speak gen_ai.*

Before touching vendor SDKs, I hand-rolled spans with the plain opentelemetry-sdk to learn the schema. The conventions define a small, deliberate attribute vocabulary: operation name, provider, model, token usage, finish reason, and a handful of error and latency fields 6. Every gen_ai.* attribute, span, metric, and event in the official registry is still marked Development (formerly “experimental”) — none are Stable as of mid-2026 5. The signal set has settled enough to build on anyway 5.

A minimal, fully conformant chat span, current schema:

from opentelemetry import trace

tracer = trace.get_tracer("niteagent.agents")

def call_llm(client, messages, model="claude-sonnet-4-5"):
    # Span name convention: "{gen_ai.operation.name} {gen_ai.request.model}"
    with tracer.start_as_current_span(f"chat {model}") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", "anthropic")
        span.set_attribute("gen_ai.request.model", model)

        response = client.messages.create(model=model, messages=messages, max_tokens=1024)

        span.set_attribute("gen_ai.response.model", response.model)
        span.set_attribute("gen_ai.response.finish_reasons", response.stop_reason)
        span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)
        return response

Notes from the spec, not my opinion: gen_ai.operation.name and gen_ai.provider.name are Required on inference spans; gen_ai.request.model, gen_ai.response.model, gen_ai.response.finish_reasons, gen_ai.usage.input_tokens/output_tokens are Recommended 4. Span kind SHOULD be CLIENT, and MAY be INTERNAL for models running in the same process 4. Also note what’s absent: no prompt text, no completion text. Content capture (gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages) is strictly opt-in under the current conventions — a sane default if you operate under GDPR 4.

The execute_tool span is where agents diverge from plain LLM apps:

def execute_tool(name: str, call_id: str, fn, *args, **kwargs):
    with tracer.start_as_current_span(f"execute_tool {name}") as span:
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", name)
        span.set_attribute("gen_ai.tool.call.id", call_id)
        span.set_attribute("gen_ai.agent.name", "support-agent")
        # gen_ai.tool.call.arguments / gen_ai.tool.call.result are Opt-In
        result = fn(*args, **kwargs)
        return result

Span name is execute_tool {gen_ai.tool.name}, kind INTERNAL, with gen_ai.tool.name Required alongside gen_ai.operation.name 4. The spec explicitly encourages hand-instrumenting tool calls that auto-instrumentation misses, and notes MCP tool executions can be traced by the corresponding MCP instrumentation 4. The full operation vocabulary now covers the agent lifecycle: create_agent, invoke_agent, invoke_workflow, execute_tool, retrieval, plan, plus memory operations like search_memory and upsert_memory 4. For remote agent services, invoke_agent is a CLIENT span (OpenAI Assistants API, AWS Bedrock Agents); for same-process frameworks like LangChain or CrewAI agents, it’s an INTERNAL span 3.

3. The v1.42.0 Migration Nightmare

Here’s where the 2025 blog posts actively hurt us. Our original instrumentation, written in early 2025, looked like this:

# BEFORE (2025-era, now deprecated)
span.set_attribute("gen_ai.system", "anthropic")
span.set_attribute("gen_ai.usage.prompt_tokens", response.usage.input_tokens)
span.set_attribute("gen_ai.usage.completion_tokens", response.usage.output_tokens)
span.set_attribute("gen_ai.prompt", prompt_text)          # removed entirely
span.set_attribute("gen_ai.completion", completion_text)  # removed entirely

The current schema:

# AFTER (2026, what you should emit today)
span.set_attribute("gen_ai.provider.name", "anthropic")
span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)
# content capture is opt-in, structured, and separate:
# gen_ai.input.messages / gen_ai.output.messages / gen_ai.system_instructions

The rename traps, in order of how much they cost us:

  • gen_ai.system → gen_ai.provider.name. Renamed in semantic-conventions v1.37.0 with the old name deprecated 11. Our provider dashboards went dark the day the SDK bumped.
  • gen_ai.usage.prompt_tokens → gen_ai.usage.input_tokens and gen_ai.usage.completion_tokens → gen_ai.usage.output_tokens. Cost dashboards built on the old names silently under-count once emitters switch 5.
  • gen_ai.prompt / gen_ai.completion removed entirely — not renamed. Content capture now lives behind the opt-in gen_ai.input.messages / gen_ai.output.messages attributes 5.

The nightmare part is the timing. On June 12, 2026, semantic-conventions v1.42.0 deprecated all GenAI conventions in the main repo (model/gen-ai/, model/openai/, and the MCP conventions under model/mcp/) and moved them to a dedicated repository, open-telemetry/semantic-conventions-genai 5. The old opentelemetry.io/docs/specs/semconv/gen-ai/ pages are now move pointers 2. That repo has no tagged release — as of mid-July 2026 its releases page is empty and the docs still say “Status: Development” 5. So you’re migrating against a moving target with no versioned cut to pin. Meanwhile the last big additions inside the main repo landed in v1.41.0: streaming metrics (gen_ai.client.operation.time_to_first_chunk, .time_per_output_chunk), invoke_workflow as an operation, and the split of invoke_agent into client vs. internal spans 5.

Three things saved us:

  1. Grep your alert rules and saved queries, not just your code. The emitting side and the querying side drift independently 5.
  2. Plan for dual emission. OTel’s transition mechanism is OTEL_SEMCONV_STABILITY_OPT_IN — instrumentations can emit both legacy and latest-experimental schemas so backends migrate on their own clocks 5. Set OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental to opt participating instrumentations into the newest names, and COALESCE on query side with gen_ai.provider.name taking precedence 13.
  3. Lock the spec in tests. A golden-span exporter that asserts exact attribute keys catches a churn-induced rename in CI before it reaches production 13.

Expect more of this. Development status explicitly means names can still change 5.

4. Causal Chains: LLM Call → Tool Call → Retry in the Waterfall

The whole point of agent spans is that the trace reads as a reasoning chain, not a flat list. The conventions model the agent execution as a span tree 5:

invoke_agent support-agent            ← INTERNAL span: the reasoning loop
├── chat claude-sonnet-4-5           ← model decides to call search_tickets
│   └── execute_tool search_tickets  ← tool runs, returns malformed cursor
├── chat claude-sonnet-4-5           ← retry with same cursor (the bug!)
│   └── execute_tool search_tickets
└── chat claude-sonnet-4-5           ← success path
    └── execute_tool search_tickets

The parent-child structure does the debugging for you: the spec says the invoke_agent internal span is the agent’s top-level reasoning loop and the parent of model and tool calls 13. When you open one of these traces in a waterfall UI, you immediately see the loop — three sibling chat spans of similar duration, each with identical gen_ai.usage.input_tokens and a failing execute_tool child. That’s the causal chain the conventions exist to preserve: which LLM call produced which tool call, and which retry finally settled on a result 6.

Implementation details that matter:

  • Set sampling-critical attributes at span creation time. gen_ai.agent.name, gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model and server.address SHOULD be provided at span creation if provided at all, because they drive sampling decisions 3.
  • Retries belong inside one logical span. Per the spec, if a request is retried automatically, the span SHOULD cover the duration of the logical operation including all retries 4. We initially created a new chat span per attempt — wrong. The retry is a property of the operation, not a separate operation. Record the attempt count as an attribute or a log event correlated to the span instead.
  • Correlate with conversation ID. gen_ai.conversation.id on model spans lets you join a single user turn across providers and services 4.
  • Framework auto-instrumentation is real but partial. Google Cloud’s LangGraph walkthrough is a good reference pattern: instrument the Vertex AI SDK with opentelemetry-instrumentation-vertexai, then wrap the agent invocation manually:
from opentelemetry.instrumentation.google.genai import GoogleGenAiSdkInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

GoogleGenAiSdkInstrumentor().instrument()

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
    endpoint="https://telemetry.googleapis.com:443/v1/traces",
)))
trace.set_tracer_provider(provider)

# Invoke the agent within a span
with tracer.start_as_current_span("invoke agent"):
    result = agent.invoke({"messages": [prompt]}, config=config)

In Google Cloud Trace Explorer the resulting trace shows an invoke_agent span whose Gemini child span carries the GenAI events 8. The lesson: let instrumentors own the leaf spans (LLM, DB), and hand-own the skeleton — invoke_agent, execute_tool for anything custom, and the outer workflow span.

5. Crossing the MCP Boundary: SEP-414 and traceparent in _meta

Our agents call tools through MCP servers, and for months the trace died at the JSON-RPC boundary — the client half was one trace, the MCP server’s work was another. The fix is now standardized. The MCP conventions (which moved into the same GenAI repo in the v1.42.0 extraction 5) specify _meta as the carrier for W3C Trace Context keys 9, and SEP-414 (Final status, by Adrian Cole) documents it in the MCP spec: when OTel trace context is propagated via _meta, the keys traceparent, tracestate, and baggage follow W3C Trace Context and W3C Baggage value formats 9. The July 28, 2026 MCP release candidate locks these key names in so distributed traces correlate across SDKs and gateways 12.

The wire shape, straight from the SEP’s non-normative example 9:

{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/call",
  "params": {
    "name": "get_weather",
    "arguments": { "location": "New York" },
    "_meta": {
      "traceparent": "00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01"
    }
  }
}

The subtle part is that _meta normally uses DNS-prefixed keys; traceparent is a deliberate exception so existing implementations keep working — if someone namespaced it as io.modelcontextprotocol.traceparent, distributed traces and log correlation would break 9. Injecting the current OTel context into _meta is a one-liner with the standard propagation API:

from opentelemetry import context, propagate

def inject_otel_context_to_meta() -> dict:
    carrier = {}
    propagate.inject(carrier, context=context.get_current())
    return carrier  # {"traceparent": "00-...", "tracestate": "...", "baggage": "..."}

# ...then include it in the MCP request params:
# params["_meta"] = inject_otel_context_to_meta()

This pattern is already implemented in the C# MCP SDK, the Python SDK (PR #1693), OpenInference’s MCP instrumentation (Python and TypeScript), Envoy AI Gateway, Logfire, and ToolHive 9. If your MCP server framework doesn’t do this yet, patch it in yourself — it’s the difference between traces that stop at the process boundary and traces that span the whole agent + tool graph. A follow-up SEP (2028) builds on this to forward _meta values into HTTP headers for streamable HTTP transports 9.

6. Vendor Reality Check: Datadog vs Langfuse vs Helicone

We evaluated three backend strategies with real traces. The landscape has split into three camps: traditional APM platforms treating GenAI as a new signal type, AI-native tracing tools racing to be your OTLP sink, and AI gateways that emit spans as a side effect of proxying 6.

Datadog (native semconv) Langfuse (OTLP endpoint) Helicone (gateway)
Integration Point OTLP exporter at Datadog; v1.37+ schema OTEL_EXPORTER_OTLP_ENDPOINT=https://cloud.langfuse.com/api/public/otel + Basic Auth Proxy your provider calls through Helicone
Schema handling Auto-maps gen_ai.* to Agent Observability schema Maps incoming gen_ai.* onto Langfuse data model Emits OTel GenAI spans as side effect of routing
Gotchas Requires OTel SDK/Collector v1.37+; 40k LLM spans/month free tier OTLP over HTTP (JSON/protobuf) only — no gRPC; v4 needs x-langfuse-ingestion-version: 4 header for real-time ingest You’re coupled to the gateway’s span tree; app never sees raw spans

Datadog actually works: native support for OTel GenAI semantic conventions v1.37 and up, ingest via OTLP exporter, Datadog Agent, or Collector, with automatic mapping of gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.provider.name, and gen_ai.operation.name into Agent Observability 7. It’s free for up to 40,000 LLM spans/month 7. Two caveats from our pilot: their blog’s example operation values (tool_call, agent_run) don’t match the current spec vocabulary (execute_tool, invoke_agent) — vendor mapping is opinionated 7 — and once you’re in Datadog’s schema you’re on their semantic layer.

Langfuse is the most faithful OTLP citizen of the three: a real /api/public/otel endpoint, documented attribute mapping, and explicit support for extra attributes popular in the OTel GenAI ecosystem 10. Because the GenAI conventions are still evolving, they map received traces onto their own data model 10. Practical traps: gRPC is not supported (HTTP/JSON and HTTP/protobuf only), and for real-time ingestion on v4 you must send the x-langfuse-ingestion-version: 4 header or data can be delayed up to 10 minutes 10. It’s the best trace-exploration UX we tested, and it’s the camp betting hardest on OTLP as the on-ramp 6.

Helicone is the interesting one: it sits between your app and the provider and emits OTel GenAI spans as a side effect of routing, caching, and cost tracking 6. Zero instrumentation in your agent code — but you inherit the gateway’s span vocabulary, and the proxy layer becomes part of your blast radius. We use gateways for cost controls, not as our primary trace source. (Worth knowing for procurement: the fragmentation this schema fight ends was real — Langfuse, Arize Phoenix’s OpenInference, and Traceloop’s OpenLLMetry each shipped incompatible schemas through 2024–2025 6, and Traceloop/OpenLLMetry was acquired by ServiceNow for a reported $60–80M in March 2026 6.)

Our verdict: instrument once against gen_ai.*, treat OTLP as the socket, and treat backends as swappable. That’s the playbook the ecosystem has converged on 5. We ship OTLP to a Collector, fan out to Langfuse for day-to-day debugging and Datadog for cross-layer correlation, and we can add or drop backends without touching agent code.

7. What I’d Do Differently (and What I’d Tell You)

  • Don’t wait for stability, but budget for renames. Every gen_ai.* element is Development 5; the gen_ai.system rename won’t be the last 5. Build the golden-span test now 13.
  • Emit the skeleton by hand, leaves by instrumentors. Manual invoke_agent / execute_tool spans with the plain SDK are fully conformant 5; instrumentors handle the LLM and DB leaf spans 8.
  • Keep prompt capture opt-in. Metadata-only is the default for good reason; enable gen_ai.input.messages/gen_ai.output.messages only where the privacy question is settled 4.
  • Fix MCP propagation once, in the SDK layer. traceparent in _meta per SEP-414 — do it before you need it 9.
  • Traces tell you what the agent did; they don’t tell you if it was any good. Latency and token counts are necessary but not sufficient — wire evals into the same pipeline 5.

The GenAI conventions are an unstable standard that the market adopted anyway. By mid-2026, Datadog ships native support, Langfuse ingests it over OTLP, and the MCP spec carries your trace context across process boundaries 79. That’s enough to build on. Just keep the migration checklist handy.

Sources

📖 Related Reads

  • ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
  • Hermes Tutorials — Hermes Agent setup, configuration, and advanced workflows

Cross-links automatically generated from NiteAgent.

← Back to all posts