Observability — Tracing

OpenTelemetry tracing in Forge is off by default. When enabled, every inbound A2A request becomes one trace whose span tree covers the dispatcher, the agent execution loop, every LLM completion, every tool call, and every outbound HTTP request. Trace context propagates across multi-hop A2A flows, audit events carry the active span's trace_id + span_id, and the OTLP collector host is auto-allowlisted at build time so deployments need no second egress edit.

Status: shipped as OTel Tracing v1 — initiative tracking issue #108, delivered across phases #101–#107 (PRs #122–#128).

Quick start

# forge.yaml
observability:
  tracing:
    enabled: true
    endpoint: https://otel-collector.monitoring.svc.cluster.local:4318/v1/traces
    sampler: parentbased_always_on

Run:

forge run                    # tracing on, defaults applied
forge build && forge package # collector host auto-added to egress allowlist
kubectl apply -f ...

Spans arrive at the collector. The agent's agent_id is the service.name your trace backend groups by.

forge.yaml schema

observability:
  tracing:
    enabled: true                           # off by default
    endpoint: https://collector:4318/v1/traces
    protocol: http/protobuf                  # or "grpc"
    sampler: parentbased_always_on           # standard OTEL_TRACES_SAMPLER name
    sampler_ratio: 1.0                       # used by *traceidratio* samplers
    timeout: 10s                             # per-request exporter timeout
    service_name: my-agent                   # default: agent_id
    headers:                                  # OTLP request headers (auth tokens etc.)
      x-tenant: demo
    resource_attrs:                          # extra OTel resource attributes
      deployment.environment: prod
    redact: true                             # scrub vendor secret tokens when capture_content is on
    capture_content: false                   # opt-in: stamp prompt/completion/tool I/O on spans

Config precedence

Lowest → highest:

1. Defaults
2. observability.tracing block in forge.yaml
3. OTEL_* environment variables (standard SDK names)
4. CLI flags (--otel-*)

A set-but-empty env var does not wipe a non-empty yaml field. Absence-of-value is "no override," not "unset."

Environment variables

CLI flags

Each flag detection uses cmd.Flags().Changed(...) rather than zero-value sentinels, because every "zero" is a legitimate explicit ask (--otel-sampler-ratio 0 = drop everything, --otel-enabled=false = force off).

Samplers

The six standard OTEL_TRACES_SAMPLER names — Forge maps them to the OTel SDK directly:

Name parsing is case-insensitive and whitespace-tolerant. Unknown names error loudly at startup with the offending string named — a typo like parent_based_always_on is caught immediately rather than silently falling through to a default.

Span hierarchy

auth.verify                           [pre-request; parents provider HTTP calls]
└── http.client (× JWKS/STS/IAP/Graph)

admission.check                       [pre-dispatch; opt-in via FORGE_ADMISSION_URL]
└── http.client (to platform admission endpoint)

a2a.<method>                          [SpanKindServer; dispatcher]
└── agent.execute                     [outer loop; root for the task]
    ├── llm.completion (× N turns)    [per LLM provider call]
    │   └── http.client (× outbound)  [auto via otelhttp on egress transport]
    └── tool.<tool_name> (× M calls)  [per tool invocation]
        └── http.client (if HTTP)

channel.<adapter>.deliver             [inbound Slack/Telegram/Teams]
└── a2a.tasks/send (via internal POST + injected traceparent)
    └── (full a2a.<method> subtree above)

schedule.fire                         [file-backend tick; opens per fire]
└── (a2a.<method> / agent.execute subtree, depending on dispatcher)

auth.verify, channel.<adapter>.deliver, and schedule.fire were added in issue #187 to cover three latency / causality surfaces operators previously couldn't see in traces. Each one:

• Is opened once per operation (auth: per inbound request; channel: per inbound message; schedule: per file-backend tick).
• Uses the global Tracer() — no new tracer install — so the off-by-default tracing posture extends to them automatically (no-op when tracing is disabled, zero allocation).
• Sets codes.Error Status on the failure path so the error-rate dashboards work uniformly across span types.

auth.verify

Wraps the Provider.Chain.Verify call in forge-core/auth/middleware.go. Without this span the provider's outbound HTTP calls (JWKS fetch, AWS STS verify, IAP token introspect, AAD Graph) showed up as orphan root spans with no "why was this called" context, and total auth latency was invisible.

Span closes BEFORE installSequenceCounterMiddleware runs, so it sits outside the per-invocation sequence counter scope — the right scope, since the question is "did the caller authenticate?", not "what did the agent do?"

channel.<adapter>.deliver

Wraps the per-message handler in each channel adapter (Slack / Telegram / Teams) around the parse + thread-context fetch + internal A2A POST. The router's internal POST injects the W3C traceparent from the calling ctx, so the agent server's a2a.tasks/send span nests under channel.<adapter>.deliver and you can finally answer "how long does Slack→agent take?" from the flame graph alone.

schedule.fire

Wraps Scheduler.fire in forge-core/scheduler/scheduler.go. Before this span the dispatched executor work looked unsourced — no current span at fire time, so any downstream agent.execute was an orphan root. File-backend only for v1. The K8s backend's trigger Pod is a separate curl-based Pod and would need traceparent injected into the rendered CronJob YAML at forge package time — tracked as a follow-up.

admission.check

Wraps the platform admission call in forge-cli/server/admission_middleware.go. Sibling of auth.verify — fires after auth, before the dispatcher. Off by default; opt-in via the FORGE_ADMISSION_URL + FORGE_PLATFORM_TOKEN env-var pair. See Platform Admission Hook for the wire contract.

Status = Error on deny. The HTTP call to the platform nests under the span as http.client so total admission latency = span duration, platform-side latency = HTTP child duration.

Attribute conventions

Forge mixes OTel GenAI semconv with Forge-specific forge.* namespaced attributes. Backends key dashboards by these:

Tool spans follow the OTel GenAI semantic conventions (gen_ai.tool.*) — these replaced the former proprietary forge.tool.* keys, which never shipped to production. MCP-backed tools (namespaced <server>__<tool>) are typed extension and additionally carry mcp.method.name=tools/call. (mcp.session.id / mcp.protocol.version require plumbing the MCP manager to the executor and are tracked as a follow-up.)

Tool errors do not fail the outer agent.execute span — they surface to the LLM as text and the loop continues. The tool span carries the failure detail (error.type + span status Error) so operators can pivot from a trace to the specific failed invocation.

Span content capture

Prompts, completions, tool args, and tool results are off by default — Phase 3 spans ship metadata only (provider, model, usage, finish reasons, tool name). Operators who need content attributes for in-trace debugging or supervised-learning corpora opt in via observability.tracing.capture_content: true (Phase 3.5 / issue #130).

When capture_content: true and redact: true (the default when capture is on), attribute values pass through a redactor that scrubs the same vendor secret-token shapes the runtime guardrails default rules cover (Anthropic sk-ant-…, OpenAI sk-…, GitHub ghp_/gho_/ghs_/github_pat_…, AWS AKIA…, Slack xoxb-/xoxp-…, RSA/EC/OPENSSH/PRIVATE key blocks, Telegram bot tokens). Matched values become [REDACTED]. Setting redact: false is the enterprise raw-capture path — content is stamped verbatim with the byte cap still applied.

Every captured value is byte-capped at 4 KiB (below the 5 KiB attribute soft-cap most backends apply). When the input exceeds the cap, the value ends with a …[truncated:N] marker where N is the original byte length. The marker is byte-identical to what the audit payload-capture path emits for the same input, so an operator grepping [truncated: across span attributes and audit rows sees aligned output.

Default posture (no opt-in): the gen_ai.input.messages, gen_ai.output.messages, gen_ai.tool.call.arguments, gen_ai.tool.call.result, gen_ai.tool.description, and gen_ai.tool.definitions keys are absent from spans — not set to empty string. Backends that gate dashboards on "is this key present?" can distinguish "metadata-only by default" from "operator opted in but the field happened to be empty."

OTel semconv versioning note: the GenAI semantic conventions moved from flat-string (gen_ai.prompt, gen_ai.completion) to structured (gen_ai.input.messages, gen_ai.output.messages) attributes, and renamed gen_ai.system → gen_ai.provider.name. Forge emits the current structured keys and gen_ai.provider.name, and continues to emit the deprecated gen_ai.system for one release for compatibility. Backends that only recognize the older attributes should upgrade their semconv mapping or use a span processor to translate.

Guardrail spans (issue #161)

The LibraryGuardrailEngine opens a child span around every gate evaluation, symmetric to the guardrail_check audit-event emission. Trace consumers see "PII was masked here" inline with the LLM and tool spans without having to pivot to the audit stream.

Attribute reference:

Span status: block decisions stamp OTel Error status with the violation summary as the status description — surfaces blocked invocations as red bars in the trace UI without custom attribute queries. mask / warn decisions leave the default OK status.

Default posture: forge.guardrail.evidence is absent unless capture_content: true. The other five attributes are always present when a gate fires (cheap, no PII risk). When tracing is disabled, the noop tracer short-circuits and the spans are not produced at all.

Content-capture parity: the evidence attribute uses the exact same PrepareSpanContent(redact, maxBytes) pipeline as gen_ai.input.messages and gen_ai.tool.call.arguments — same vendor secret-token scrub, same 4 KiB byte cap, same …[truncated:N] marker. Operators get one mental model across all four content streams (LLM input / LLM output / tool args / tool result / guardrail evidence).

End-to-end propagation (Phase 5)

Forge installs the W3C tracecontext + baggage composite propagator on the OTel global at startup. The JSON-RPC dispatcher extracts inbound traceparent + baggage headers before opening its own span, so multi-hop A2A flows show as one connected trace:

orchestrator
    │  traceparent: 00-T-S1-01
    ▼
┌───────────────┐
│  forge agent  │  span_id=S2, parent=S1   (a2a.tasks/send)
│      ▼        │  span_id=S3, parent=S2   (agent.execute)
│      ▼        │  span_id=S4, parent=S3   (llm.completion)
└──────│────────┘
       ▼  traceparent: 00-T-S2-01  ← otelhttp re-injects on outbound
                                      via the egress-enforced transport
┌───────────────┐
│  downstream   │  span_id=S5, parent=S2   (a2a.tasks/send)
│  forge agent  │
└───────────────┘

All five spans share trace_id = T and chain by parent_span_id. The operator sees one connected flame graph.

A malformed inbound traceparent returns ctx unchanged from the propagator — Forge then starts a fresh root rather than carrying a broken context forward.

baggage (the other half of the composite) flows through to the handler ctx so application-level identifiers (tenant id, A/B bucket) travel with the trace.

Subprocess propagation (skills + tools)

The same composite propagator is used to plumb context into skill / tool subprocesses so an OTel-instrumented child binary's spans nest under the agent's tool.<name> span instead of starting a fresh root (issue #182).

Before invoking a skill, SkillCommandExecutor.Run (forge-cli/tools/exec.go) calls otel.GetTextMapPropagator().Inject(ctx, propagation.MapCarrier{}) and translates the produced W3C keys into env vars on the child's cmd.Env:

A curated subset of OTel SDK config env vars also passes through unchanged so the child exports to the same backend with consistent sampling:

OTEL_EXPORTER_OTLP_ENDPOINT           OTEL_TRACES_SAMPLER
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT    OTEL_TRACES_SAMPLER_ARG
OTEL_EXPORTER_OTLP_PROTOCOL           OTEL_RESOURCE_ATTRIBUTES
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL    OTEL_PROPAGATORS
OTEL_EXPORTER_OTLP_INSECURE           OTEL_SERVICE_NAME
OTEL_EXPORTER_OTLP_TRACES_INSECURE    OTEL_SDK_DISABLED

Deliberately excluded: OTEL_EXPORTER_OTLP_HEADERS (and its _TRACES_ sibling) — those can carry collector auth tokens. Treating them as secrets means a skill that needs collector auth declares them via its SKILL.md env.optional like every other credential.

When tracing is off, the global propagator's Inject is a no-op composite and writes nothing into the carrier — the child sees no TRACEPARENT and its env is byte-identical to pre-#182. Tracing-disabled deploys see no behavior change.

Binary skills. A metadata.forge.runtime: binary skill (see SKILL.md format) execs the declared binary directly — no intermediate bash fork — so the binary's own OTel SDK reads TRACEPARENT from env on startup and chains its spans under the agent's tool.<name> span. Bash-script skills still get the env injection too; the binary path just removes the wrapper hop.

Pinned by TestSkillCommandExecutor_TraceparentInjectedWhenCtxHasSpan, TestSkillCommandExecutor_TraceparentAbsentWhenNoSpan, TestSkillCommandExecutor_OTelSubsetPassedThrough (issue #182).

Audit ↔ trace cross-link (Phase 4)

Audit events emitted via EmitFromContext carry the active span's IDs:

{
  "event": "llm_call",
  "task_id": "t-1234",
  "trace_id": "4a8f95a0e1bedda42c9dd5350fb3b33a",
  "span_id":  "ad8b2c91e44f0a72",
  ...
}

• Pivot audit → trace: paste the trace_id into your backend's search box → land on the matching trace tree. Paste the span_id → land directly on the llm.completion child carrying matching gen_ai.usage.* tokens.
• Pivot trace → audit: copy the trace_id from Tempo / Jaeger / Honeycomb → grep the audit log for the matching row → get the FWS-8 payload metadata the trace doesn't carry.

Both fields use omitempty. When tracing is disabled (the default), audit JSON is byte-identical to the pre-Phase-4 shape — backward-compatible by construction. See Audit Logging for full schema details.

Egress-enforced OTLP transport

Forge wraps the OTLP HTTP exporter's transport with the same egress enforcer every other in-process HTTP client uses. The operator's egress allowlist therefore bounds where Forge can send spans — a misconfigured collector URL cannot exfiltrate span content to an unapproved destination.

forge package auto-injects the collector host into the build's egress allowlist so the generated NetworkPolicy admits OTLP traffic. The same auto-merge fires at forge run time so dev mode matches prod. No second egress edit, no NetworkPolicy patch.

# forge.yaml — this is sufficient. The collector is added automatically.
egress:
  mode: allowlist
  allowed_domains:
    - api.anthropic.com    # operator-declared
observability:
  tracing:
    enabled: true
    endpoint: https://otel-collector.monitoring.svc.cluster.local:4318/v1/traces
# → all_domains in egress_allowlist.json = [api.anthropic.com, otel-collector.monitoring.svc.cluster.local]

Disabled tracing produces no allowlist entry — turning tracing off in yaml does NOT leave a stale entry punched through the NetworkPolicy.

HTTP vs gRPC

Disabled-path semantics

When tracing is off (default, or enabled: false, or empty endpoint):

• forge-core/runtime.Tracer() returns the no-op tracer; spans are non-recording and near-zero cost.
• EmitFromContext does not stamp trace_id / span_id; audit JSON is byte-identical to pre-Phase-4.
• OTelDomain returns nil; no entry in egress_allowlist.json.
• observability.WrapHTTPTransport is a near pass-through (noop TracerProvider short-circuits span creation).

Telemetry failures never crash the agent. A misconfigured endpoint, a malformed traceparent, an unreachable collector — every failure mode falls through to the noop tracer with a warning in the ops log. The cli's resolver is the single place that fails loudly on bad config at startup.

Verification

Once configured:

# stub collector that prints every received span
docker run --rm -p 4318:4318 \
  otel/opentelemetry-collector-contrib:latest \
  --config=/dev/stdin <<'YAML'
receivers:
  otlp:
    protocols:
      http: { endpoint: 0.0.0.0:4318 }
exporters:
  debug: { verbosity: detailed }
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]
YAML

# in another terminal
forge run --otel-enabled \
  --otel-endpoint http://localhost:4318/v1/traces \
  --otel-sampler always_on

# fire a task
curl -H "Authorization: Bearer $(cat .forge/runtime.token)" \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","id":1,"method":"tasks/send",
          "params":{"id":"t-1","message":{"role":"user","parts":[{"kind":"text","text":"hi"}]}}}' \
     http://localhost:8080/

The collector should print one trace with a2a.tasks/send → agent.execute → llm.completion (× N) → tool.<name> (× M). The agent.execute span carries gen_ai.system, forge.task.id, forge.task.final_state. Each llm.completion carries gen_ai.usage.input_tokens / output_tokens. Each audit row in stderr now carries trace_id / span_id matching the spans.

Cross-references

• forge.yaml schema — the full schema reference including observability:.
• CLI reference — all --otel-* flags on forge run.
• Audit logging — schema including trace_id / span_id, complementary stream posture vs OTel.
• Egress control — auto-merge mechanics including the OTel collector.
• Workflow correlation — FWS-2 X-Workflow-* headers surface on the dispatch span.
• Deployment monitoring — operator-level integration with Tempo / Jaeger / Honeycomb.