Distributed Tracing

What GoFr propagates

OpenTelemetry setup in GoFr (verified in pkg/gofr/otel.go) configures:

otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{}, propagation.Baggage{}))

That means every GoFr binary, on boot, registers two propagators:

• W3C TraceContext — the standard traceparent and tracestate headers.
• W3C Baggage — the baggage header for cross-service key/value context.

The HTTP server middleware extracts both on every inbound request (pkg/gofr/http/middleware/tracer.go). The HTTP service client injects both on every outbound request (pkg/gofr/service/new.go).

Trace ID format

W3C trace IDs are 16-byte (32 hex character) values; span IDs are 8-byte (16 hex). When a request carries a trace context, GoFr's logger writes the trace ID to the top-level trace_id field on the JSON log envelope (the field is omitempty, so it only appears when a trace context is set). On HTTP request logs specifically, the message field is itself a RequestLog struct that carries its own nested trace_id and span_id alongside method, uri, etc. — that nested copy is only present on the HTTP middleware's request log line, not on every log entry. Correlate logs and traces using either occurrence (see Production Logging for the exact log shape and shipper configuration).

A trace looks the same across HTTP and gRPC: the same trace ID, with each service contributing one or more spans.

Configuration

Tracing is opt-in. Set:

The zipkin value emits a deprecation warning at startup and recommends switching to otlp (verified in pkg/gofr/otel.go).

Two failure modes are worth knowing before you set these:

• An unparsable TRACER_RATIO samples everything. TRACER_RATIO=10% is not a number, so it is rejected and the ratio falls back to 1 — every trace is exported, and the error names both the rejected value and the ratio actually applied. Use 0.1, not 10%, and watch for that log line if export volume is higher than you expect.
• An unrecognized TRACE_EXPORTER disables tracing rather than pretending to work. The app starts, logs the unsupported name, and installs a non-recording provider. Trace and span IDs stay valid, so X-Correlation-ID and the trace_id log field keep working — but no span leaves the process. Check the startup log if a backend you configured is receiving nothing.

A credential inside TRACER_URL — https://user:token@collector:4317, or ?api-key=... on a Zipkin spans URL — is replaced with REDACTED wherever GoFr logs the endpoint: at startup, and in the export errors the OTel SDK raises later. The host, port and path are kept, so the line still tells you where spans are going. A secret pasted into TRACE_EXPORTER is redacted too, while a plain typo (otpl) is echoed back so you can spot it.

Resource attributes from the environment

Every exported span carries a resource — the attributes that describe which service emitted it. GoFr fills it with service.name (from APP_NAME) and framework_version, and merges in the standard OTEL_RESOURCE_ATTRIBUTES variable, so attributes only the operator knows reach the backend without a code change:

OTEL_RESOURCE_ATTRIBUTES="deployment.environment=prod,cloud.region=asia-south1"

service.name is the one exception. GoFr always takes it from APP_NAME, so OTEL_SERVICE_NAME — and a service.name= entry inside OTEL_RESOURCE_ATTRIBUTES — is discarded. Setting either logs a warning naming the value that was dropped:

traces: service.name="checkout-from-env" from the environment is ignored; GoFr sets it from APP_NAME ("checkout"). Set APP_NAME to rename the service.

Rename the service with APP_NAME. The reason for the exception is consistency across signals: GoFr's metrics resource resolves service.name from APP_NAME the same way, and letting only traces follow OTEL_SERVICE_NAME would report one service name to your trace backend and a different one to your metric backend — breaking the join between a service's traces and its metrics exactly where you need it.

End-to-end example

Service A receives an HTTP request, calls Service B over HTTP, which writes to a database. With GoFr defaults, the trace contains:

1. Server span on A (from the HTTP middleware).
2. Custom application spans on A if you call c.Trace("step-name") — see Custom Spans in Tracing.
3. Client span on A's outbound HTTP call to B.
4. Server span on B from the same traceparent.
5. Spans for B's database call (when using GoFr's instrumented datasources).

Both A and B point TRACE_EXPORTER=otlp and TRACER_URL at the same collector. In Jaeger or Tempo, a single search by trace ID shows the whole path.

gRPC

The gRPC server in GoFr also participates in tracing. Cross-protocol traces — HTTP into gRPC and back — work because both servers use the same OpenTelemetry SDK and propagators.

Pub/Sub

Trace propagation through a message bus is partial. The Google Pub/Sub datasource injects trace context into message attributes (verified in pkg/gofr/datasource/pubsub/google/tracing.go), which means a producer span and consumer span share the same trace ID. Other Pub/Sub backends (Kafka, NATS, SQS, MQTT, EventHub) may or may not propagate trace context end-to-end; if your span graph breaks at the bus, log the trace ID into the message payload manually as a fallback so downstream logs can still be correlated.

Sampling: keep it consistent

If A samples at 10% and B samples at 100%, you'll have lots of B-only traces with no parent — useless. Two rules:

• Use head-based sampling (TRACER_RATIO) consistently across all services in the request path.
• Or use tail-based sampling at the collector level, where the decision happens after the trace is assembled.

For a typical OTLP collector setup, configure tail sampling once in the collector and set TRACER_RATIO=1 in every service so all spans are exported and the collector decides what to keep.

Visualizing in Jaeger

Run Jaeger in OTLP-receiver mode, point GoFr at the gRPC OTLP endpoint (typically host:4317), and traces appear in the Jaeger UI within a second or two.

TRACE_EXPORTER=otlp
TRACER_URL=jaeger.observability.svc.cluster.local:4317
TRACER_RATIO=1

For Tempo or Honeycomb, point at their OTLP gRPC endpoints and add TRACER_HEADERS for any required auth.

Custom application spans

For business-level operations inside a handler, wrap them with c.Trace("name"). This is the recommended way to add span granularity without touching the OTel SDK directly. See Custom Spans in Tracing.

Common gotchas

• Mismatched propagators — if a non-GoFr service in the chain uses B3 instead of W3C, traces split. Standardize on W3C TraceContext across the fleet.
• Sidecar tracing — Istio and Linkerd inject their own spans. Configure them to use the same backend, not a parallel one.
• Logs without trace IDs — if trace_id is empty in a log, the request didn't carry a traceparent. Likely the entry point (Ingress, gateway) is not adding one.
• High cardinality span names — never put a path parameter (e.g., /orders/12345) directly in a span name. Use the route template.
• A short SHUTDOWN_GRACE_PERIOD cuts the final flush off — on shutdown GoFr flushes the spans still sitting in the batch processor before it closes the datasources, and that flush is bounded by SHUTDOWN_GRACE_PERIOD along with the rest of the drain. Against a collector that is unreachable, the OTLP exporter spends its own 10s export timeout there. Keep the grace period comfortably above that, or a pod's last spans are dropped exactly when you are debugging why it went away.

What spans cost

Each exported span is a few hundred bytes over the network plus storage. At 100% sampling and high RPS, span volume can dominate egress. Sample the way you sample logs: aggressively for routine traffic, fully for errors and slow requests (tail-based sampling).