Skip to content

Tracing

Tracing is built on the official OpenTelemetry SDK of each language. This SDK adds a common setup function with the right exporter defaults for Datadog APM, span helpers, propagation, framework adapters and — when Datadog APM has a query API — a TraceQueryClient to read traces back. The service setup function (initTracing) and the query client accessor are described on Functions.

Set up

initTracing(config) installs a tracer provider (globally unless setGlobal is false) and returns a TracingHandle with tracer(name), forceFlush() and an idempotent shutdown().

Setting Default Notes
serviceName required service.name resource attribute
serviceVersion, environment, resourceAttributes none service.version, deployment.environment, extra attributes
endpoint, protocol Datadog APM defaults grpc, http/protobuf or native
headers none auth headers are redacted in toString
sampler, samplerRatio parentbased_always_on, 1.0 also always_on, always_off, traceidratio, parentbased_always_off, parentbased_traceidratio
propagators tracecontext, baggage b3, b3multi, datadog, sw8 where supported
batch queue 2048, batch 512, delay 5 s, timeout 30 s span batch processor
compression gzip or none
tls, setGlobal none, true

Always call shutdown() before the process exits so queued spans are exported.

import { initTracing } from "@grow-codes/datadog-apm";

const tracing = initTracing({ serviceName: "checkout", serviceVersion: "1.4.0", environment: "prod" });
process.on("SIGTERM", () => void tracing.shutdown());
use grow_datadog_apm::tracing::TracingConfig;

let mut config = TracingConfig::new("checkout");
config.service_version = Some("1.4.0".into());
let tracing = grow_datadog_apm::init_tracing(config)?;
// … at shutdown:
tracing.shutdown().await?;
tracing, err := datadogapm.InitTracing(ctx, datadogapm.TracingConfig{
    ServiceName:    "checkout",
    ServiceVersion: "1.4.0",
    Environment:    "prod",
})
if err != nil {
    return err
}
defer tracing.Shutdown(context.Background())
from grow_datadog_apm import TracingConfig, init_tracing

tracing = init_tracing(TracingConfig(service_name="checkout", service_version="1.4.0"))
try:
    run_app()
finally:
    tracing.shutdown()
TracingConfig config = TracingConfig.builder("checkout").serviceVersion("1.4.0").build();
try (TracingHandle tracing = DatadogApmTracing.init(config)) {
  runApp();
} // shutdown() flushes queued spans

Spans

Helper What it does
withSpan(name, fn, { attributes, kind, links, tracer }) runs fn in a new span; records the exception and sets ERROR status on failure, then rethrows
traced wraps a function (decorator/annotation where the language has one)
currentSpan(), currentTraceId(), currentSpanId() the active span and its ids (lower-case hex)
addEvent(name, attributes?), recordException(error), setError(error) act on the active span
traceFields() { trace_id, span_id } for log correlation (snake_case on purpose)
import { addEvent, traced, withSpan } from "@grow-codes/datadog-apm";

const total = await withSpan("checkout.total", async (span) => {
  span.setAttribute("cart.items", 3);
  addEvent("prices.loaded");
  return computeTotal();
}, { attributes: { "cart.id": "c-1" } });

const charge = traced(async (amount: number) => pay(amount), "checkout.charge");

class Orders {
  @traced("orders.create")
  async create() { /* … */ }
}
use grow_datadog_apm::tracing::{add_event, traced, with_span, SpanOptions};

let total = with_span("checkout.total", SpanOptions::default(), async {
    add_event("prices.loaded", Default::default());
    compute_total().await
})
.await?;

let charged = traced("checkout.charge", pay(42)).await?;
// or annotate functions with #[tracing::instrument] and use the otel_layer() bridge
total, err := datadogapm.WithSpan(ctx, "checkout.total", func(ctx context.Context) (int, error) {
    datadogapm.AddEvent(ctx, "prices.loaded", nil)
    return computeTotal(ctx)
}, datadogapm.WithSpanAttributes(map[string]any{"cart.id": "c-1"}))

err = datadogapm.Traced(ctx, "checkout.charge", func(ctx context.Context) error {
    return pay(ctx, 42)
})
from grow_datadog_apm import add_event, traced, with_span

with with_span("checkout.total", attributes={"cart.id": "c-1"}) as span:
    add_event("prices.loaded")
    total = compute_total()

@traced(name="checkout.charge")
async def charge(amount: int) -> None:
    await pay(amount)
int total = Spans.withSpan("checkout.total", () -> {
  Spans.addEvent("prices.loaded");
  return computeTotal();
});
Traced.run("checkout.charge", () -> pay(42));

Interfaces can be traced with the @Traced annotation through TracedProxy.

Propagation

inject(carrier) writes the current context into a string map (for example message headers) and extract(carrier) reads it back. injectHeaders/extractHeaders do the same for HTTP headers.

import { extract, inject } from "@grow-codes/datadog-apm";

const headers: Record<string, string> = {};
inject(headers);                  // traceparent, tracestate, baggage
const parent = extract(incoming); // Context for a child span
let mut carrier = std::collections::HashMap::new();
grow_datadog_apm::tracing::inject(&mut carrier);
let parent = grow_datadog_apm::tracing::extract(&incoming);
carrier := map[string]string{}
datadogapm.Inject(ctx, carrier)
ctx = datadogapm.Extract(ctx, incoming)
from grow_datadog_apm import extract, inject

carrier: dict[str, str] = {}
inject(carrier)
parent = extract(incoming)
Map<String, String> carrier = new HashMap<>();
Spans.inject(carrier);
Context parent = Spans.extract(incoming);

Framework adapters

Language Adapters
TypeScript Express middleware, Fastify plugin, NestJS interceptor, tracedFetch(fetch)
Rust otel_layer() for tracing-opentelemetry, tower TraceLayer (axum, hyper; feature tower), TracingMiddleware for reqwest-middleware (feature reqwest-middleware)
Go HTTPMiddleware (server) and HTTPTransport (http.RoundTripper); gRPC interceptors in grpcadapter
Python TracingASGIMiddleware, TracingWSGIMiddleware, httpx_event_hooks(), instrument_requests_session()
Java Servlet Filter (jakarta.servlet), TracedHttpClient for java.net.http, OkHttp Interceptor

Read traces back

TraceQueryClient is available when Datadog APM has a query API (otherwise its methods raise UNSUPPORTED; see Compatibility). Ids are lower-case hex; 64-bit trace ids are left-padded to 32 characters.

Function Returns
getTrace(traceId) Trace { traceId, spans } (spans sorted by start time) or null
searchTraces(query) TraceSummary[]; query: service, operation, tags, minDuration, maxDuration, start, end, limit (20)
listServices() / listOperations(service) names
getDependencies({ start, end }) DependencyLink { parent, child, callCount }[]
waitForTrace(traceId, { timeout, interval, minSpans }) Trace once it has minSpans spans; TIMEOUT otherwise

buildTree(spans) turns the flat span list into a parent/child tree.

const trace = await traces.waitForTrace(traceId, { timeoutMs: 15_000, minSpans: 2 });
for (const span of trace.spans) console.log(span.name, span.durationMicros);
const recent = await traces.searchTraces({ service: "checkout", limit: 10 });
let trace = traces.get_trace(&trace_id).await?;
let services = traces.list_services().await?;
trace, err := traces.GetTrace(ctx, traceID)
services, err := traces.ListServices(ctx)
trace = traces.wait_for_trace(trace_id, timeout_s=15.0, min_spans=2)
for span in trace.spans:
    print(span.name, span.duration_micros)
Optional<Trace> trace = traces.getTrace(traceId);
List<String> services = traces.listServices();

Testing

The testing module has an in-memory span exporter and trace query client (see Docker testing).