Skip to content

Spans query

DatadogSpansQuery reads spans back from the Datadog API at https://api.<site> (the site setting, default datadoghq.com). It implements the shared TraceQueryClient interface, so code written for the other Grow Codes tracing SDKs works unchanged, and adds the Datadog spans search and analytics endpoints.

It needs an API key (DD_API_KEY) and an application key with the apm_read scope (DD_APP_KEY / DD_APPLICATION_KEY). Both are sent as DD-API-KEY / DD-APPLICATION-KEY and are always redacted. Without them the constructor (and client.spans) raises CONFIG.

Indexed spans only

The spans API only sees indexed spans (kept by a retention filter) and indexing takes from seconds to about a minute. getTrace therefore may return part of a trace; use waitForTrace in tests and keep a retention filter for the services you query.

Function Datadog endpoint Description
searchSpans({query, from, to, limit, sort}) POST /api/v2/spans/events/search lazy iterator over every matching span; follows the cursor; the time window is fixed on the first page (default: the last 15 minutes, -timestamp)
searchSpansPage(opts) same one page: spans, nextCursor, warnings, requestId; limit 1–1000
aggregateSpans(compute, groupBy, {query, from, to}) POST /api/v2/spans/analytics/aggregate count, cardinality, pc75–pc99, sum, min, max, avg, median; total or timeseries
getTrace(traceId) spans search trace_id:<id> all pages merged, sorted; null when nothing is indexed; window: the last 24 hours
getTrace(traceId, {preview: true}) GET /api/v2/trace/{id} the full trace from Datadog's preview trace-by-id API (unstable, 60 requests/minute)
searchTraces(query) spans search summaries of distinct traces by service, operation, tags, minDuration, maxDuration, start, end, limit (20); at most 5 pages
listServices({env}) GET /api/v2/apm/services sorted service names; env defaults to the configured environment, else all
listOperations(service) span analytics grouped by resource_name sorted resource names of a service (last 15 minutes)
waitForTrace(traceId, {timeout, interval, minSpans}) getTrace in a loop default timeout 30 s, interval 2 s (Datadog allows 50 searches per minute); TIMEOUT when not found
getDependencies(range) — always UNSUPPORTED: Datadog has no public service dependency API
ping() GET /api/v2/validate_keys validates both keys, returns the latency; 401 and 403 → AUTHENTICATION
quoteQueryValue(v) / traceSearchQueryString(q) — Datadog query syntax: quote a value / build the query of a TraceSearchQuery (tags sorted, durations in ns, * when empty)
close() — releases connections; later calls raise CANCELLED

Times (from, to) accept an absolute time, epoch milliseconds or a Datadog expression such as now-1h (TypeScript Date | number | string; Rust DatadogTime from SystemTime or &str; Go TimeAt(t) / TimeExpr("now-1h"); Python datetime | int | str; Java DatadogTime.at(…), epochMillis(…), expr(…)). Spans come back in the normalized model (Span, ids in hex), with the raw Datadog attributes next to them.

A 429 waits for Retry-After or X-RateLimit-Reset (within totalTimeout) before it raises RATE_LIMITED; 400 is VALIDATION with Datadog's error text, 403 on queries is PERMISSION_DENIED (missing apm_read).

import { DatadogSpansQuery } from "@grow-codes/datadog-apm";

const query = DatadogSpansQuery.fromEnv();           // or client.spans
for await (const s of query.searchSpans({ query: "service:checkout @http.status_code:500", from: "now-1h", limit: 50 })) {
  console.log(s.span.name, s.span.durationMicros);
}
const p90 = await query.aggregateSpans(
  [{ aggregation: "pc90", metric: "@duration" }],
  [{ facet: "resource_name", limit: 10 }],
  { query: "service:checkout" },
);
const trace = await query.getTrace(traceId);         // or { preview: true }
const recent = await query.searchTraces({ service: "checkout", limit: 10 });
console.log(await query.listServices(), await query.listOperations("checkout"));
const waited = await query.waitForTrace(traceId, { timeoutMs: 120_000 });
await query.close();
use futures::TryStreamExt;
use grow_datadog_apm::tracing::{TraceQueryClient as _, TraceSearchQuery, WaitForTraceOptions};
use grow_datadog_apm::{AggregateSpansOptions, DatadogSpansQuery, GetTraceOptions, SearchSpansOptions, SpanCompute, SpanGroupBy};

let query = DatadogSpansQuery::from_env()?;          // or client.spans()?
let spans: Vec<_> = query
    .search_spans(SearchSpansOptions::query("service:checkout").from("now-1h").limit(50))
    .try_collect()
    .await?;
let p90 = query
    .aggregate_spans(
        &[SpanCompute::new("pc90").metric("@duration")],
        &[SpanGroupBy::facet("resource_name").limit(10)],
        &AggregateSpansOptions { query: Some("service:checkout".into()), ..Default::default() },
    )
    .await?;
let trace = query.get_trace(&trace_id).await?;
let full = query
    .get_trace_with_options(&trace_id, &GetTraceOptions { preview: true, ..Default::default() })
    .await?;
let recent = query.search_traces(&TraceSearchQuery::service("checkout").limit(10)).await?;
let services = query.list_services().await?;
let waited = query.wait_for_trace(&trace_id, WaitForTraceOptions::default()).await?;
query.close();
query, err := datadogapm.DatadogSpansQueryFromEnv() // or client.Spans()
if err != nil {
    return err
}
defer query.Close()
for s, err := range query.SearchSpans(ctx, datadogapm.SearchSpansOptions{
    Query: "service:checkout", From: datadogapm.TimeExpr("now-1h"), Limit: 50,
}) {
    if err != nil {
        return err
    }
    fmt.Println(s.Span.Name, s.Span.DurationMicros)
}
p90, err := query.AggregateSpans(ctx,
    []datadogapm.SpanCompute{{Aggregation: datadogapm.AggregationPc90, Metric: "@duration"}},
    []datadogapm.SpanGroupBy{{Facet: "resource_name", Limit: 10}},
    datadogapm.AggregateSpansOptions{Query: "service:checkout"})
trace, err := query.GetTrace(ctx, traceID)
full, err := query.GetTraceWithOptions(ctx, traceID, datadogapm.GetTraceOptions{Preview: true})
recent, err := query.SearchTraces(ctx, datadogapm.TraceSearchQuery{Service: "checkout", Limit: 10})
services, err := query.ListServices(ctx)
waited, err := query.WaitForTrace(ctx, traceID, datadogapm.WaitForTraceOptions{Timeout: 2 * time.Minute})
from grow_datadog_apm import DatadogSpansQuery, SpanCompute, SpanGroupBy, TraceSearchQuery

with DatadogSpansQuery.from_env() as query:          # or client.spans
    for s in query.search_spans(query="service:checkout", from_="now-1h", limit=50):
        print(s.span.name, s.span.duration_micros)
    p90 = query.aggregate_spans(
        [SpanCompute("pc90", "@duration")],
        [SpanGroupBy("resource_name", limit=10)],
        query="service:checkout",
    )
    trace = query.get_trace(trace_id)                # or preview=True
    recent = query.search_traces(TraceSearchQuery(service="checkout", limit=10))
    print(query.list_services(), query.list_operations("checkout"))
    waited = query.wait_for_trace(trace_id, timeout_s=120)

AsyncDatadogSpansQuery has the same methods as coroutines (async for over search_spans).

try (DatadogSpansQuery query = DatadogSpansQuery.fromEnv()) { // or client.spans()
    for (DatadogSpan s : query.searchSpans(SearchSpansOptions.builder()
            .query("service:checkout").from(DatadogTime.expr("now-1h")).limit(50).build())) {
        System.out.println(s.span().name() + " " + s.span().durationMicros());
    }
    SpanAggregateResult p90 = query.aggregateSpans(
        List.of(SpanCompute.of("pc90", "@duration")),
        List.of(SpanGroupBy.of("resource_name", 10)),
        AggregateSpansOptions.of("service:checkout", null, null));
    Optional<Trace> trace = query.getTrace(traceId);
    Optional<Trace> full = query.getTrace(traceId, GetTraceOptions.previewApi());
    List<TraceSummary> recent = query.searchTraces(TraceSearchQuery.forService("checkout"));
    List<String> services = query.listServices();
    Trace waited = query.waitForTrace(traceId);
}

Every call has an …Async variant returning a CompletableFuture.

Mapping to the normalized model

Datadog search span Span
attributes.trace_id (decimal 64-bit or 32 hex) traceId (32 hex)
attributes.span_id / parent_id (decimal) spanId / parentSpanId (16 hex; 0 = root); a missing or 0 span id is VALIDATION
resource_name, else operation_name name (both also kept as attributes)
start_timestamp / end_timestamp (ISO-8601) startMicros / endMicros / durationMicros
attributes.error / status status (error with error.message / error.msg, ok for otel.status_code=Ok)
span.kind kind
service serviceName

Large integers (ids, nanosecond times above 2^53) are parsed without loss in every language (TypeScript parseJsonPreservingBigInts, Go DecodeJSONPreservingNumbers).