Tracing setup¶
initTracing(config) sets up the official OpenTelemetry SDK of each language to export spans over
OTLP to the Datadog Agent and returns a DatadogTracingHandle with tracerProvider,
tracer(name, version), forceFlush(), an idempotent shutdown() and the exporter endpoint.
The Agent converts the OTLP spans to Datadog spans and sends them to Datadog with its own API key,
so the application needs no Datadog credentials.
| Default | |
|---|---|
| Protocol | http/protobuf to http://<agent>:4318/v1/traces; grpc to http://<agent>:4317 |
| Agent host | the host of url (DD_AGENT_HOST, DD_TRACE_AGENT_URL, GROW_DATADOG_APM_URL), else 127.0.0.1 |
| Service, version, env | serviceName, serviceVersion, environment from the config, else DD_SERVICE, DD_VERSION, DD_ENV |
| Resource attributes | DD_TAGS plus resourceAttributes (explicit keys win) |
| Propagators | datadog, tracecontext, baggage (Datadog headers win on conflict) |
| Sampler | parentbased_always_on |
| Compression | gzip |
setGlobal |
true |
The Agent must have its OTLP receiver on:
DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT=0.0.0.0:4318 and/or
DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT=0.0.0.0:4317
(Datadog guide).
import { initTracing, withSpan } from "@grow-codes/datadog-apm";
const tracing = initTracing({ serviceName: "checkout", serviceVersion: "1.4.0", environment: "prod" });
console.log(tracing.endpoint); // http://127.0.0.1:4318/v1/traces
await withSpan("charge", async () => {
// ...
});
await tracing.shutdown();
use grow_datadog_apm::{init_tracing, tracing::TracingConfig};
let tracing = init_tracing(
TracingConfig::new("checkout").with_service_version("1.4.0").with_environment("prod"),
)?;
println!("{}", tracing.endpoint());
// ... spans via tracing.tracer("checkout", None), with_span, or the `tracing` crate layer
tracing.shutdown().await?;
h, err := datadogapm.InitTracing(ctx, datadogapm.TracingConfig{
ServiceName: "checkout", ServiceVersion: "1.4.0", Environment: "prod",
})
if err != nil {
return err
}
defer h.Shutdown(context.Background())
fmt.Println(h.Endpoint())
err = datadogapm.Traced(ctx, "charge", func(ctx context.Context) error { return nil })
DatadogApmClient.initTracing(config) does the same with the client's configuration as the
defaults and shuts the handle down in client.close().
Configuration sources¶
Unset tracing fields come from DatadogApmConfig (explicit config passed to initTracing, else
the environment, see Functions). The shared OpenTelemetry
variables (OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG, OTEL_PROPAGATORS,
OTEL_RESOURCE_ATTRIBUTES, OTEL_EXPORTER_OTLP_*_HEADERS, …_COMPRESSION, …_TIMEOUT,
…_CERTIFICATE) are read as described in Tracing.
| Language | Pass Agent settings instead of the environment |
|---|---|
| TypeScript | initTracing(config, { config: DatadogApmConfig \| options, env: map \| false }) |
| Rust | init_tracing_with_options(config, InitTracingOptions { config, env, ignore_env }) |
| Go | InitTracingWithOptions(ctx, cfg, InitTracingOptions{Config, Env, IgnoreEnv, SpanProcessors}) |
| Python | init_tracing(config, apm_config=..., env=..., ignore_env=True, span_processors=[...], **fields) |
| Java | DatadogTracing.initTracing(config, DatadogApmConfig) |
Unset fields. TypeScript (Partial<TracingConfig>) and Go (zero values) can leave a field
unset. Rust, Python and Java always carry the template defaults of TracingConfig, so there a
field equal to its template default counts as unset: protocol http/protobuf means "the
configured protocol" (GROW_DATADOG_APM_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_*_PROTOCOL), the
propagator list tracecontext, baggage means "the Datadog default list", and an empty service name
(Rust, Python) means DD_SERVICE. To force HTTP while the environment says gRPC, pass an explicit
endpoint or change the environment. See Compatibility.
The protocol native raises CONFIG: this package always exports OTLP.
Unified service tagging¶
| Config | OpenTelemetry resource attribute | Datadog tag |
|---|---|---|
serviceName |
service.name |
service (missing → the Agent uses otlpresourcenoservicename) |
serviceVersion |
service.version |
version |
environment |
deployment.environment.name and the legacy deployment.environment |
env |
tags / DD_TAGS |
each key:value as an attribute |
copied to the span meta |
Both environment keys are set in every language; the Agent maps either to env.
Propagators¶
The default list is datadog, tracecontext, baggage. orderPropagators(names) moves
tracecontext in front of datadog (every other name keeps its place), so the Datadog propagator
sees the W3C result and applies Datadog's own rules (verified against ddtrace):
- Datadog headers win for the trace id.
- A W3C context with a different trace id becomes a span link with
reason=terminated_contexton the first span started from the extracted context. - The same trace id with a different parent: the W3C parent id wins.
Inject writes x-datadog-trace-id (low 64 bits, decimal), x-datadog-parent-id (decimal),
x-datadog-sampling-priority (1/0) and, for a 128-bit id, x-datadog-tags: _dd.p.tid=<high
64 bits hex>, plus traceparent/tracestate and baggage. Details in
Propagation and ids.
Error attributes¶
The Agent keeps an OpenTelemetry exception event only as JSON in the span meta and sets
error.msg from the status message; it does not fill error.type or error.stack. Datadog
Error Tracking needs those, so this package sets the span attributes error.type,
error.message and error.stack:
| Function | TypeScript | Rust | Go | Python | Java |
|---|---|---|---|---|---|
| exception event + attributes, status unchanged | recordException(err) |
record_exception(&err) |
RecordDatadogException(ctx, err) |
record_exception(err) |
DatadogSpanErrors.recordException(err) |
| ERROR status + event + attributes | setError(err) |
set_error(&err) |
SetDatadogError(ctx, err) |
set_error(err) |
DatadogSpanErrors.setError(err) |
| attributes only | setErrorAttributes(err, span?) |
set_error_attributes(&err) |
SetErrorAttributes(ctx, err) |
set_error_attributes(err, span=None) |
DatadogSpanErrors.setErrorAttributes([span,] err) |
| the attributes as a map | datadogErrorAttributes(err) |
datadog_error_attributes(&err) |
DatadogErrorAttributes(err) |
datadog_error_attributes(err) |
DatadogSpanErrors.datadogErrorAttributes(err) |
- TypeScript, Python and Rust export the Datadog versions under the generic names
recordException/setErrorat the package root (they replace the generic helpers of the shared tracing module). Rust also has the Go-style aliasesrecord_datadog_exception/set_datadog_error. In Go the generic namesRecordException/SetErrorbelong to the shared helpers, so the Datadog versions areRecordDatadogException/SetDatadogError. - Spans exported through
initTracingget the attributes even when the generic helpers,withSpan,tracedor the HTTP adapters recorded the error: the attributes are derived from the lastexceptionevent (or, for an ERROR status without an event, from the status message). Attributes that are already set are kept.
error.type per language (the exception type name in the language's own format):
| Language | error.type for a plain error |
Rule |
|---|---|---|
| TypeScript | Error, TypeError |
error.name, else the constructor name; a thrown non-Error gives typeof ("Error" for strings) |
| Rust | std::io::error::Error |
std::any::type_name::<E>(); error.stack is a backtrace of the caller |
| Go | *errors.errorString, *fs.PathError |
the type as reflect prints it (pkgpath.Name for named types); error.stack is the calling goroutine's stack |
| Python | ValueError |
type(error).__name__ |
| Java | java.lang.IllegalStateException |
the canonical class name (as OpenTelemetry's exception.type) |
Queries in Datadog should therefore not assume one naming scheme across services written in different languages.
Where the attributes and links are added¶
| Language | Error attributes | terminated_context link |
|---|---|---|
| TypeScript | DatadogErrorSpanProcessor (before the span ends) |
DatadogSpanLinkProcessor (span start) |
| Go | DatadogErrorExporter (export time) |
DatadogSpanLinkProcessor (span start) |
| Python | DatadogErrorExporter (export time) |
DatadogSpanLinkProcessor (span start) |
| Rust | DatadogSpanExporter (export time) |
DatadogSpanExporter (export time) |
| Java | DatadogSpanExporter (export time; the link travels on the winning remote span context) |
DatadogSpanExporter (export time) |
The template initTracingWith of Rust and Java accepts no extra span processors, so both use an
exporter wrapper. initTracing installs all of them. Spans exported through your own tracer
provider (not created by initTracing) need this wiring by hand: wrap your exporter
(DatadogErrorExporter / DatadogSpanExporter), add the processors (TypeScript, Go, Python), or
add terminatedContextLinks(ctx) to the span links yourself when you start a server span from an
extracted context.
What the Agent makes of the spans¶
Verified against datadog/agent:7.85.4:
| OpenTelemetry | Datadog span |
|---|---|
128-bit traceId |
trace_id = low 64 bits (decimal); the full id is kept in meta otel.trace_id; no _dd.p.tid for OTLP spans |
spanId / parentSpanId |
span_id / parent_id (decimal) |
| span name | resource (for SERVER spans with http.route: GET /items/{id}) |
| span kind | name (Internal, server.request, http.server.request, …) and meta span.kind |
| status ERROR + message | error=1, meta error.msg |
| events / links | meta events / _dd.span_links (JSON) |
| string / bool attributes | meta; numbers → metrics |
datadogTraceSpanToSpan maps such a span back to the normalized Span: name is the Datadog
resource (= the OpenTelemetry span name), the Datadog operation is kept as attribute
operation_name.