Skip to content

API reference

Generated reference documentation for each package. The pages are rebuilt and published with this site from main.

Language Reference Generated by
TypeScript TypeScript API TypeDoc
Rust Rust API · docs.rs rustdoc
Go pkg.go.dev Go doc comments
Python below mkdocstrings
Java Java API Javadoc

Build them locally with make docs (per language: make -C <lang> docs).

Python

Grow Codes SDK for Grafana Tempo.

from grow_grafana_tempo import TempoClient, TraceQL, init_tracing client = TempoClient.from_env() # doctest: +SKIP client.query.search(TraceQL.span({"resource.service.name": "api"})) # doctest: +SKIP

AsyncBaseTraceQueryClient

Async :class:BaseTraceQueryClient.

query_trace_id

query_trace_id(trace_id: str) -> str

The id wait_for_trace passes to get_trace; override to keep native ids.

AsyncHttpTransport

Asynchronous transport (one per async client); options as for :class:HttpTransport.

Cancelling the calling task raises :class:CancelledError (an asyncio.CancelledError subclass) so asyncio's cancellation machinery keeps working. Every call after :meth:aclose raises :class:CancelledError ("client is closed").

aclose async

aclose() -> None

Close the connection pool and abort backoff sleeps (idempotent).

request async

request(req: Request) -> Response

Send req with retries; return the response or raise the mapped error.

AsyncTempoClient

Asynchronous Grafana Tempo client; use async with or call :meth:aclose.

Same methods as :class:TempoClient, awaited; client.query is an :class:AsyncTempoQueryClient. Cancel a call by cancelling its task: the SDK raises :class:CancelledError, which is also an :class:asyncio.CancelledError.

query property

query: AsyncTempoQueryClient

The async query API sharing this client's transport and tenant.

resources property

resources: ResourceRegistry

Background resources stopped by :meth:aclose.

aclose async

aclose() -> None

Stop background resources and close connections (idempotent).

build_info async

build_info() -> BuildInfo

See :meth:TempoClient.build_info.

echo async

echo() -> str

See :meth:TempoClient.echo.

from_env classmethod

from_env(env: Mapping[str, str] | None = None, **overrides: Any) -> AsyncTempoClient

Build a client from the environment (see :meth:TempoClient.from_env).

health async

health() -> HealthStatus

See :meth:TempoClient.health.

ping async

ping() -> float

See :meth:TempoClient.ping.

ready async

ready() -> bool

See :meth:TempoClient.ready.

request async

request(req: Request) -> Response

Send a raw request through the client's transport.

server_version async

server_version() -> str

See :meth:TempoClient.server_version.

AsyncTempoQueryClient

Bases: AsyncBaseTraceQueryClient

Asynchronous :class:TempoQueryClient (same methods, awaited). Cancel a call by cancelling its task.

client property

client: AsyncTempoClient

The underlying :class:AsyncTempoClient.

aclose async

aclose() -> None

Close the underlying client when this query client owns it (idempotent).

build_info async

build_info() -> BuildInfo

See :meth:TempoQueryClient.build_info.

echo async

echo() -> str

See :meth:TempoQueryClient.echo.

from_env classmethod

from_env(env: Mapping[str, str] | None = None, **overrides: Any) -> AsyncTempoQueryClient

An owned async query client configured from the environment.

get_dependencies async

get_dependencies(*, start: datetime | None = None, end: datetime | None = None) -> list[DependencyLink]

Always :class:UnsupportedError: Tempo has no dependency API.

get_trace async

get_trace(trace_id: str, *, start: _TimeLike | None = None, end: _TimeLike | None = None) -> Trace | None

See :meth:TempoQueryClient.get_trace.

list_operations async

list_operations(service: str, *, start: _TimeLike | None = None, end: _TimeLike | None = None) -> list[str]

See :meth:TempoQueryClient.list_operations.

list_services async

list_services(*, start: _TimeLike | None = None, end: _TimeLike | None = None) -> list[str]

See :meth:TempoQueryClient.list_services.

ready async

ready() -> bool

See :meth:TempoQueryClient.ready.

search async

search(traceql: str | TraceQL | None = None, *, start: _TimeLike | None = None, end: _TimeLike | None = None, limit: int | None = None, spss: int | None = None) -> TempoSearchResult

See :meth:TempoQueryClient.search.

search_tag_values async

search_tag_values(tag: str, *, query: str | TraceQL | None = None, start: _TimeLike | None = None, end: _TimeLike | None = None) -> list[TagValue]

See :meth:TempoQueryClient.search_tag_values.

search_tags async

search_tags(*, scope: TagScope | None = None, query: str | TraceQL | None = None, start: _TimeLike | None = None, end: _TimeLike | None = None) -> dict[str, list[str]]

See :meth:TempoQueryClient.search_tags.

search_traces async

search_traces(query: TraceSearchQuery | None = None) -> list[TraceSummary]

See :meth:TempoQueryClient.search_traces.

traceql_metrics async

traceql_metrics(query: str | TraceQL, *, start: _TimeLike | None = None, end: _TimeLike | None = None, since_s: float | timedelta | None = None, step_s: float | timedelta | None = None, exemplars: int | None = None) -> TraceQLMetricsResult

See :meth:TempoQueryClient.traceql_metrics.

AsyncTraceQueryClient

Bases: Protocol

Asynchronous trace query API (same methods, awaited).

AuthenticationError

Bases: TempoError

Credentials missing or rejected (HTTP 401).

BaseConfig dataclass

Common, immutable client settings. Durations are float seconds (timedelta accepted).

Service configs subclass this (@dataclass(frozen=True, repr=False)), override url with the service default, add their own fields (Secret for credentials), extend :meth:_env_values and :meth:validate, and list vendor variables in ENV_VENDOR.

from_env classmethod

from_env(env: Mapping[str, str] | None = None, **overrides: Any) -> _C

Build from the environment, then validate.

Precedence: explicit keyword > GROW_GRAFANA_TEMPO_<NAME> > vendor variables > default. env defaults to os.environ; pass a dict in tests.

known_secrets

known_secrets() -> list[str]

Revealed values of every :class:Secret field (used to redact error text).

validate

validate() -> _C

Check the common rules; raise :class:ConfigError. Returns self.

BaseTraceQueryClient

Base for service query clients; every method raises UnsupportedError until overridden (wait_for_trace uses :func:wait_for_trace on top of get_trace, passing the id through :meth:query_trace_id).

query_trace_id

query_trace_id(trace_id: str) -> str

The id wait_for_trace passes to get_trace; override to keep native ids.

BuildInfo dataclass

Build information from /api/status/buildinfo (empty strings for unset fields).

CancelToken

Thread-safe cancellation flag passed to sync calls as cancel=.

Cancelling interrupts an in-flight request (the call returns immediately with :class:CancelledError; the abandoned HTTP exchange finishes in the background) and any retry backoff sleep.

cancel

cancel() -> None

Request cancellation (idempotent).

on_cancel

on_cancel(callback: Callable[[], None]) -> Callable[[], None]

Run callback on cancellation (now if already cancelled); returns an unregister.

wait

wait(timeout: float | None = None) -> bool

Block until cancelled or timeout; return whether the token is cancelled.

CancelledError

Bases: TempoError, CancelledError

The caller cancelled the operation (task cancel, :class:CancelToken, client close).

Also an :class:asyncio.CancelledError so asyncio keeps treating the task as cancelled.

ConfigError

Bases: TempoError

Invalid or missing configuration.

ConflictError

Bases: TempoError

Conflict or failed precondition (HTTP 409, 412, CAS mismatch).

EnvReader

Reads GROW_<KEY_UPPER>_<NAME> and vendor variables from an injectable mapping.

Every getter returns None when no variable is set (empty strings count as unset). A present but invalid value raises :class:ConfigError that names the variable, never its value.

duration

duration(name: str, vendor_names: Iterable[str] = ()) -> builtins.float | None

Duration in seconds.

lookup

lookup(name: str, vendor_names: Iterable[str] = ()) -> tuple[str, str] | None

Return (variable_name, raw_value) for the first variable that is set.

ErrorCode

Bases: str, Enum

Canonical error codes; the value is the wire form used by the conformance suite.

ErrorInfo dataclass

Passed to Hooks.on_error; error is the (redacted) SDK error.

HealthStatus dataclass

Result of health(); status is ok/degraded/unhealthy/unreachable or a vendor word.

Hooks dataclass

Optional callbacks invoked on every attempt. Exceptions raised by hooks are ignored.

HttpTransport

Synchronous transport (one per client).

Service options (all optional):

  • request_id_headers: vendor request id headers, checked before :data:REQUEST_ID_HEADERS.
  • retry_after_headers: headers read (in order) for the delay a 429/503 response asks for; default ("retry-after",), e.g. ("retry-after", "x-ratelimit-reset"). See :func:retry_after_from_headers.
  • error_mapper: mapper(error, response) returns a service-specific error for an HTTP error response (or None to keep the default). The mapped error is what Hooks.on_error, the retry planner (retry_after) and the caller see; whether a status is retried stays the SPEC rule. Exceptions raised by the mapper are logged and ignored.

Every call after :meth:close raises :class:CancelledError ("client is closed").

close

close() -> None

Close the connection pool and abort backoff sleeps (idempotent).

request

request(req: Request, *, cancel: CancelToken | None = None) -> Response

Send req with retries; return the response or raise the mapped error.

LoggerLike

Bases: Protocol

Structured logger interface accepted in configuration (SPEC Logger).

MetricExemplar dataclass

An exemplar span behind a series (labels include trace:id).

MetricSample dataclass

One sample of a TraceQL metrics series (Tempo omits zero values; they read as 0).

MetricSeries dataclass

One series: labels (__name__, by() attributes, quantile p), samples, exemplars.

NotFoundError

Bases: TempoError

Missing record (HTTP 404).

PermissionDeniedError

Bases: TempoError

Authenticated but not allowed (HTTP 403).

RateLimitError

Bases: TempoError

Too many requests (HTTP 429); retry_after holds the requested delay when known.

Request dataclass

One logical API call (possibly several attempts).

path is joined to the base URL ({name} style values must already be encoded; use :func:urllib.parse.quote); an absolute http:///https:// URL is used as is (for pagination links the service returns; the default headers, config.headers included, are sent there too, so only pass URLs of the same service). json is encoded when not None; content sends raw bytes. idempotent overrides the method rule; an idempotency_key sends the Idempotency-Key header and marks the request idempotent. raise_for_status=False returns error responses instead of raising (retries still apply). Statuses in allow_status are expected answers (a 404 that carries data, a health endpoint's 503): the response is returned at once, like a success, without retries or on_error.

RequestInfo dataclass

Passed to Hooks.on_request; url and headers are redacted.

ResourceRegistry

Holds :class:Stoppable objects; :meth:stop_all stops each once, ignoring errors.

register

register(item: Stoppable) -> None

Track item; when the registry is already closed the item is stopped now.

Response dataclass

A completed HTTP response (body fully read).

json

json() -> Any

Decode the body as JSON; :class:ValidationError when it is not valid JSON.

ResponseInfo dataclass

Passed to Hooks.on_response; headers are the redacted response headers (rate limit headers and the like stay readable).

RetryPolicy dataclass

Retry settings. Durations are seconds (datetime.timedelta is accepted).

Backoff for retry n (0-based) is full jitter: random(0, min(max_backoff, initial_backoff * multiplier ** n)).

random returns a float in [0, 1); inject it in tests for deterministic sleeps.

max_attempts property

max_attempts: int

Total attempts including the first one.

backoff

backoff(retry: int) -> float

Full-jitter sleep in seconds before retry retry (0-based).

backoff_cap

backoff_cap(retry: int) -> float

Upper bound of the jittered sleep before retry retry (0-based).

Secret

Wraps a credential so it never shows up in str/repr/logs.

s = Secret("hunter2") str(s), s.reveal() ('[REDACTED]', 'hunter2')

reveal

reveal() -> str

Return the wrapped value.

ServiceStats dataclass

Spans and errors of one service in a search hit.

Span dataclass

A finished span as returned by a query API.

Ids are normalized on construction (an all-zero or empty parent_span_id becomes None); a negative duration_micros (the default) is computed from start and end. A subclass with normalize_ids = False keeps native ids verbatim (an empty parent_span_id still becomes None); link_class is used by :meth:from_json.

from_json classmethod

from_json(data: Mapping[str, Any]) -> Span

Parse the wire form; ValidationError for malformed input.

to_json

to_json() -> dict[str, Any]

Normalized wire form (camelCase, schema Span).

SpanBatchOptions dataclass

BatchSpanProcessor settings; durations are float seconds (timedelta accepted).

A link to another span. Set normalize_ids = False on a subclass for native ids.

SpanNode dataclass

A span with its children (sorted like :func:sort_spans).

find

find(name: str) -> SpanNode | None

Depth-first search for a node by span name.

SpanStatus dataclass

from_json classmethod

from_json(data: Any) -> SpanStatus

Parse {"code", "message"?} (a bare code string is accepted too).

TagValue dataclass

One tag value. value is always text; type tells how to read it (string, int, keyword …).

TempoClient

Synchronous Grafana Tempo client (query API, port 3200). Thread-safe; use as a context manager or call :meth:close.

client.query gives the query API (get_trace, search, search_tags, search_tag_values, traceql_metrics and the SPEC TraceQueryClient methods). Every I/O method accepts cancel= (a :class:CancelToken) to abort from another thread.

with TempoClient.from_env() as client: # doctest: +SKIP ... trace = client.query.get_trace("5b8efff798038103d269b633813fc60c")

query property

query: TempoQueryClient

The query API sharing this client's transport and tenant.

resources property

resources: ResourceRegistry

Background resources stopped by :meth:close.

build_info

build_info(*, cancel: CancelToken | None = None) -> BuildInfo

GET /api/status/buildinfo (see :meth:TempoQueryClient.build_info).

close

close() -> None

Stop background resources and close connections (idempotent).

echo

echo(*, cancel: CancelToken | None = None) -> str

GET /api/echo → "echo".

from_env classmethod

from_env(env: Mapping[str, str] | None = None, **overrides: Any) -> TempoClient

Build a client from GROW_GRAFANA_TEMPO_* (then TEMPO_URL, TEMPO_TENANT_ID); keyword overrides win.

health

health(*, cancel: CancelToken | None = None) -> HealthStatus

Readiness: GET /ready (no tenant needed). ready → healthy ok; HTTP 503 while Tempo starts → unhealthy with details["reason"]; network errors → unreachable. Never raises for an unhealthy or unreachable service.

ping

ping(*, cancel: CancelToken | None = None) -> float

Tenant-aware round trip GET /api/v2/search/tags?scope=intrinsic; returns the latency in milliseconds. A multi-tenant Tempo without a tenant → :class:AuthenticationError.

ready

ready(*, cancel: CancelToken | None = None) -> bool

GET /ready → True when ready, False while starting; raises on network errors.

request

request(req: Request, *, cancel: CancelToken | None = None) -> Response

Send a raw request through the client's transport (retries, hooks, redaction).

server_version

server_version(*, cancel: CancelToken | None = None) -> str

Server version ("3.1.1") from /api/status/buildinfo, falling back to the text of /status/version; :class:UnsupportedError when neither reports one.

TempoConfig dataclass

Bases: BaseConfig

Immutable settings for :class:~grow_grafana_tempo.TempoClient (query API, port 3200).

Common fields (url, timeouts, retry, tls, proxy, headers, user_agent_suffix, hooks, logger) come from :class:BaseConfig. Durations are float seconds; datetime.timedelta is accepted.

Tempo has no authentication of its own. token (sent as Authorization: Bearer) and headers exist for an authenticating reverse proxy in front of Tempo; both are redacted in repr(), logs and error messages.

Environment (from_env), precedence explicit > GROW_GRAFANA_TEMPO_<NAME> > vendor > default: URL (vendor TEMPO_URL, GROW_TEMPO_URL), TENANT_ID (TEMPO_TENANT_ID, GROW_TEMPO_TENANT_ID), TOKEN (GROW_TEMPO_TOKEN), CONNECT_TIMEOUT, REQUEST_TIMEOUT, TOTAL_TIMEOUT, MAX_RETRIES, CA_CERT, CLIENT_CERT, CLIENT_KEY, INSECURE_SKIP_VERIFY, PROXY (vendor aliases GROW_TEMPO_<NAME>).

request_headers

request_headers() -> dict[str, str]

Headers added to every query call: the tenant and, when set, the bearer token.

tenant_headers

tenant_headers() -> dict[str, str]

{"X-Scope-OrgID": tenant} when a tenant is set, else {}.

validate

validate() -> _C

Common checks plus the token and tenant rules; raises :class:ConfigError.

TempoQueryClient

Bases: BaseTraceQueryClient

Synchronous client for Tempo's HTTP query API (port 3200).

Get one from client.query, or build one from a config (it then owns its client and :meth:close closes it). The tenant (X-Scope-OrgID) comes from the config. Every method accepts cancel= (a :class:CancelToken).

client property

client: TempoClient

The underlying :class:TempoClient.

build_info

build_info(*, cancel: CancelToken | None = None) -> BuildInfo

GET /api/status/buildinfo → version, revision, branch, build user/date, Go.

close

close() -> None

Close the underlying client when this query client owns it (idempotent).

echo

echo(*, cancel: CancelToken | None = None) -> str

GET /api/echo → "echo" (no tenant needed).

from_env classmethod

from_env(env: Mapping[str, str] | None = None, **overrides: Any) -> TempoQueryClient

An owned query client configured from GROW_GRAFANA_TEMPO_* / TEMPO_*.

get_dependencies

get_dependencies(*, start: datetime | None = None, end: datetime | None = None) -> list[DependencyLink]

Always :class:UnsupportedError: Tempo has no dependency API.

get_trace

get_trace(trace_id: str, *, start: _TimeLike | None = None, end: _TimeLike | None = None, cancel: CancelToken | None = None) -> Trace | None

Fetch a trace by id: GET /api/v2/traces/{id} (v1 /api/traces/{id} when the v2 route is missing). None when the trace does not exist or belongs to another tenant. start/end narrow the backend blocks searched (recent data is returned anyway). :class:ValidationError for an invalid id.

list_operations

list_operations(service: str, *, start: _TimeLike | None = None, end: _TimeLike | None = None, cancel: CancelToken | None = None) -> list[str]

Span names of one service (name values filtered by the service), sorted.

list_services

list_services(*, start: _TimeLike | None = None, end: _TimeLike | None = None, cancel: CancelToken | None = None) -> list[str]

Service names (resource.service.name values) seen recently or in the window, sorted.

ready

ready(*, cancel: CancelToken | None = None) -> bool

GET /ready: True when Tempo answers ready, False while it starts (HTTP 503) or reports another state. Raises on network errors (unlike health()).

search

search(traceql: str | TraceQL | None = None, *, start: _TimeLike | None = None, end: _TimeLike | None = None, limit: int | None = None, spss: int | None = None, cancel: CancelToken | None = None) -> TempoSearchResult

TraceQL search: GET /api/search?q=…&start&end&limit&spss (no q without a query). Invalid TraceQL → :class:ValidationError (HTTP 400).

search_tag_values

search_tag_values(tag: str, *, query: str | TraceQL | None = None, start: _TimeLike | None = None, end: _TimeLike | None = None, cancel: CancelToken | None = None) -> list[TagValue]

Values of one tag: GET /api/v2/search/tag/{tag}/values. tag is a TraceQL identifier (resource.service.name, span.http.response.status_code, name, status, span.grow weird.key); an unscoped name is a :class:ValidationError.

search_tags

search_tags(*, scope: TagScope | None = None, query: str | TraceQL | None = None, start: _TimeLike | None = None, end: _TimeLike | None = None, cancel: CancelToken | None = None) -> dict[str, list[str]]

Tag names per scope: GET /api/v2/search/tags → {"resource": [...], …}. Without scope every scope that has data; query keeps tags of matching spans.

search_traces

search_traces(query: TraceSearchQuery | None = None, *, cancel: CancelToken | None = None) -> list[TraceSummary]

SPEC §5 search mapped onto one TraceQL spanset (see :func:search_traces_traceql); durations have millisecond precision.

traceql_metrics

traceql_metrics(query: str | TraceQL, *, start: _TimeLike | None = None, end: _TimeLike | None = None, since_s: float | timedelta | None = None, step_s: float | timedelta | None = None, exemplars: int | None = None, cancel: CancelToken | None = None) -> TraceQLMetricsResult

TraceQL metrics over a range: GET /api/metrics/query_range (e.g. { resource.service.name = "api" } | rate()). since_s is a window ending now instead of start/end. Zero samples omitted by Tempo read as 0. A query without a metrics pipeline → :class:ValidationError.

TempoSearchResult dataclass

Result of search: traces plus query statistics (inspectedBytes …) as numbers.

TempoSearchSpan dataclass

One span of a search spanset. name is set only when the query selects it.

TempoSearchTrace dataclass

One trace found by search. trace_id is padded back to 32 hex characters; duration_micros has millisecond precision (Tempo reports whole milliseconds).

TempoSpanSet dataclass

Spans of one trace that matched; matched may exceed len(spans) (see spss).

TimeoutError

Bases: TempoError

A connect, request or total deadline was exceeded (also HTTP 408, 504).

TlsConfig dataclass

TLS settings. insecure_skip_verify is for tests only and logs a warning once.

Trace dataclass

All spans of one trace, sorted by start time then span id.

A subclass with normalize_ids = False keeps a native trace id verbatim; span_class is used by :meth:from_json.

TraceQL

Immutable TraceQL builder: span-set filters, structural operators and select().

Field names are validated (unknown scopes or intrinsics raise :class:ValidationError, because some Tempo endpoints silently ignore an invalid query); attribute names with spaces or special characters are quoted after the scope (span."grow weird.key"); the bare intrinsic duration is written as span:duration. Every method returns a new builder. str(query) renders the TraceQL text.

ancestor

ancestor(other: TraceQL) -> TraceQL

self << other: spans of other above self at any depth.

and_

and_(field: str, op: TraceQLOperator | str, value: TraceQLValue) -> TraceQL

Add && <field> <op> <value> to the last span-set filter.

child

child(other: TraceQL) -> TraceQL

self > other: spans of other that are direct children.

descendant

descendant(other: TraceQL) -> TraceQL

self >> other: spans of other below self at any depth.

field staticmethod

field(name: str) -> str

Render a field name: validate the scope or intrinsic and quote odd attribute names.

or_

or_(field: str, op: TraceQLOperator | str, value: TraceQLValue) -> TraceQL

Add || <field> <op> <value> to the last span-set filter.

parent

parent(other: TraceQL) -> TraceQL

self < other: spans of other that are direct parents.

quote staticmethod

quote(text: str) -> str

Quote text as a TraceQL string literal (\, " and control characters escaped; Unicode passes through).

select

select(*fields: str) -> TraceQL

Add fields to | select(…) so search returns them on every matched span.

sibling

sibling(other: TraceQL) -> TraceQL

self ~ other: spans of other with the same parent.

span classmethod

span(filters: Mapping[str, TraceQLValue] | None = None) -> TraceQL

Start a span-set filter; each filters entry becomes <field> = <value> joined with && in insertion order. TraceQL.span() alone matches every span ({}).

structural

structural(op: str, other: TraceQL) -> TraceQL

Join other's span-set filters with a span-set operator: > (child), >> (descendant), < (parent), << (ancestor), ~ (sibling), && / || (both / either). other's select() is ignored.

value staticmethod

value(field: str, value: TraceQLValue) -> str

Render value for field (a name already rendered by :meth:field).

where classmethod

where(field: str, op: TraceQLOperator | str, value: TraceQLValue) -> TraceQL

Start a span-set filter with one condition: { <field> <op> <value> }.

TraceQLMetricsResult dataclass

Result of traceql_metrics; step_ms is the step Tempo used, when reported.

TraceQueryClient

Bases: Protocol

Synchronous trace query API.

TraceSearchQuery dataclass

matches

matches(trace: Trace) -> bool

Client-side filter: a span must match service/operation/tags; duration and time bounds apply to the whole trace.

TraceSummary dataclass

Search result row; normalize_ids = False on a subclass keeps native ids.

from_trace classmethod

from_trace(trace: Trace) -> TraceSummary

Summary of a trace (root = earliest span without a known parent).

to_json

to_json() -> dict[str, Any]

Wire form (traceSummary in trace.schema.json).

TracingASGIMiddleware

ASGI middleware: one SERVER span per HTTP request, parent taken from request headers.

Non-HTTP scopes (lifespan, websocket) and excluded_paths pass through untraced.

TracingConfig dataclass

Settings for :func:init_tracing_with. repr redacts credential headers.

protocol and endpoint are interpreted by the service's exporter factory; the template default protocol is http/protobuf. propagators beyond tracecontext, baggage, b3 and b3multi need a factory from the service package.

protocol and propagators remember whether the caller set them: left out (or passed as None) they read as :data:DEFAULT_TRACE_PROTOCOL / :data:DEFAULT_TRACE_PROPAGATORS exactly as before, and :attr:protocol_set / :attr:propagators_set are False so a service default can apply instead. The marker travels with the value, so :meth:with_changes, dataclasses.replace and TracingConfig(**fields) keep an unset field unset; passing the default value explicitly counts as set.

propagators_set property

propagators_set: bool

Whether propagators was given explicitly (False: the default applies).

protocol_set property

protocol_set: bool

Whether protocol was given explicitly (False: the default applies).

validate

validate() -> None

Raise ConfigError for invalid settings (never echoing values).

with_changes

with_changes(**changes: Any) -> TracingConfig

Copy with some fields replaced (unset protocol/propagators stay unset).

TracingHandle

Result of :func:init_tracing_with: provider, propagator, flush and shutdown.

force_flush

force_flush(timeout_s: float | timedelta | None = None) -> None

Export every finished span now; TimeoutError when it does not finish in time.

shutdown

shutdown() -> None

Flush and stop the provider and its exporter; idempotent.

tracer

tracer(name: str, version: str | None = None) -> otel_trace.Tracer

A tracer from this handle's provider (works with set_global=False too).

TracingWSGIMiddleware

WSGI middleware: one SERVER span per request; ends when the response is closed.

UnavailableError

Bases: TempoError

Network error or server unavailable (HTTP 500, 502, 503).

UnsupportedError

Bases: TempoError

The service or deployment does not offer this capability (HTTP 501, missing API).

ValidationError

Bases: TempoError

Bad input (HTTP 400, 413, 422, other 4xx, client-side validation).

add_event

add_event(name: str, attributes: Attributes = None) -> None

Add an event to the active span (no-op without one).

apoll_until async

apoll_until(fn: Callable[[], Awaitable[T | None]], *, timeout_s: float = 30.0, interval_s: float = 1.0, predicate: Callable[[T], bool] | None = None, operation: str = 'poll', clock: Callable[[], float] = time.monotonic) -> T

Async :func:poll_until; fn is an async callable.

async_httpx_event_hooks

async_httpx_event_hooks(*, tracer: Tracer | None = None, propagator: TextMapPropagator | None = None) -> dict[str, list[Callable[..., Any]]]

event_hooks for httpx.AsyncClient (same behaviour as :func:httpx_event_hooks).

async_wait_for_trace async

async_wait_for_trace(client: AsyncTraceQueryClient, trace_id: str, *, timeout_s: float | timedelta = 30.0, interval_s: float | timedelta = 1.0, min_spans: int = 1, normalize: Callable[[str], str] | None = normalize_trace_id) -> Trace

Async form of :func:wait_for_trace.

build_propagator

build_propagator(names: Sequence[str], factories: Mapping[str, PropagatorFactory] | None = None) -> TextMapPropagator

Composite propagator for names; factories adds or overrides propagators by name.

build_resource

build_resource(config: TracingConfig) -> Resource

Resource with service.name/service.version/deployment.environment and extras.

The environment is written both as deployment.environment.name (current semantic conventions) and as the older deployment.environment. Merged on top of the SDK default resource (telemetry.sdk.*, OTEL_RESOURCE_ATTRIBUTES); explicit config values win.

build_sampler

build_sampler(name: str, ratio: float = 1.0) -> Sampler

OpenTelemetry sampler for a TracingConfig.sampler name.

build_tree

build_tree(spans: Iterable[Span]) -> list[SpanNode]

Parent/child forest; spans whose parent is missing become roots.

check_traceql

check_traceql(query: str, operation: str) -> str

Light client-side check of a TraceQL text; returns it trimmed.

The query must start with { and quotes, braces and parentheses must balance. Tempo's tag endpoints ignore an invalid q and return unfiltered values, so the SDK refuses obviously broken queries there (:class:ValidationError).

create_tempo_exporter

create_tempo_exporter(config: TracingConfig) -> SpanExporter

The official OpenTelemetry OTLP exporter for a resolved config.

grpc → OTLP/gRPC (x-scope-orgid metadata; the grpc extra); http/protobuf and native → OTLP/HTTP protobuf to <endpoint>/v1/traces. Applies headers, TLS, compression and the export timeout.

current_span

current_span() -> otel_trace.Span

The active span (a non-recording invalid span when none is active).

current_span_id

current_span_id() -> str | None

16-char lower-hex span id of the active span, or None.

current_trace_id

current_trace_id() -> str | None

32-char lower-hex trace id of the active span, or None.

datetime_to_micros

datetime_to_micros(value: datetime) -> int

Microseconds since the epoch (naive datetimes are UTC).

default_tracer

default_tracer() -> otel_trace.Tracer

Tracer of this package from the global provider.

dependencies_from_spans

dependencies_from_spans(spans: Iterable[Span]) -> list[DependencyLink]

Service dependency links (parent service → child service, different services only).

error_for_code

error_for_code(code: ErrorCode | str) -> type[TempoError]

Return the error class for a code (ErrorCode member or wire value).

error_from_status

error_from_status(status: int, operation: str, *, content: bytes = b'', headers: Headers | None = None, known_secrets: list[str] | None = None, retry_after: float | None = None, request_id_headers: Sequence[str] = ()) -> ServiceError

Map an HTTP error status to the SPEC 2.1 error with a redacted, truncated message.

request_id_headers are checked before the default :data:REQUEST_ID_HEADERS.

extract

extract(carrier: Mapping[str, str], *, context: Context | None = None, propagator: TextMapPropagator | None = None) -> otel_context.Context

Read a context from carrier (exact key match; use :func:extract_headers for HTTP).

extract_headers

extract_headers(headers: HeaderSource, *, context: Context | None = None, propagator: TextMapPropagator | None = None) -> otel_context.Context

Read a context from HTTP headers (case-insensitive; mapping or ASGI-style byte pairs).

from_any_value

from_any_value(value: Any) -> Any

Convert an OTLP AnyValue to a plain value (None for an empty value).

intValue strings become int, doubleValue float, arrays lists, key/value lists dicts; bytesValue stays base64 text.

from_key_values

from_key_values(items: Iterable[Any] | None) -> dict[str, Any]

Convert an OTLP KeyValue list to a dict (entries with empty values are dropped).

get_logger

get_logger() -> logging.Logger

The package logger (LOGGER_NAME), silent by default (NullHandler).

go_duration

go_duration(value: float | timedelta, name: str) -> str

Format a positive duration (seconds or timedelta) as a Go duration (5s, 1500ms); :class:ValidationError otherwise.

httpx_event_hooks

httpx_event_hooks(*, tracer: Tracer | None = None, propagator: TextMapPropagator | None = None) -> dict[str, list[Callable[..., Any]]]

event_hooks for httpx.Client: a CLIENT span per request with injected headers.

A request that fails before a response arrives (connection error) never reaches the response hook, so its span is not exported; use :func:instrument_requests_session style wrapping or with_span around the call when such failures must be traced.

init_tracing

init_tracing(config: TracingConfig | None = None, *, tenant_id: str | None = None, env: Mapping[str, str] | None = None, propagator_factories: Mapping[str, PropagatorFactory] | None = None, span_processors: Sequence[SpanProcessor] = (), id_generator: IdGenerator | None = None, **fields: Any) -> TracingHandle

Set up OpenTelemetry tracing that exports to Grafana Tempo over OTLP.

Resource, sampler, batch span processor, propagators and (unless set_global=False) global registration come from the template :func:init_tracing_with; settings are resolved by :func:tempo_tracing_config (explicit > GROW_GRAFANA_TEMPO_OTLP_* > OTEL_* > default). Call shutdown() before the process exits.

A multi-tenant Tempo answers an export without a tenant with HTTP 503 / gRPC Unavailable, which the exporter retries until it drops the spans: set tenant_id whenever Tempo runs with multi-tenancy.

handle = init_tracing(service_name="checkout", tenant_id="team-a") # doctest: +SKIP with_span("work", lambda span: do_work()) # doctest: +SKIP handle.shutdown() # doctest: +SKIP

init_tracing_with

init_tracing_with(config: TracingConfig, exporter_factory: ExporterFactory, *, propagator_factories: Mapping[str, PropagatorFactory] | None = None, span_processors: Sequence[SpanProcessor] = (), id_generator: IdGenerator | None = None, sampler: Sampler | None = None) -> TracingHandle

Build a TracerProvider for config with a BatchSpanProcessor around the exporter.

Services wrap this in their init_tracing(config). propagator_factories lets a service add datadog/sw8 style propagators; span_processors (extra processors, e.g. a second exporter) are added after the batch processor; sampler replaces the one built from config.sampler/sampler_ratio (remote or vendor samplers). With config.set_global the provider and propagator become the OpenTelemetry globals (the API allows one global provider per process: is_global on the handle tells whether it took effect).

inject

inject(carrier: MutableMapping[str, str] | None = None, *, context: Context | None = None, propagator: TextMapPropagator | None = None) -> MutableMapping[str, str]

Write the current (or given) context into carrier and return it.

Uses propagator or the global propagator (set by init_tracing_with).

inject_headers

inject_headers(headers: MutableMapping[str, str] | None = None, *, context: Context | None = None, propagator: TextMapPropagator | None = None) -> MutableMapping[str, str]

Add propagation headers (traceparent, baggage …) to headers and return them.

instrument_requests_session

instrument_requests_session(session: Session, *, tracer: Tracer | None = None, propagator: TextMapPropagator | None = None) -> requests.Session

Trace every request sent through session (CLIENT span, headers injected).

Wraps session.send once; calling it again on the same session is a no-op.

map_tempo_error

map_tempo_error(err: ServiceError, operation: str, unsupported_404: bool = False) -> ServiceError

Tempo-specific mapping on top of the SPEC status mapping.

A 401 no org id (multi-tenant Tempo without X-Scope-OrgID) becomes an :class:AuthenticationError that names the missing tenant header; with unsupported_404 a 404 on a feature route becomes :class:UnsupportedError.

mark_span_error

mark_span_error(span: Span, error: BaseException | str) -> None

Set ERROR status on span and record the exception (when error is one).

micros_to_datetime

micros_to_datetime(micros: int) -> datetime

Exact UTC datetime for microseconds since the epoch.

nanos_to_micros

nanos_to_micros(value: Any) -> int

Integer microseconds from nanoseconds given as a decimal string or number (0 if unset).

new_idempotency_key

new_idempotency_key() -> str

A random key for the Idempotency-Key header.

normalize_headers

normalize_headers(headers: HeaderSource) -> dict[str, list[str]]

Lower-cased header names → values; accepts mappings and (name, value) pairs incl. bytes.

normalize_span_id

normalize_span_id(value: str | int | bytes) -> str

16-char lower-hex span id (left-padded); raises ValidationError when invalid.

normalize_trace_id

normalize_trace_id(value: str | int | bytes) -> str

32-char lower-hex trace id; 64-bit ids (16 hex chars, ints, 8 bytes) are left-padded.

Raises ValidationError for non-hex, too long or all-zero ids.

otlp_http_traces_url

otlp_http_traces_url(endpoint: str) -> str

The OTLP/HTTP traces URL: a bare scheme://host:port gets /v1/traces appended.

otlp_id_to_hex

otlp_id_to_hex(value: Any, hex_length: int) -> str | None

Convert an id from trace-by-id JSON (base64 bytes) to lower-case hex of hex_length.

Ids that already are hex of that length are kept. Returns None for a missing, empty, undecodable or all-zero id.

parse_bool

parse_bool(text: str) -> bool

Parse true/false/1/0/yes/no/on/off (case-insensitive); raise :class:ValueError.

parse_duration

parse_duration(text: str) -> float

Parse a duration into seconds.

Accepts integer milliseconds ("1500" → 1.5) or Go-style units ("500ms", "10s", "2m", "1h", "1m30s"). Raises :class:ValueError for anything else.

parse_key_value_list

parse_key_value_list(text: str, variable: str) -> dict[str, str]

Parse key=value,key2=value2 (values percent-decoded) as in OTEL_EXPORTER_OTLP_HEADERS; :class:ConfigError naming variable when malformed.

parse_retry_after

parse_retry_after(value: str | None, *, now: Callable[[], float] = time.time) -> float | None

Parse Retry-After as delta seconds or an HTTP-date; return seconds (>= 0) or None.

poll_until

poll_until(fn: Callable[[], T | None], *, timeout_s: float = 30.0, interval_s: float = 1.0, predicate: Callable[[T], bool] | None = None, operation: str = 'poll', sleep: Callable[[float], None] = time.sleep, clock: Callable[[], float] = time.monotonic) -> T

Call fn every interval_s until it returns a ready value; return that value.

A value is ready when it is not None and predicate(value) is true (without a predicate: not an empty container). Retryable SDK errors raised by fn are remembered and polling continues; other errors propagate. Raises :class:TimeoutError after timeout_s.

record_exception

record_exception(error: BaseException, attributes: Attributes = None) -> None

Record error as an exception event on the active span (status unchanged).

redact_fields

redact_fields(value: Any, extra_keys: Iterable[str] = ()) -> Any

Recursively redact values whose key looks sensitive in nested mappings and lists.

redact_headers

redact_headers(headers: Mapping[str, str] | Iterable[tuple[str, str]]) -> dict[str, str]

Copy of headers with sensitive values replaced by [REDACTED].

redact_text

redact_text(text: str, known_secrets: Iterable[str | Secret | None] = ()) -> str

Replace every known secret value (length >= 4) in text with [REDACTED].

redact_url

redact_url(url: str) -> str

Redact user info and every query value (all values, not only sensitive ones).

Query names are kept so the URL stays useful in logs.

retry_after_from_headers

retry_after_from_headers(headers: Headers | Mapping[str, str], names: Sequence[str] = RETRY_AFTER_HEADERS, *, now: Callable[[], float] = time.time) -> float | None

Server requested delay in seconds from the first of names that parses.

Retry-After is delta seconds or an HTTP-date. Other headers (X-RateLimit-Reset, RateLimit-Reset …) may be delta seconds or a Unix timestamp in seconds or milliseconds; timestamps are turned into the remaining delay.

search_traces_traceql

search_traces_traceql(query: TraceSearchQuery) -> TraceQL

The TraceQL search_traces sends: one spanset with resource.service.name, name, unscoped .tag filters and trace:duration bounds in microseconds.

set_error

set_error(error: BaseException | str) -> None

Mark the active span as failed: ERROR status, plus an exception event for exceptions.

sort_spans

sort_spans(spans: Iterable[Span]) -> list[Span]

Spans sorted by start time, then span id.

span_from_readable

span_from_readable(span: ReadableSpan) -> Span

Convert a finished OpenTelemetry SDK span (e.g. from an in-memory exporter).

span_kind

span_kind(kind: SpanKindLike) -> SpanKind

SpanKind from an enum member or a name ("server"); None → INTERNAL.

spans_from_resource_spans

spans_from_resource_spans(resource_spans: Iterable[Any] | None) -> list[Span]

Flatten OTLP resourceSpans → scopeSpans → spans into model spans.

Ids base64 → hex, nanoseconds → microseconds, enums → names, service_name from the resource service.name ("unknown_service" when absent). Spans with unusable ids are skipped. Sorted by start time, then span id.

tempo_tracing_config

tempo_tracing_config(config: TracingConfig | None = None, *, tenant_id: str | None = None, env: Mapping[str, str] | None = None, **fields: Any) -> TracingConfig

Resolve the Tempo tracing settings from explicit values and the environment.

Explicit values are config (a :class:TracingConfig, whose protocol counts as explicit) and/or keyword fields of :class:TracingConfig (they win over config). Unset values come from the environment, per field:

  • endpoint: GROW_GRAFANA_TEMPO_OTLP_ENDPOINT > OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (verbatim) > OTEL_EXPORTER_OTLP_ENDPOINT (+ /v1/traces for HTTP) > default (http://127.0.0.1:4318, or :4317 for grpc);
  • protocol: GROW_GRAFANA_TEMPO_OTLP_PROTOCOL > OTEL_EXPORTER_OTLP_TRACES_PROTOCOL

    OTEL_EXPORTER_OTLP_PROTOCOL > http/protobuf;

  • headers: OTEL_EXPORTER_OTLP_TRACES_HEADERS > OTEL_EXPORTER_OTLP_HEADERS, under the explicit headers;
  • service_name: OTEL_SERVICE_NAME > service.name in OTEL_RESOURCE_ATTRIBUTES;
  • compression: GROW_GRAFANA_TEMPO_OTLP_COMPRESSION > OTEL_EXPORTER_OTLP_TRACES_COMPRESSION > OTEL_EXPORTER_OTLP_COMPRESSION (gzip or none; applies while compression is the default gzip);
  • sampler / sampler_ratio: OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG (while they hold the defaults parentbased_always_on / 1.0);
  • tenant: tenant_id > GROW_GRAFANA_TEMPO_TENANT_ID > TEMPO_TENANT_ID > GROW_TEMPO_TENANT_ID; it becomes the X-Scope-OrgID header.

env defaults to os.environ. Raises :class:ConfigError naming an invalid variable (never its value). The result is not validated (init_tracing does that).

to_seconds

to_seconds(value: float | timedelta) -> float

Normalise a duration given as float seconds or :class:datetime.timedelta.

trace_fields

trace_fields() -> dict[str, str]

{"trace_id": …, "span_id": …} of the active span for log fields; {} without one.

traced

traced(fn: Callable[P, R]) -> Callable[P, R]
traced(name: str | None = None, /, *, attributes: Attributes = None, kind: SpanKindLike = None, tracer: Tracer | None = None) -> Callable[[Callable[P, R]], Callable[P, R]]
traced(name_or_fn: str | Callable[P, R] | None = None, /, *, attributes: Attributes = None, kind: SpanKindLike = None, tracer: Tracer | None = None) -> Callable[P, R] | Callable[[Callable[P, R]], Callable[P, R]]

Decorator that runs a function (sync or async def) inside a span.

@traced, @traced() and @traced("name") all work; the default span name is the function's qualified name. Async generators are not supported (the span would end before iteration).

traceql_text

traceql_text(query: str | TraceQL) -> str

Render a query argument (text or :class:TraceQL) to text.

truncate

truncate(text: str, limit: int = 300) -> str

Truncate text to limit characters (adds an ellipsis when cut).

user_agent

user_agent(suffix: str | None = None) -> str

grow-codes-<key>/<version> (python) plus <suffix> when given.

validate_tenant_id

validate_tenant_id(tenant_id: str | None, what: str = 'tenant_id') -> None

Raise :class:ConfigError when tenant_id breaks Tempo's tenant id rules.

Each tenant is 1-150 characters of letters, digits and !-_.*'(); | joins several tenants for a federated query ("tenant-b|tenant-a"). None means "no tenant". The message never echoes the value.

wait_for_trace

wait_for_trace(client: TraceQueryClient, trace_id: str, *, timeout_s: float | timedelta = 30.0, interval_s: float | timedelta = 1.0, min_spans: int = 1, normalize: Callable[[str], str] | None = normalize_trace_id) -> Trace

Poll get_trace until the trace has min_spans spans; TimeoutError otherwise.

NotFoundError and retryable errors count as "not there yet". normalize turns trace_id into the id passed to get_trace (default: 32-char lower hex); None passes native ids verbatim.

with_span

with_span(name: str, fn: None = None, *, attributes: Attributes = None, kind: SpanKindLike = None, links: Sequence[Link | SpanContext] | None = None, tracer: Tracer | None = None) -> AbstractContextManager[otel_trace.Span]
with_span(name: str, fn: Callable[[Span], R], *, attributes: Attributes = None, kind: SpanKindLike = None, links: Sequence[Link | SpanContext] | None = None, tracer: Tracer | None = None) -> R
with_span(name: str, fn: Callable[[Span], R] | None = None, *, attributes: Attributes = None, kind: SpanKindLike = None, links: Sequence[Link | SpanContext] | None = None, tracer: Tracer | None = None) -> R | AbstractContextManager[otel_trace.Span]

Run fn(span) inside a new current span, or return a context manager without fn.

The span always ends. An exception is recorded, sets ERROR status and is re-raised (cancellation, asyncio.CancelledError or the SDK CancelledError, is not). The context-manager form also works inside async code (context is kept in contextvars).