Troubleshooting¶
The client cannot connect¶
health()returnshealthy: false, status: "unreachable": the URL is wrong or the service is down. CheckGROW_GRAFANA_LOKI_URL(defaulthttp://127.0.0.1:3100) and that the port is reachable from where your code runs (inside Docker,127.0.0.1is the container itself).- Calls fail with
UNAVAILABLEafter several seconds: the client retried with backoff. LowermaxRetriesortotalTimeoutto fail faster. - Behind a corporate proxy:
HTTP_PROXY/HTTPS_PROXYare ignored on purpose. Setproxy(orGROW_GRAFANA_LOKI_PROXY).
CONFIG errors¶
The message names the setting or environment variable, never its value. Typical causes: a URL
without http:// or https://, a duration such as 10 sec (use 10s or 10000), a client
certificate without its key, MAX_RETRIES below 0.
AUTHENTICATION or PERMISSION_DENIED¶
ping()is the quickest credential check: it makes one authenticated call.401 no org id: the Loki server runs withauth_enabled: trueand needs a tenant. SettenantId(GROW_GRAFANA_LOKI_TENANT_IDorLOKI_TENANT_ID) or useclient.withTenant(id).- Behind a gateway or on Grafana Cloud, check the credentials the SDK reads:
GROW_GRAFANA_LOKI_USERNAME/GROW_GRAFANA_LOKI_PASSWORD(orLOKI_USERNAME/LOKI_PASSWORD,LOKI_BASIC_AUTH_USER/LOKI_BASIC_AUTH_PASSWORD) orGROW_GRAFANA_LOKI_BEARER_TOKEN(LOKI_BEARER_TOKEN); explicit config values win over the environment (Configuration). - The SDK never prints credentials, so
toString()of the config shows[REDACTED]even when a password or token is set.
TLS errors¶
- Self-signed or private CA: set
tls.caCertPath(GROW_GRAFANA_LOKI_CA_CERT). - Mutual TLS: set both
tls.clientCertPathandtls.clientKeyPath. tls.insecureSkipVerifyonly for local tests.
RATE_LIMITED¶
The service asked the client to slow down. The client waits for Retry-After up to
maxRetryAfter (60 s); longer waits fail at once with retryAfter set on the error so your code
can schedule a later attempt.
TIMEOUT¶
requestTimeout (30 s) bounds one attempt and totalTimeout (120 s) the whole call including
retries. Raise them for slow operations, or pass a per-call deadline (signal, ctx, cancel token,
future timeout).
See what the client does¶
Turn on the logger or add hooks: both see every attempt with redacted headers and URLs.
Docker integration tests fail¶
- Run one language:
make test-docker LANGS=go. - In CI, download the
compose-logs-<lang>artifact of the failed job. - Locally, the stack is removed after each run; rerun with Docker logs open
(
docker compose -f docker/compose.yml logs -f) in a second terminal.
Still stuck?¶
Open an issue with the language, package version, Grafana Loki version and a minimal reproduction (with secrets removed): https://github.com/grow-codes-packages/grafana-loki/issues.
Loki-specific problems¶
- Every push fails with
UNAVAILABLE"Ingester is shutting down" although reads work: the filesystem that holds Loki's WAL is at least 90 % full (-ingester.wal-disk-full-threshold, default 0.9) and Loki throttles all writes. The SDK retries it as a transient error; free disk space on the Loki volume. pushsucceeds but a selector finds nothing: the cardinality guard moved the label to structured metadata. Filter it with{service_name="x"} | run="r1"or add the name toallowedLabels.400 … timestamp too old/too new: Loki rejects entries older thanreject_old_samples_max_age(7 days in the Grow Codes config) or more than about 10 minutes in the future. The whole push is rejected.query()rejects a selector withVALIDATION: log selectors are not instant queries in Loki; usequeryRangeorsearch.volume/volumeRangereturn no data: the server needslimits_config.volume_enabled: true(the image default config has it off).indexStatsandvolumeRangeread the TSDB index, which sees data only after the ingester flushed its chunks.- A tail sees old lines: pass no
start(the default is "live only") or a recent one; with an explicitstartLoki replays from there (up tolimit).