Skip to content

Troubleshooting

The client cannot connect

  • health() returns healthy: false, status: "unreachable": the endpoint cannot be reached. Without url the AWS SDK calls https://logs.<region>.amazonaws.com: check the region (GROW_AWS_CLOUDWATCH_LOGS_REGION, AWS_REGION), outbound HTTPS and, behind a proxy, proxy. With an endpoint override (GROW_AWS_CLOUDWATCH_LOGS_URL, AWS_ENDPOINT_URL_CLOUDWATCH_LOGS, AWS_ENDPOINT_URL, for example Moto on http://127.0.0.1:5000 or a VPC endpoint) check that it is reachable from where your code runs (inside Docker, 127.0.0.1 is the container itself).
  • CONFIG naming the region: no region was found; set one in code, in the environment or in the AWS profile.
  • Calls fail with UNAVAILABLE after several seconds: the client retried with backoff. Lower maxRetries or totalTimeout to fail faster.
  • Behind a corporate proxy: HTTP_PROXY/HTTPS_PROXY are ignored on purpose. Set proxy (or GROW_AWS_CLOUDWATCH_LOGS_PROXY).

CONFIG errors

The message names the setting or environment variable, never its value. Typical causes: a URL without http:// or https://, a duration such as 10 sec (use 10s or 10000), a client certificate without its key, MAX_RETRIES below 0.

AUTHENTICATION or PERMISSION_DENIED

  • ping() is the quickest credential check: it makes one authenticated call.
  • Credentials come from the AWS default chain (environment, shared config and SSO, web identity, container and instance roles) unless accessKeyId/secretAccessKey, profile, credentials or roleArn is set; explicit config values win over the environment. When GROW_AWS_CLOUDWATCH_LOGS_ACCESS_KEY_ID is set, the secret key and session token are read from GROW_AWS_CLOUDWATCH_LOGS_* only, never mixed with AWS_* (see Service settings).
  • AUTHENTICATION: the key is unknown, expired (temporary credentials need sessionToken) or the clock is skewed. PERMISSION_DENIED: the IAM policy lacks the logs: permission for the operation named in the message; with roleArn the caller also needs sts:AssumeRole.
  • The SDK never prints credentials, so toString() of the config shows [REDACTED] even when a credential is set.

TLS errors

  • Self-signed or private CA: set tls.caCertPath (GROW_AWS_CLOUDWATCH_LOGS_CA_CERT).
  • Mutual TLS: set both tls.clientCertPath and tls.clientKeyPath.
  • tls.insecureSkipVerify only for local tests.

RATE_LIMITED

The service asked the client to slow down. The client waits for Retry-After up to maxRetryAfter (60 s); longer waits fail at once with retryAfter set on the error so your code can schedule a later attempt.

TIMEOUT

requestTimeout (30 s) bounds one attempt and totalTimeout (120 s) the whole call including retries. Raise them for slow operations, or pass a per-call deadline (signal, ctx, cancel token, future timeout).

See what the client does

Turn on the logger or add hooks: both see every attempt with redacted headers and URLs.

Docker integration tests fail

  • Run one language: make test-docker LANGS=go.
  • In CI, download the compose-logs-<lang> artifact of the failed job.
  • Locally, the stack is removed after each run; rerun with Docker logs open (docker compose -f docker/compose.yml logs -f) in a second terminal.

putLogEvents reports rejected events or more batches than expected

  • One PutLogEvents call may not span more than 24 hours, so the SDK splits the timestamp-sorted events into several batches. Events older than 14 days (or older than the group's retention) are usually more than 24 hours older than the rest, so they end up in their own batch, which AWS rejects as a whole: batches is higher and rejected counts them, with rejectedInfo (tooOldLogEventEndIndex, expiredLogEventEndIndex) pointing into that batch. Events more than 2 hours in the future are rejected as too new. Rejected events are never retried.
  • See Ingest for the limits and the index semantics.

Error messages do not show the AWS error text

On purpose: AWS error messages can echo request data (log messages, group names, filter patterns), so the SDK never copies them. The message is "<operation>: <reason>[: <AwsCode>] (HTTP <status>)" (the AWS error code for every code in the error mapping), and the error carries the AWS request id, which is what AWS Support needs. The original AWS SDK error stays available as the redacted cause.

Still stuck?

Open an issue with the language, package version, Amazon CloudWatch Logs version and a minimal reproduction (with secrets removed): https://github.com/grow-codes-packages/aws-cloudwatch-logs/issues.