Troubleshooting¶
The client cannot connect¶
health()returnshealthy: false, status: "unreachable": the endpoint cannot be reached. Withouturlthe AWS SDK callshttps://logs.<region>.amazonaws.com: check the region (GROW_AWS_CLOUDWATCH_LOGS_REGION,AWS_REGION), outbound HTTPS and, behind a proxy,proxy. With an endpoint override (GROW_AWS_CLOUDWATCH_LOGS_URL,AWS_ENDPOINT_URL_CLOUDWATCH_LOGS,AWS_ENDPOINT_URL, for example Moto onhttp://127.0.0.1:5000or a VPC endpoint) check that it is reachable from where your code runs (inside Docker,127.0.0.1is the container itself).CONFIGnaming the region: no region was found; set one in code, in the environment or in the AWS profile.
- Calls fail with
UNAVAILABLEafter several seconds: the client retried with backoff. LowermaxRetriesortotalTimeoutto fail faster. - Behind a corporate proxy:
HTTP_PROXY/HTTPS_PROXYare ignored on purpose. Setproxy(orGROW_AWS_CLOUDWATCH_LOGS_PROXY).
CONFIG errors¶
The message names the setting or environment variable, never its value. Typical causes: a URL
without http:// or https://, a duration such as 10 sec (use 10s or 10000), a client
certificate without its key, MAX_RETRIES below 0.
AUTHENTICATION or PERMISSION_DENIED¶
ping()is the quickest credential check: it makes one authenticated call.
- Credentials come from the AWS default chain (environment, shared config and SSO, web identity,
container and instance roles) unless
accessKeyId/secretAccessKey,profile,credentialsorroleArnis set; explicit config values win over the environment. WhenGROW_AWS_CLOUDWATCH_LOGS_ACCESS_KEY_IDis set, the secret key and session token are read fromGROW_AWS_CLOUDWATCH_LOGS_*only, never mixed withAWS_*(see Service settings). AUTHENTICATION: the key is unknown, expired (temporary credentials needsessionToken) or the clock is skewed.PERMISSION_DENIED: the IAM policy lacks thelogs:permission for the operation named in the message; withroleArnthe caller also needssts:AssumeRole.
- The SDK never prints credentials, so
toString()of the config shows[REDACTED]even when a credential is set.
TLS errors¶
- Self-signed or private CA: set
tls.caCertPath(GROW_AWS_CLOUDWATCH_LOGS_CA_CERT). - Mutual TLS: set both
tls.clientCertPathandtls.clientKeyPath. tls.insecureSkipVerifyonly for local tests.
RATE_LIMITED¶
The service asked the client to slow down. The client waits for Retry-After up to
maxRetryAfter (60 s); longer waits fail at once with retryAfter set on the error so your code
can schedule a later attempt.
TIMEOUT¶
requestTimeout (30 s) bounds one attempt and totalTimeout (120 s) the whole call including
retries. Raise them for slow operations, or pass a per-call deadline (signal, ctx, cancel token,
future timeout).
See what the client does¶
Turn on the logger or add hooks: both see every attempt with redacted headers and URLs.
Docker integration tests fail¶
- Run one language:
make test-docker LANGS=go. - In CI, download the
compose-logs-<lang>artifact of the failed job. - Locally, the stack is removed after each run; rerun with Docker logs open
(
docker compose -f docker/compose.yml logs -f) in a second terminal.
putLogEvents reports rejected events or more batches than expected¶
- One
PutLogEventscall may not span more than 24 hours, so the SDK splits the timestamp-sorted events into several batches. Events older than 14 days (or older than the group's retention) are usually more than 24 hours older than the rest, so they end up in their own batch, which AWS rejects as a whole:batchesis higher andrejectedcounts them, withrejectedInfo(tooOldLogEventEndIndex,expiredLogEventEndIndex) pointing into that batch. Events more than 2 hours in the future are rejected as too new. Rejected events are never retried. - See Ingest for the limits and the index semantics.
Error messages do not show the AWS error text¶
On purpose: AWS error messages can echo request data (log messages, group names, filter patterns),
so the SDK never copies them. The message is "<operation>: <reason>[: <AwsCode>] (HTTP <status>)"
(the AWS error code for every code in the error mapping), and the
error carries the AWS request id, which is what AWS Support needs. The original AWS SDK error stays
available as the redacted cause.
Still stuck?¶
Open an issue with the language, package version, Amazon CloudWatch Logs version and a minimal reproduction (with secrets removed): https://github.com/grow-codes-packages/aws-cloudwatch-logs/issues.