Troubleshooting¶
The client cannot connect¶
health()returnshealthy: false, status: "unreachable": the endpoint cannot be reached. With nourlset the AWS SDK useshttps://secretsmanager.<region>.amazonaws.com— check the region and outbound HTTPS (VPC endpoints, firewalls). With an override, checkGROW_AWS_SECRETS_MANAGER_URL/AWS_ENDPOINT_URL_SECRETS_MANAGER(inside Docker,127.0.0.1is the container itself).CONFIG"no AWS region configured": setregion,GROW_AWS_SECRETS_MANAGER_REGION,AWS_REGIONor a region in the shared config profile.- Calls fail with
UNAVAILABLEafter several seconds: the AWS SDK retried with backoff. LowermaxRetriesortotalTimeoutto fail faster. - Behind a corporate proxy:
HTTP_PROXY/HTTPS_PROXYare ignored on purpose. Setproxy(orGROW_AWS_SECRETS_MANAGER_PROXY).
CONFIG errors¶
The message names the setting or environment variable, never its value. Typical causes: a URL
without http:// or https://, a duration such as 10 sec (use 10s or 10000), a client
certificate without its key, MAX_RETRIES below 0.
AUTHENTICATION or PERMISSION_DENIED¶
ping()is the quickest credential check: one signedListSecretscall.health()staysokwithdetails.auth = "failed"when only the credentials are wrong.AUTHENTICATION: unknown access key, wrong secret key, expired session token or no credentials found in the AWS chain. Check which layer is used: explicit config >GROW_AWS_SECRETS_MANAGER_ACCESS_KEY_ID(+ its own secret key and session token) >AWS_ACCESS_KEY_ID> profile / SSO / role. AGROW_*key id is never combined with anAWS_*secret key.PERMISSION_DENIED: the principal lacks the IAM action (for examplesecretsmanager:GetSecretValue,kms:Decryptfor a customer-managed key) or a resource policy denies it. WithroleArn, the base credentials also needsts:AssumeRole(and the rightexternalId).- The SDK never prints credentials, so
toString()of the config shows[REDACTED]even when keys are set.
CONFLICT on reads or creates¶
getSecretValue→CONFLICT: the secret is scheduled for deletion (InvalidRequestException);restoreSecretit or wait for the recovery window to end. The provider reports such a secret as missing.getSecretValueright after a forced delete (forceDeleteWithoutRecovery/permanent: true) can also returnCONFLICT("marked for deletion") for a few seconds before it becomesNOT_FOUND.createSecret→CONFLICT: the name exists, possibly in its recovery window. UseforceDeleteWithoutRecovery/permanent: truefor names you want to re-create at once.
Reads right after a write see old data¶
AWS Secrets Manager is eventually consistent; these are AWS behaviours, not SDK bugs. Observed against a live account:
getSecretValue(id, {versionStage: "AWSPREVIOUS"})right afterputSecretValuecan returnNOT_FOUNDuntil the stage labels have moved.describeSecretand the AWS secret cache'srefreshNowcan briefly still return the previous version (versionIdsToStages, value).listSecretsmisses secrets that were just created or just scheduled for deletion for about 3 seconds.- Right after a forced delete,
getSecretValuemaps toCONFLICT("marked for deletion"), see above.
Code that writes and then reads at once (tests, rotation steps) should retry the read for a few
seconds, or use the values the write call returned (versionId, versionStages).
TLS errors¶
- Self-signed or private CA: set
tls.caCertPath(GROW_AWS_SECRETS_MANAGER_CA_CERT). - Mutual TLS: set both
tls.clientCertPathandtls.clientKeyPath. tls.insecureSkipVerifyonly for local tests.
RATE_LIMITED¶
AWS throttled the calls (ThrottlingException). Use the AWS secret cache
for hot paths. The AWS SDK retries throttled calls with backoff; when the retries are used up the
error is RATE_LIMITED (retryable, with retryAfter when AWS sent Retry-After) so your code can
schedule a later attempt.
TIMEOUT¶
requestTimeout (30 s) bounds one attempt and totalTimeout (120 s) the whole call including
retries. Raise them for slow operations, or pass a per-call deadline (signal, ctx, cancel token,
future timeout).
See what the client does¶
Turn on the logger or add hooks: both see every attempt with redacted headers and URLs.
Docker integration tests fail¶
- Run one language:
make test-docker LANGS=go. - In CI, download the
compose-logs-<lang>artifact of the failed job. - Locally, the stack is removed after each run; rerun with Docker logs open
(
docker compose -f docker/compose.yml logs -f) in a second terminal.
Still stuck?¶
Open an issue with the language, package version, AWS region (or emulator version) and a minimal reproduction (with secrets removed): https://github.com/grow-codes-packages/aws-secrets-manager/issues.