Rook-Ceph RGW S3 access

This playbook validates that the Rook-Ceph RADOS Gateway accepts S3 requests from outside the cluster using the static access keys created at deployment time. Run it after a fresh deployment or after any change to the RGW configuration.

Note RGW supports both static access keys and STS federation. Steps 1–4 validate the static bootstrap user only; they do not prove a tenant's federated access or catalog permissions. Use Federated access checks for that boundary.

Prerequisites

• aws CLI installed locally.
• jq installed locally.
• kubectl access to the cluster, with permission to read secrets in the rook-ceph namespace.
• The RGW endpoint reachable from the workstation as https://<ceph-s3-host>.

1. Export the static RGW credentials

export RGW_URL="https://<ceph-s3-host>"

export AWS_ACCESS_KEY_ID=$(kubectl -n rook-ceph get secret rook-ceph-rgw-bootstrap \
  -o jsonpath='{.data.sts_client_access_key}' | base64 -d)

export AWS_SECRET_ACCESS_KEY=$(kubectl -n rook-ceph get secret rook-ceph-rgw-bootstrap \
  -o jsonpath='{.data.sts_client_secret}' | base64 -d)

unset AWS_SESSION_TOKEN
export AWS_DEFAULT_REGION="us-east-1"

2. Validate connectivity

aws --endpoint-url "$RGW_URL" s3api list-buckets --no-paginate

A successful response returns the (possibly empty) list of buckets owned by the bootstrap user.

3. Upload and download a test object

Create a bucket and upload a file:

export BUCKET="rgw-static-test-$(date +%s)"
printf 'hello rgw static\n' > /tmp/rgw-test.txt

aws --endpoint-url "$RGW_URL" s3 mb "s3://$BUCKET"
aws --endpoint-url "$RGW_URL" s3 cp /tmp/rgw-test.txt "s3://$BUCKET/rgw-test.txt"
aws --endpoint-url "$RGW_URL" s3 ls "s3://$BUCKET/"

Download and verify:

aws --endpoint-url "$RGW_URL" s3 cp "s3://$BUCKET/rgw-test.txt" /tmp/rgw-test-downloaded.txt
cat /tmp/rgw-test-downloaded.txt

4. Clean up

aws --endpoint-url "$RGW_URL" s3 rm "s3://$BUCKET/rgw-test.txt"
aws --endpoint-url "$RGW_URL" s3 rb "s3://$BUCKET"
rm -f /tmp/rgw-test.txt /tmp/rgw-test-downloaded.txt

Clear the exported bootstrap credentials when finished:

unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN

Federated access checks

Platform operators validate this path separately for the Rook-Ceph provider. Use a tenant test identity and an approved test object; do not distribute the shared bootstrap credentials to notebook users.

1. Confirm sts_federation = true in [object_storage.capabilities], RGW STS is enabled, and the RGW IAM bootstrap and Lakekeeper role-provisioning Jobs completed. External S3 does not use these RGW federation hooks.
2. Inspect the intended role's trust policy: the issuer, client claim, and any tenant-claim condition must match the test identity. Check the role's S3 resource and prefix policy. A tenant-scoped storage policy does not, by itself, restrict which identities may assume that role.
3. Exercise a catalog-backed read/write through Run Spark workloads, using the caller's credentials and the configured tenant role. Lakekeeper credential vending uses its separate bucket-scoped role. Do not substitute the broad bootstrap role to make a failed test pass.
4. Verify both allowed access and refusal outside the intended boundary: test a permitted table/object, then a known foreign-tenant table and object prefix without modifying them. Check catalog authorization and S3 authorization separately.
5. When access fails, inspect token expiry, issuer/client claims, role trust, catalog grants, TLS trust, and S3 prefix policy. An enabled STS endpoint or successful static-key test does not prove this flow works.

Optional: create an additional static RGW user

If a dedicated user is needed instead of the bootstrap sts-client user, create it from inside the cluster with radosgw-admin and distribute the generated key and secret through the platform's standard secret distribution flow.

Troubleshooting

• 503 Service Unavailable on list-buckets — the RGW pod is not yet ready. Wait for the rollout to complete and confirm the pod is Ready.
• 403 Forbidden — the exported environment variables are missing or do not match a valid RGW user. Re-export them in the same shell.
• AWS CLI parser errors (NoneType errors) — rerun with --debug and inspect the underlying RGW XML error code (AccessDenied, InvalidAccessKeyId, etc.).
• TLS or endpoint errors — verify RGW_URL and that the certificate chain is trusted on the workstation.

Related pages

• Rook-Ceph component