The problem

An ECR login token is valid for 12 hours. That is the entire lifespan of the credential you get from aws ecr get-login-password, and it is the thing that makes ECR awkward inside Kubernetes.

The trap is that a hand-made pull secret looks like it works. You run kubectl create secret docker-registry, your Pods pull their images, and everything is fine — until the next day, when the token has expired and new Pods sit in ImagePullBackOff with a 403 from the registry. Existing Pods keep running, which makes the failure worse: the first sign of trouble is a deploy or a node reboot, long after the change that caused it.

The approach

Treat the pull secret as a rotating credential rather than a static one. Run a CronJob inside the cluster that fetches a fresh token every 8 hours and rewrites the Secret. The 8-hour interval leaves plenty of margin inside the 12-hour window, so there is no moment where the stored token is close to expiring.

The trade-off is a credential with ECR read access living in the cluster, plus RBAC to let one ServiceAccount manage Secrets in its own namespace. Here is the flow: 

CronJob (every 8h)
  |- read AWS keys from Secret "ecr-secret"
  |- aws ecr get-login-password
  |- kubectl delete secret ecr-registry-secret --ignore-not-found
  '- kubectl create secret docker-registry ecr-registry-secret

Deployment
  '- imagePullSecrets: [ecr-registry-secret]

Prerequisites: the AWS credentials Secret

Store the IAM keys in the namespace where the workloads live. Create a dedicated IAM user for this and attach only the managed policy AmazonEC2ContainerRegistryReadOnly. A key with broader permissions turns any cluster compromise into an AWS compromise. 

kubectl create secret generic ecr-secret \
  --from-literal=access_key='<AWS_ACCESS_KEY_ID>' \
  --from-literal=secret_key='<AWS_SECRET_ACCESS_KEY>' \
  -n passtobridge

Use placeholders and inject the real values from your own pipeline or a password manager. Do not paste live keys into a manifest, a shell history, or a blog post.

The CronJob

Save this as ecr-cronjob.yaml. It creates the ServiceAccount, the RBAC to let that account manage Secrets in one namespace, and the CronJob itself. 

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ecr-updater
  namespace: passtobridge

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ecr-updater-role
  namespace: passtobridge
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "delete", "create", "patch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ecr-updater-rolebinding
  namespace: passtobridge
subjects:
- kind: ServiceAccount
  name: ecr-updater
  namespace: passtobridge
roleRef:
  kind: Role
  name: ecr-updater-role
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: ecr-token-refresher
  namespace: passtobridge
spec:
  # every 8 hours (00:00, 08:00, 16:00)
  schedule: "0 */8 * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: ecr-updater
          containers:
          - name: refresher
            image: amazon/aws-cli:latest
            env:
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: ecr-secret
                  key: access_key
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: ecr-secret
                  key: secret_key
            - name: AWS_DEFAULT_REGION
              value: "ap-east-1"
            command:
            - /bin/sh
            - -c
            - |-
              set -e
              KUBECTL=/usr/local/bin/kubectl

              echo "--- 1. verify the mounted kubectl ---"
              $KUBECTL version --client

              echo "--- 2. fetch an ECR login token ---"
              TOKEN=$(aws ecr get-login-password --region ${AWS_DEFAULT_REGION})

              echo "--- 3. recreate the Secret ---"
              # delete then create; --ignore-not-found keeps the first run clean
              $KUBECTL delete secret ecr-registry-secret -n passtobridge --ignore-not-found

              $KUBECTL create secret docker-registry ecr-registry-secret \
                --docker-server=839823279171.dkr.ecr.ap-east-1.amazonaws.com \
                --docker-username=AWS \
                --docker-password="${TOKEN}" \
                -n passtobridge

              echo "--- done: ECR credentials refreshed ---"
            volumeMounts:
            - name: host-kubectl
              mountPath: /usr/local/bin/kubectl
              readOnly: true
          restartPolicy: OnFailure
          volumes:
          - name: host-kubectl
            hostPath:
              # RKE2's default kubectl path
              path: /var/lib/rancher/rke2/bin/kubectl
              type: File

Why mount kubectl from the host?

amazon/aws-cli ships the AWS CLI but not kubectl. Rather than build a custom image, the CronJob mounts the host binary through a hostPath volume. Two consequences are worth knowing: the CronJob is pinned to whichever node has that file, and type: File matters — if the path is missing, the Pod fails loudly at mount time instead of silently running an empty directory.

The alternative is to bake both binaries into one image. It is more work up front but removes the node affinity and works the same on any distribution, not just RKE2.

Why delete then create, instead of patching?

kubectl create secret docker-registry writes the credential into the .dockerconfigjson key in one shot. Updating that key in place is possible but fiddly — the value is a nested JSON document, so a patch has to re-encode the whole thing. Deleting and recreating is idempotent and obvious.

The one thing to keep in mind is the brief window between the delete and the create where the Secret does not exist. Any Pod scheduled in that window has no pull credential. In practice the gap is milliseconds and Pods retry, but if you want to eliminate it entirely, generate the new token first, write it under a temporary name, and swap the reference.

The workload manifest

Every Deployment that pulls from ECR needs to reference the refreshed Secret. Note that imagePullSecrets does not merge — if you list it here, keep it here and do not rely on the ServiceAccount's own pull secrets. 

spec:
  template:
    spec:
      # 1. reference the rotating credential
      imagePullSecrets:
      - name: ecr-registry-secret
      # 2. schedule onto worker nodes, away from the masters
      nodeSelector:
        ingress-target: "true"
      containers:
      - name: my-app
        image: 839823279171.dkr.ecr.ap-east-1.amazonaws.com/<repository>:<tag>

Verify it works

Do not wait a day to find out whether the rotation actually runs. Trigger the CronJob manually and check the Secret's age: 

# run the job once, right now
kubectl create job --from=cronjob/ecr-token-refresher ecr-manual-test -n passtobridge

# watch it finish
kubectl get pods -n passtobridge -l job-name=ecr-manual-test -w
kubectl logs -n passtobridge -l job-name=ecr-manual-test

# confirm the Secret was rewritten
kubectl get secret ecr-registry-secret -n passtobridge \
  -o jsonpath='{.metadata.creationTimestamp}{"\n"}'

The creationTimestamp should be seconds ago, not the day you first created it. That timestamp is the real proof the rotation path works end to end.

To confirm the previous token would indeed have failed, an easy check is to compare the decoded credential before and after: 

kubectl get secret ecr-registry-secret -n passtobridge \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | head -c 120

Set a calendar reminder for the day after deployment and confirm that a Pod created then still pulls successfully. That catches the case where the CronJob runs but writes a credential the kubelet rejects.

Things that bit me

  • The 12-hour limit is on the token, not the IAM key. The keys stay valid indefinitely; only the derived login token expires. That is exactly why rotating the Secret works and rotating the IAM user is unnecessary.
  • Region must match the registry. A token from one region cannot authenticate against another region's registry. Both AWS_DEFAULT_REGION and --docker-server here say ap-east-1; keep them in sync.
  • Existing Pods do not re-pull. A stale Secret only surfaces for new Pods, so the failure appears during deploys, scaling, or node reboots. Expect the first symptom to be a deploy that will not roll out.
  • The CronJob needs the Secret to already exist. The secretKeyRef entries are required at container start, so ecr-secret must be created before the first run, otherwise the Pod fails to start with a CreateContainerConfigError.
  • Keep the IAM policy minimal. ECR read-only is all this needs. A key with write or broad S3 access widens the blast radius for no benefit.