Building container images in GitHub actions from a Git-based deployment workflow is incredibly common these days. Luckily caching the output layers is incredibly cheap and easy.

Docker's "new (since 2023)" build engine named buildx supports both pulling cached layers for the build, and pushing cached layers after the build out of the box using the cache-from and cache-to options. Along with these options buildx also supports multiple backends such as local files (local), GitHub Actions (gha), and remote registry (registry). Other backends such as S3 and Azure Blob Storage are also supported.

1. The local backend simply caches the layers to disk which is the default behavior when you build images locally on your computer too. Because the GitHub runners are ephemeral, this option has practically zero value when running in Actions.
2. GitHub Actions supports caching artifacts from Actions runs natively, with a 10GB per-repo cache. This might just be sufficient for smaller artifacts, but since this cache is repo-wide it is very easy to fill it when using monorepos. If you are on a paid GitHub plan, you can upgrade the 10GB limit, but in my opinion this cache is better for non-layer artifacts such as JavaScript dependencies to run linters.
3. The registry backend creates a separate image tag for the produced image, and pushes the individual layers to a specified upstream registry. Since we already plan on pushing the built image to a registry, this is my preferred cache backend.

docker buildx build \
  --cache-from type=registry,ref=<account-id>.dkr.ecr.eu-north-1.amazonaws.com/my-repo:cache \
  --cache-to type=registry,ref=<account-id>.dkr.ecr.eu-north-1.amazonaws.com/my-repo:cache,mode=max

In the example here I called the tag cache, but you could name it whatever you would like.

Depending on the current Docker Engine version used in the GitHub Actions runner you might have to enable the containerd image store. My preferred way of doing this is using the docker/setup-docker-action action.

    - name: Set up Docker with containerd image store
      uses: docker/setup-docker-action@1a6edb0ba9ac496f6850236981f15d8f9a82254d # v5
      with:
        daemon-config: |
          {
            "features": {
              "containerd-snapshotter": true
            }
          }

Using AWS ECR

Here are some tips and experiences from configuring this setup when using AWS ECR as the container registry.

Since we are good citizens, we like to keep release tags immutable using a lifecycle policy like this:

resource "aws_ecr_repository" "this" {
  name                 = module.label.id
  image_tag_mutability = "IMMUTABLE"

  ...
}

This won't work because the cache needs to be mutable, i.e writable. We have two options here; either use a separate ECR repository as the cache registry, or modify our policies with exceptions. I prefer to keep everything related to the image in one place, so I went with approach number two.

-  image_tag_mutability = "IMMUTABLE"
+  image_tag_mutability = "IMMUTABLE_WITH_EXCLUSIONS"
+
+  image_tag_mutability_exclusion_filter {
+    filter      = "cache*"
+    filter_type = "WILDCARD"
+  }

Tags that start with cache are now possible to mutate. I also like to add an exclusion for the latest tag, but that's of course not needed for this.

If we set the builx cache mode to max, all layers of the image will be pushed to the repository, unlike min which will only push the final image. When choosing max, we will also get a ton of untagged layers. These add up quickly so add a lifecycle policy to remove them periodically:

resource "aws_ecr_lifecycle_policy" "this" {
  policy     = <<-EOF
{
  "rules": [
    {
      "action": {
        "type": "expire"
      },
      "description": "Delete untagged images after 7 days",
      "rulePriority": 1,
      "selection": {
        "countNumber": 7,
        "countType": "sinceImagePushed",
        "countUnit": "days",
        "tagStatus": "untagged"
      }
    }
  ]
}
  EOF
  repository = aws_ecr_repository.this.id
}

Additionally, I like to only keep a small number of tagged images around to reduce the size of the repository and consequently my cloud bill.

{
  "action": {
    "type": "expire"
  },
  "description": "Keep the last 10 tagged images",
  "rulePriority": 2,
  "selection": {
    "countNumber": 10,
    "countType": "imageCountMoreThan",
    "tagStatus": "any"
  }
}

Finally, when using AWS ECR, you might also need to pass the image-manifest=true,oci-mediatypes=true options to buildx --cache-to.

docker buildx build \
  --cache-from type=registry,ref=<account-id>.dkr.ecr.eu-north-1.amazonaws.com/my-repo:cache \
  --cache-to type=registry,ref=<account-id>.dkr.ecr.eu-north-1.amazonaws.com/my-repo:cache,mode=max,oci-mediatypes=true,image-manifest=true

The GitHub Actions principal now also needs to pull images, so use a policy like this one:

jsonencode({
  Version = "2012-10-17",
  Statement = [
    {
      Effect = "Allow",
      Action = [
        "ecr:GetAuthorizationToken"
      ],
      Resource = "*"
    },
    {
      Effect = "Allow",
      Action = [
        "ecr:BatchCheckLayerAvailability",
        "ecr:CompleteLayerUpload",
        "ecr:InitiateLayerUpload",
        # PutImage is needed if you intend to push later
        "ecr:PutImage",
        "ecr:UploadLayerPart",
        "ecr:BatchGetImage",
        # Needed for Docker build cache
        "ecr:GetDownloadUrlForLayer"
      ],
      Resource = aws_ecr_repository.this.arn
    }
  ]
})

Hopefully these tricks can save you some $$ in CI minutes and accelerate your deployment flows :)