Optimizing Docker builds in GitHub Actions

I like containers, and I like building my containers automatically (and I am likely not the only one :D). Unfortunately GitHub Actions is not tuned for this out of the box, so here's my tips on how to speed this up!

In 2023 Docker stabilized the buildx engine which supports pushing and pulling cached layers from a Docker build with configurable storage backends. Assuming your Docker files make good use of layer caching this enables a considerable speedup in incremental builds. Docker gave us a few options to choose from:

Picking a backend

The local backend uses your local file system. Unfortunately we don't own the disk for our runners (although some self-hosted runners like Blacksmith have sticky disks!) plus they are ephemeral so this is quite worthless in CI.

When I first discovered this I thought hmm, can't I just use the GitHub Actions cache? That would let me just save those layers to cache and re-use them next time! Turns out the cache available in GitHub Actions is only 10GB large, and even worse; it is shared across all your workflows. If you have a heavier build process (such as compiling Rust binaries) you will blow through this cache and mess up everything that uses the cache. Not good. I think I'll keep using this cache for lighter tasks like running TypeScript linters.

Luckily for us we can use the registry backend to send the cached layers to the same place our image is heading! Oh and by the way, this is how easy it is to set up:

docker buildx build \
  --cache-from type=registry,ref=<account-id>.dkr.ecr.eu-north-1.amazonaws.com/my-repo:cache \
  --cache-to type=registry,ref=<account-id>.dkr.ecr.eu-north-1.amazonaws.com/my-repo:cache,mode=max

Also if you care, you can name the tag anything. I typically call it cache.

Quick tip: depending on the current Docker Engine version used in the GitHub Actions runner you might have to enable the containerd image store. If you use the official Docker setup action it's as easy as so:

    - name: Set up Docker with containerd image store
      uses: docker/setup-docker-action@1a6edb0ba9ac496f6850236981f15d8f9a82254d # v5
      with:
        daemon-config: |
          {
            "features": {
              "containerd-snapshotter": true
            }
          }

Let's stay tidy

Now you're probably like me; easily spooked by large cloud bills. Let's take a quick step to avoid blowing up our registry bill. Here, for Amazon ECR:

We are good citizens: we like to keep release tags immutable using a lifecycle policy like this:

resource "aws_ecr_repository" "this" {
  name                 = module.label.id
  image_tag_mutability = "IMMUTABLE"

  ...
}

Oh whoops! Can't do that, because the cache image needs to be mutable. If you are plagued by compliance auditors you can use a separate ECR repository for the cache. If you're free like me you can just make the cache image mutable.

-  image_tag_mutability = "IMMUTABLE"
+  image_tag_mutability = "IMMUTABLE_WITH_EXCLUSIONS"
+
+  image_tag_mutability_exclusion_filter {
+    filter      = "cache*"
+    filter_type = "WILDCARD"
+  }

If we set the builx cache mode to max, all layers of the image will be pushed to the repository, unlike min which will only push the final image. When choosing max, we will also get a ton of untagged layers. These add up quickly so add a lifecycle policy to remove them periodically:

resource "aws_ecr_lifecycle_policy" "this" {
  policy     = <<-EOF
{
  "rules": [
    {
      "action": {
        "type": "expire"
      },
      "description": "Delete untagged images after 7 days",
      "rulePriority": 1,
      "selection": {
        "countNumber": 7,
        "countType": "sinceImagePushed",
        "countUnit": "days",
        "tagStatus": "untagged"
      }
    }
  ]
}
  EOF
  repository = aws_ecr_repository.this.id
}

Using a time-based rather than count based policy here lets us have a somewhat hot cache if we're pushing a lot of work :) And then it'll just clean up on its own when you're away.

Additionally, I like to only keep a small number of tagged images around to reduce the size of the repository and consequently my cloud bill.

{
  "action": {
    "type": "expire"
  },
  "description": "Keep the last 10 tagged images",
  "rulePriority": 2,
  "selection": {
    "countNumber": 10,
    "countType": "imageCountMoreThan",
    "tagStatus": "any"
  }
}

Finally, when using AWS ECR, you might also need to pass the image-manifest=true,oci-mediatypes=true options to buildx --cache-to.

docker buildx build \
  --cache-from type=registry,ref=<account-id>.dkr.ecr.eu-north-1.amazonaws.com/my-repo:cache \
  --cache-to type=registry,ref=<account-id>.dkr.ecr.eu-north-1.amazonaws.com/my-repo:cache,mode=max,oci-mediatypes=true,image-manifest=true

Let Actions do Actions

I'm hoping you're a good citizen and grant GitHub Actions authorized access via OpenID Connect.

The GitHub Actions principal now also needs to pull images, so use a policy like this one:

jsonencode({
  Version = "2012-10-17",
  Statement = [
    {
      Effect = "Allow",
      Action = [
        "ecr:GetAuthorizationToken"
      ],
      Resource = "*"
    },
    {
      Effect = "Allow",
      Action = [
        "ecr:BatchCheckLayerAvailability",
        "ecr:CompleteLayerUpload",
        "ecr:InitiateLayerUpload",
        # PutImage is needed if you intend to push later
        "ecr:PutImage",
        "ecr:UploadLayerPart",
        "ecr:BatchGetImage",
        # Needed for Docker build cache
        "ecr:GetDownloadUrlForLayer"
      ],
      Resource = aws_ecr_repository.this.arn
    }
  ]
})

Hopefully these tricks can save you some $$ in CI minutes and accelerate your deployments :)