Pull-Through Cache
Docker Hub, Quay, and other registry providers all have pull limits, and costs associated with running them. Running large builds (or many small builds, frequently) may incur excess costs, rate limiting, or both. This guide will help you set up your own "pull-through" cache to optimize traffic, and bypass the limitations imposed by registry providers.
A pull through cache is a registry mirror that contains no images. When your client checks the registry for an image, the registry will either:
Give an existing response from its cache; thereby avoiding egress (or a pull) from your registry,
Or pull the image and its metadata from the registry on your behalf; caching it for later use.
To run a cache, you'll need the ability to deploy a persistent service, somewhere. This could be a dedicated instance with Docker installed, or a container in your Kubernetes cluster. While we won't be giving details for how to set up either of these ways to run a service, we will be sharing configuration and usage details, and how you can use it with Earthly.
Docker has a guide for getting a pull-through cache up and running, and good documentation of the available options. You can get the registry image (and details) here.
Pull-through caches run unsecured by default. Add an htpasswd file for basic authentication, at a minimum:
auth:htpasswd:realm: basic-realmpath: /auth/htpasswd
Adding TLS is also highly recommended. you can bring your own certificates, or use the built-in LetsEncrypt support:
http:tls:letsencrypt:cachefile: /certs/cachefileemail: [email protected]hosts: [my.cool.mirror.horse]
The currently shipping library/registry image does not support the DNS-01 challenge yet, and some of the LetsEncrypt challenge support is getting out of date. If you need this, there is a tracking issue; We have had success by building the binary ourselves and replacing it in the image that Docker ships.
