LogoLogo
HomeBlogGitHubGet Started FreeLogin
Earthly 0.8
Earthly 0.8
  • 👋Introduction
  • 💻Install Earthly
  • 🎓Learn the basics
    • Part 1: A simple Earthfile
    • Part 2: Outputs
    • Part 3: Adding dependencies With Caching
    • Part 4: Args
    • Part 5: Importing
    • Part 6: Using Docker In Earthly
    • Part 7: Using remote runners
    • Part 8a: Using Earthly in your current CI
    • Final words
  • ⭐Featured guides
    • Rust
  • 📖Docs
    • Guides
      • Importing
      • Build arguments and variables
      • Secrets
      • Functions
      • Using Docker in Earthly
      • Multi-platform builds
      • Authenticating Git and image registries
      • Integration Testing
      • Debugging techniques
      • Podman
      • Configuring registries
        • AWS ECR
        • GCP Artifact Registry
        • Azure ACR
        • Self-signed certificates
      • Using the Earthly Docker Images
        • earthly/earthly
        • earthly/buildkitd
      • ✅Best practices
    • Caching
      • Caching in Earthfiles
      • Managing cache
      • Caching via remote runners
    • Remote runners
    • Earthfile reference
      • Builtin args
      • Excluding patterns
      • Version-specific features
    • The earthly command
    • Earthly lib
    • Configuration reference
    • Examples
    • Misc
      • Alternative installation
      • Data collection
      • Definitions
      • Public key authentication
  • 🔧CI Integration
    • Overview
    • Use the Earthly CI Image
    • Build your own Earthly CI Image
    • Pull-Through Cache
    • Remote BuildKit
    • Vendor-Specific Guides
      • GitHub Actions
      • Circle CI
      • GitLab CI/CD
      • Jenkins
      • AWS CodeBuild
      • Google Cloud Build
      • Bitbucket Pipelines
      • Woodpecker CI
      • Kubernetes
  • ☁️Earthly Cloud
    • Overview
    • Managing permissions
    • Cloud secrets
    • Earthly Satellites
      • Managing Satellites
      • Using Satellites
      • Self-Hosted Satellites
      • GitHub runners
      • Best Practices
      • Bring Your Own Cloud (BYOC)
        • AWS
          • Requirements
          • CloudFormation
          • Terraform
          • Manual
        • VPN
          • Tailscale
Powered by GitBook
On this page
  • Overview
  • Compatibility
  • Resources
  • Setup (earthly/earthly Only)
  • Dependencies
  • Installation
  • Configuration
  • Setup (Remote earthly/buildkitd)
  • Configuration
  • Example

Was this helpful?

Edit on GitHub
  1. CI Integration
  2. Vendor-Specific Guides

Kubernetes

PreviousWoodpecker CINextOverview

Last updated 12 months ago

Was this helpful?

Note

This guide is related to self-hosting a remote BuildKit, however, Self-Hosted Satellites beta are now available. Self-Hosted Satellites provide more features, have better security, and are easier to deploy than remote BuildKit. Check out the for more details and instructions to deploy in Kubernetes or AWS EC2.

Overview

Kubernetes isn't a CI per-se, but it can serve as the underpinning for many modern CI systems. As such, this example serves as a bare-bones example to base your implementations on.

Compatibility

earthly has been tested with the all-in-one earthly/earthly mode, and works as long as the pod runs in a privileged mode.

It has also been tested with a single remote earthly/buildkitd running in privileged mode, and an earthly/earthly pod running without any additional security concerns. This configuration is considered experimental. See .

Multi-node earthly/buildkitd configurations are currently unsupported.

Resources

Setup (earthly/earthly Only)

This is the recommended approach when using Earthly within Kubernetes. Assuming you are following the steps outlined in the , here are the additional things you need to configure:

Dependencies

Your Kubernetes cluster needs to allow privileged mode pods. It's possible to use a separate instance group, along with Taints and Tolerations to effectively segregate these pods.

Installation

Configuration

earthly/earthly currently requires the use of privileged mode. Use this in your container spec to enable it:

securityContext:
  privileged: true

The earthly/earthly container will operate best when provided with decent storage for intermediate operations. Mount a volume like this:

volumeMounts:
  - mountPath: /tmp/earthly
    name: buildkitd-temp
...
volumes:
  - name: buildkitd-temp
    emptyDir: {} # Or other volume type

The location within the container for this temporary folder is configurable with the EARTHLY_TMP_DIR environment variable.

The earthly/earthly image will expect to find the source code (with Earthfile) rooted in the default working directory, which is set to /workspace.

Setup (Remote earthly/buildkitd)

Note

This an experimental configuration.

It is possible to run multiple earthly/buildkitd instances in Kubernetes, for larger deployments. Follow the configuration instructions for using the earthly/earthly image above.

There are some caveats that come with this kind of a setup, though:

  1. Some local cache is not available across instances, so it may take a while for the cache to become "warm".

  2. Builds that occur across multiple instances simultaneously may fail in odd ways. This is not supported.

  3. The TLS configuration needs to be shared across the entire fleet.

Configuration

To mitigate some of the issues, it is recommended to run in a "sticky" mode to keep builds pinned to a single instance for the duration. You can see how to do this in our example:

# Use session affinity to prevent "roaming" across multiple BuildKit instances; if needed.
sessionAffinity: ClientIP
sessionAffinityConfig:
  clientIP:
    timeoutSeconds: 600 # This should be longer than your build.

Example

Note

To run it yourself, first you will need to install some prerequisites on your machine. This example requires kind and kubectl to be installed on your system. Here are some links to installation instructions:

earthly ./kubernetes+start

Running this target will:

  • Create a kind cluster named earthlydemo-aio

  • Create & watch a job that runs an earthly build

When the example is complete, the cluster is left up and intact for exploration and experimentation. If you would like to clean up the cluster when complete, run (again from the root of the repository):

earthly ./kubernetes+clean

The default image from earthly/earthly should be sufficient. If you need additional tools or configuration, you can .

In some instances, notably when using within your cluster, the MTU of the clusters network may end up mismatched with the internal CNI network, preventing external communication. You can set this through the CNI_MTU environment variable to force a match.

This example is not production ready, and is intended to showcase configuration needed to get Earthly off the ground. If you run into any issues, or need help, !

See our .

When you are ready, clone the , and then run (from the root of the repository):

🔧
Self-Hosted Satellites Guide
these additional instructions
Kubernetes Documentation
Kubernetes Taints & Tolerations
overview
create your own runner image
Calico
don't hesitate to reach out
Kubernetes examples
kind Quick Start
Install kubectl
ci-examples repository