COPY
only the minimal amount of files. Avoid copying .git
COPY
entire large directories into the build environment and only using a subset of the files within them. Or worse, copying the entire repository (which might also include .git
) for no good reason.README.md
or running git fetch
might cause slow commands like go mod download
to be re-executed.COPY
commands (among other things) to mark certain files as inputs to the build. If any file included in a COPY
changes, then the build will continue from that COPY
command onwards. For this reason, you want to be as specific as possible when including files in a COPY
command. In some cases, you might even have to list files individually..git
or to unrelated files, like README.md
. However, this can be arranged even better, to avoid downloading all the dependencies on every *.go
file change.COPY
command is to use the .earthlyignore
file. Note, however, that this is best left as a last resort, as new files added to the project (that may be irrelevant to builds) would need to be manually added to .earthlyignore
, which may be error-prone. It is much better to have to include every new file manually into the build (by adding it to a COPY
command), than to exclude every new file manually (by adding it to the .earthlyignore
), as whenever any such new file must be included, then the build would typically fail, making it harder to make a mistake compared to the opposite.ENV
for image env vars, ARG
for build configurabilityENV
variables and ARG
variables seem similar, however they are meant for different use-cases. Here is a breakdown of the differences, as well as how they differ from the Dockerfile-specific ARG
command:ENV
ARG
ARG
FROM
BUILD +target --<key>=<value>
or similar)ENV
is needed is when you want the value to be stored as part of the final image's configuration. This causes any FROM
or docker run
using that image to inherit the value.ARG
is the way to achieve that.GIT CLONE
if possibleGIT CLONE
or RUN git clone
, whenever possible.SAVE ARTIFACT
. This makes the code more readable and maintainable. The fact that an artifact is saved during a build constitutes an explicit API of the repository.GIT CLONE
might help to provide a faster, yet imperfect solution.GIT CLONE
is better suited is when the operation needs to take place on the whole source repository. For example, performing Git operations, such as tagging, creating branches, or merging.GIT CLONE
:GIT CLONE
FROM github.com/my-co/my-proj:my-branch+my-target
GIT CLONE --branch=my-branch [email protected]:my-co/my-proj
:<branch>
--branch
ARG
sGIT CLONE
vs RUN git clone
GIT CLONE
instruction that can be used to clone a Git repository. It is recommended that GIT CLONE
is used rather than RUN git clone
, for a few reasons:GIT CLONE
as a first-class input (BuildKit source). As such, Earthly caches the repository internally and downloading only incremental differences on changes.RUN git clone
would not be able to detect that, as it is not recognized as an input. So it would naively reuse the cache when it shouldn't.GIT CLONE
will pass-through Earthly settings for authentication, such as SSH agent access and/or HTTPS credentials.GIT CLONE
does have some limitations, however. It only performs a shallow clone, it does not have the branch information, it does not have origin information, and it does not have the tags downloaded. Even in such cases, it might be better to attempt to reintroduce the information after a GIT CLONE
, whenever possible, in order to gain the caching benefits.RUN git clone
, consider using both in conjunction, to gain the hash awareness benefits.GIT CLONE
and RUN git clone
:GIT CLONE
RUN git clone
ssh://
, https://
, [email protected]
etcssh://
, https://
, [email protected]
etcRUN --ssh
IF [...]
vs RUN if [...]
IF
command, which allows for complex control flow within Earthly recipes. However, there is also the possibility of using the shell if
command to accomplish similar behavior. Which one should you use? Here is a quick comparison:IF
RUN if
IF
is more powerful in that it can include other Earthly commands within it, allowing for rich conditional behavior. Examples might include optionally saving images, using different base images depending on a set of conditions, initializing ARG
s with varying values.RUN if
, however is often simpler, and it only uses one layer.RUN if
whenever possible (e.g. only RUN
commands would be involved), to encourage simplicity, and otherwise to use IF
.FOR ... IN ...
vs RUN for ... in ...
IF
vs RUN if
, there is a similar debate for the Earthly builtin command FOR
vs RUN for
. Here is a quick comparison of the two 'for' flavors:FOR
RUN for
IF
vs RUN if
comparison, FOR
is more powerful in that it can include other Earthly commands within it, allowing for rich iteration behavior. Examples might include iterating over a list of directories in a monorepo and calling Earthly targets within them, performing SAVE IMAGE
over a list of container image tags.RUN for
, however is often simpler, and it only uses one layer.RUN for
whenever possible (e.g. only RUN
commands would be involved), to encourage simplicity, and otherwise to use FOR
.LOCALLY
LOCALLY
. Here is an example:LOCALLY
target:ARG
to decide on whether to execute the target on the host or not:earthly +my-target --run_locally=true
, otherwise earthly +my-target
will execute in the sandboxed environment (the same way it executes in CI).IF
expression. For example, let's assume that the company provided Go image only supports the linux/amd64
platform, and therefore, you'd like to use the official golang image when ARM (linux/arm64
) is detected. Here's how this can be achieved:FROM
s within the same target. This is completely valid. On encountering another FROM
expression, the current build environment is reset and another fresh root is initialized, containing the specified images data.RUN --push
for deployment commandsSAVE IMAGE --push
).RUN
command together with the --push
flag. The --push
will ensure that:earthly --push
)RUN --no-cache
should be avoided for this use-case, as it has some potentially dangerous downsides:--push
mode.RUN --push
instead.--secret
, not ARG
s to pass secrets to the buildearthly --secret
, Earthly Cloud Secrets, and RUN --secret
.ARG
s for passing secrets is strongly discouraged, as the secrets will be leaked in build logs, the build cache and the possibly in published images.RUN
command.rm
is forgotten, or if the removal is performed under a separate RUN
command.+build
starts, the artifact would not have been output yet. In fact, +dep
and +build
will run completely parallel anyway - as Earthly does not know of a dependency between them.+dep
no longer needs to save the file locally. Also, the COPY
command no longer references the file from the local file system. It has been replaced with an artifact reference from the target +dep
. This reference will tell Earthly that these two targets depend on each other and will therefore schedule the relevant parts to run sequentially.+all
target, we no longer have to call both +dep
and +build
. The system will automatically infer that when building +build
, +dep
is also required.+test
runs, it will not have the image available, unless it has been pushed in a previous execution (which means that the image may be stale).--load
instruction will inform Earthly that the two targets depend on each other and will therefore build the image and load it into the Docker daemon provided by WITH DOCKER
.WITH DOCKER --pull
WITH DOCKER
block, it is important to declare it via WITH DOCKER --pull
, for a few reasons:WITH DOCKER
wipes the state of the Docker daemon (including its cache) after every run.WITH DOCKER
is not logged into registries. Your local Docker login config is not propagated to the daemon. This means that you may run into issues when trying to pull images from private registries, but also, DockerHub rate limiting may prevent you from pulling images consistently from public repositories.WITH DOCKER --compose
, Earthly will automatically pull images declared in the compose file for you, as long as they are not already being loaded from another target via WITH DOCKER --load
. So in this case, you do not need to declare those image with WITH DOCKER --pull
.COPY +my-target/...
to pass files to and from LOCALLY
targetsLOCALLY
, it is tempting to skip on using Earthly constructs for passing files between targets. However, this can be problematic.+dep
and +build
is not guaranteed. So in some runs, the file ./my-artifact.txt
will be created before the +build
target is executed, and in some runs it will be created after.LOCALLY
target):SAVE ARTIFACT ... AS LOCAL ...
for the transfer of the artifact to the LOCALLY
target. As Earthly outputs are written at the end of the build, the target +run-locally
will not have the file in time (or it might have it from a previous run only, meaning that it might be stale).COPY
command using an artifact reference will inform Earthly of the dependency between the two targets, and will therefore cause the transfer of artifact between the two properly.LOCALLY
targets:COPY
and an artifact reference.WITH DOCKER --load=+my-target
to pass images to LOCALLY
targetsLOCALLY
target, the image needs to be output in the middle of the build.WITH DOCKER --load
and a target reference:--load
instruction will inform Earthly of the dependency and will therefore cause the image to be output right before the WITH DOCKER
RUN
command executes.ARG
with values that include randomnessCOPY --dir
to copy multiple directoriesCOPY
command differs from the unix cp
in that it will copy directory contents, not the directories themselves. This requires that copying multiple directories to be split across multiple lines:COPY --dir
, which makes COPY
behave more like cp
and less like the Dockerfile COPY
. The --dir
flag can be used therefore to copy multiple directories in a single command:go
binary. After the application binary has been built via go build
, there is no longer a need for the go
binary. So the production image should not contain it. Here is an example:SAVE ARTIFACT ... AS LOCAL ...
for generated code, not LOCALLY
SAVE ARTIFACT ... AS LOCAL ...
via regular Earthly targets, rather than via running the generation command in LOCALLY
. There are multiple reasons for this:LOCALLY
loses the repeatability benefits. This means that the same command could end up generating different code, depending on the system it is being run on. Differences in the environment, such as the version of code generator installed (e.g. protoc
), or certain environment variables (e.g. GOPATH
) could cause the generated code to be different.LOCALLY
will not be usable in the CI, as the CI script would typically enable --strict
mode.\
). Remember to chain multiple shell commands via &&
in order to correctly exit if one of the commands fails.SAVE ARTIFACT
and then copied over using an artifact reference.some-file.txt
is copied from the sibling directory dir1
. This will not work in Earthly as the file is not in the build context of ./dir2/Earthfile
(the build context in this case is ./dir2
). To address this issue, we can create an Earthfile in dir1
that exports the file some-file.txt
as an artifact.dir1
, where all the files required outside of it are explicitly exported.LOCALLY
.LOCALLY
is not allowed in --strict
mode (or in --ci
mode), as it introduces a dependency from the host machine, which may interfere with the repeatability property of the build.COPY ../
is not possible in Earthly today, there are some rare, but valid use-cases for this functionality. This is being discussed in GitHub issue #1221.SAVE ARTIFACT
or SAVE IMAGE
commands, respectively. Those artifacts can then be referenced in higher-level Earthfiles via artifact and target references (COPY ./deep/dir+some-target/an/artifact ...
, FROM ./some/path+my-target
)..proto
files, in its own Earthfile.ast/parser
- Earthfile contains the logic for generating Go source code based on an ANTLR grammar.docker build -f ./services/app1.Dockerfile ./app1-src-dir ...
and so on.docker build -f
option.+build-for-windows
:+build-for-windows
itself exports the artifacts means that it can be referenced directly in other targets as COPY +build-for-windows/output ./
.+build-wrapper
to reuse the logic in +build
, but ultimately to create an image that is saved under a different name. This can then be used in a WITH DOCKER --load
statement directly (whereas if there was no image pass-through, then +build-wrapper
couldn't have been used).earthly/dind
WITH DOCKER
, it is recommended that you use the official earthly/dind
image (preferrably :alpine
) for running Docker-in-Docker. Earthly's WITH DOCKER
requires that the Docker engine is installed already in the image it is running in.WITH DOCKER
will need to first install it - it usually does so automatically - however, the cache will be inefficient. Consider the following example:some-other-image:latest
does not already have Docker engine installed. This means that on the WITH DOCKER
line, Earthly will add a hidden installation step, to add Docker engine. This takes some time to execute, but it will work.docker-compose.yml
. That will cause the build to re-execute without cache from the COPY
command onwards, meaning that the installation of Docker engine will be repeated.COPY
command. Please note that this particular UDC is fastest when ran on top of an alpine-based image.earthly/dind
image, if possible.WITH DOCKER
WITH DOCKER
starts up a transient Docker daemon for that specific instruction and then shuts it down and completely wipes its data afterwards. That does not mean, however, that you cannot export any information from it (or from any container running within), to be used in another part of the build. Although you may not run any non-RUN
commands within WITH DOCKER
, you can still use SAVE ARTIFACT
(and any other command) after the WITH DOCKER
instruction. The Docker daemon's data is wiped - but the rest of the build environment remains intact.--ci
when running in CILOCALLY
, which cause Earthly to be less repeatable, and yet might satisfy very much needed use-cases that are typically out of scope of a CI build.--ci
flag, which simply expands to --no-output --use-inline-cache --save-inline-cache --strict
. The --ci
flag therefore, prevents the use of commands that are not repeatable, enables inline caching and disables outputting artifacts and images on the CI host.LOCALLY
and other non-strict commands--strict
mode (earthly --strict +my-target
or earthly --ci +my-target
), the usage of these commands is not allowed.--ci
is passed in.LOCALLY
. The LOCALLY
command skips the sandboxing of the build and executes all commands directly on the host machine.LOCALLY
may be used are:main
branch onlymain
branch only, and to disable any pushing on PR builds. Although the main
build will be slower, it will allow for maximum use of cache in PR builds, without the slowdown of further pushes.earthly --ci --push +target
. PR build: earthly --ci +target
.EARTHLY_PUSH
, which may be easier to manipulate in your CI of choice.earthly --ci --push --remote-cache=.... --max-remote-cache +target
. The idea, again is to tradeoff performance on the main
branch, for the benefit of faster PR builds. Whether this is actually beneficial needs to be measured on a project-by-project basis, however.earthly -i
to debug failures-i
flag: earthly -i +my-target
. Once dropped into the container's shell, you may use the up arrow to pre-populate the previously failed command should you wish to retry it or amend it. To exit the shell, press Ctrl+D
.--push
and then repeating the same build, but with --push
enabled.ARG
outside of Earthly and then calling Earthly with that value.