Debugging techniques
Traditional debugging of errors during image builds often require a developer to place various print commands through out the build commands to help reason about the state of the system before the failure occurs. This can be slow and cumbersome.
Earthly provides an interactive mode which gives you access to a root shell when an error occurs, which we'll cover in this guide.
Let's consider a test example that prints out a randomly generated phrase:
1
# Earthfile
2
​
3
FROM python:3
4
WORKDIR /code
5
​
6
test:
7
RUN curl https://raw.githubusercontent.com/jsvine/markovify/master/test/texts/sherlock.txt > /sherlock.txt
8
COPY generate_phrase.py .
9
RUN pip3 install markovify
10
RUN python3 generate_phrase.py
Copied!
and our python code:
1
# generate_phrase.py
2
​
3
import markovify
4
text = open('sherlock.txt').read()
5
text_model = markovify.Text(text)
6
print(text_model.make_sentence())
Copied!
Now we can run it with earthly +test, and we'll see a failure has occurred:
1
=========================== FAILURE ===========================
2
+test *failed* | --> RUN python3 generate_phrase.py
3
+test *failed* | Traceback (most recent call last):
4
+test *failed* | File "generate_phrase.py", line 3, in <module>
5
+test *failed* | text = open('sherlock.txt').read()
6
+test *failed* | FileNotFoundError: [Errno 2] No such file or directory: 'sherlock.txt'
7
+test *failed* | Command /bin/sh -c python3 generate_phrase.py failed with exit code 1
8
+test *failed* | +test *failed* | ERROR: Command exited with non-zero code: RUN python3 generate_phrase.py
9
Error: solve side effects: solve: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c /usr/bin/earth_debugger /bin/sh -c 'python3 generate_phrase.py']: buildkit-runc did not terminate successfully
Copied!
Why can't it find the sherlock.txt file? Let's re-run earthly with the --interactive (or -i) flag: earthly -i +test
This time we see a slightly different message:
1
+test | --> RUN python3 generate_phrase.py
2
+test | Traceback (most recent call last):
3
+test | File "generate_phrase.py", line 3, in <module>
4
+test | text = open('sherlock.txt').read()
5
+test | FileNotFoundError: [Errno 2] No such file or directory: 'sherlock.txt'
6
+test | Command /bin/sh -c python3 generate_phrase.py failed with exit code 1
7
+test | Entering interactive debugger (**Warning: only a single debugger per host is supported**)
8
+test | [email protected]:/code#
Copied!
This time rather than exiting, earthly will drop us into an interactive root shell within the container of the build environment. This root shell will allow us to execute arbitrary commands within the container to figure out the problem:
2
generate_phrase.py
3
[email protected]:/code# find / | grep sherlock.txt
4
/sherlock.txt
5
[email protected]:/code# ls /
6
bin boot code dev etc home lib lib64 media mnt opt proc root run sbin sherlock.txt srv sys tmp usr var
7
[email protected]:/code# ls /sherlock.txt
8
/sherlock.txt
Copied!
Ah ha! the corpus text file was located in the root directory rather than under /code. We can try moving it manually to see if that fixes the problem:
1
[email protected]:/code# mv /sherlock.txt /code/.
2
[email protected]:/code# python3 generate_phrase.py
3
I struck him down with the servants and with the lantern and left a fragment in the midst of my work during the last three years, although he has cruelly wronged.
Copied!
At this point we know what needs to be done to fix the test, so we can type exit (or ctrl-D), to exit the interactive shell.
1
+test | time="2020-09-16T22:23:53Z" level=error msg="failed to read from ptmx: read /dev/ptmx: input/output error"
2
+test | time="2020-09-16T22:23:53Z" level=error msg="failed to read data from conn: read tcp 127.0.0.1:36672->127.0.0.1:5000: use of closed network connection"
3
+test | ERROR: Command exited with non-zero code: RUN python3 generate_phrase.py
Copied!
Note that even though we fixed the problem during debugging, the image will not have been saved, so we must go back to our Earthfile and fix the problem there:
1
# Earthfile
2
​
3
FROM python:3
4
WORKDIR /code
5
​
6
test:
7
RUN curl https://raw.githubusercontent.com/jsvine/markovify/master/test/texts/sherlock.txt > /code/sherlock.txt
8
COPY generate_phrase.py .
9
RUN pip3 install markovify
10
RUN python3 generate_phrase.py
Copied!

Debugging integration tests

Let's consider a more complicated example where we are running integration tests within an embedded docker setup:
1
# Earthfile
2
​
3
server:
4
COPY server.py .
5
​
6
test:
7
FROM docker:19.03.12-dind
8
RUN apk add curl
9
WITH DOCKER --load server:latest=+server
10
RUN docker run --rm -d --network=host server:latest python3 server.py && sleep 5 && curl -s localhost:8000 | grep hello
11
END
Copied!
and our server.py code:
1
from http.server import HTTPServer, BaseHTTPRequestHandler
2
​
3
class SimpleHTTPRequestHandler(BaseHTTPRequestHandler):
4
def do_GET(self):
5
self.send_response(200)
6
self.end_headers()
7
self.wfile.write(b'Hello, world!')
8
​
9
httpd = HTTPServer(('localhost', 8000), SimpleHTTPRequestHandler)
10
httpd.serve_forever()
Copied!
Let's fire up our integration test with earthly -P -i +test:
1
buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
2
+base | --> FROM python:3
3
context | --> local context .
4
+base | resolve docker.io/library/python:[email protected]:e9b7e3b4e9569808066c5901b8a9ad315a9f14ae8d3949ece22ae339fff2cad0 100%
5
context | transferring .: 100%
6
+base | *cached* --> WORKDIR /code
7
+server | *cached* --> COPY server.py .
8
+test | --> FROM docker:19.03.12-dind
9
+test | resolve docker.io/library/docker:[email protected]:674f1f40ff7c8ac14f5d8b6b28d8fb1f182647ff75304d018003f1e21a0d8771 100%
10
+test | *cached* --> RUN apk add curl
11
+test | --> WITH DOCKER RUN docker run --rm -d --network=host server:latest python3 server.py && sleep 5 && curl -s localhost:8000 | grep hello
12
+test | Loading images...
13
+test | Loaded image: server:latest
14
+test | ...done
15
+test | 1dc054c647cb75bde4897a2828edb095739cb9f864ed203ed2ddb54e62554aad
16
+test | Command /bin/sh -c docker run --rm -d --network=host server:latest python3 server.py && sleep 5 && curl -s localhost:8000 | grep hello failed with exit code 1
17
+test | Entering interactive debugger (**Warning: only a single debugger per host is supported**)
Copied!
There was a failure checking that the server output contained the string hello; let's see what is going on:
1
/ # docker ps -a
2
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3
b8a31c54dd17 server:latest "python3 server.py" 5 seconds ago Up 4 seconds frosty_rhodes
Copied!
The good news is our server container is running; let's see what happens when we try to connect to it:
1
/ # curl -s localhost:8000
2
Hello, world!/
Copied!
Ah ha! The problem is our test is expecting a lowercase h, so we can fix our grep to look for an uppercase H:
1
# Earthfile
2
​
3
server:
4
COPY server.py .
5
​
6
test:
7
FROM docker:19.03.12-dind
8
RUN apk add curl
9
WITH DOCKER --load server:latest=+server
10
RUN docker run --rm -d --network=host server:latest python3 server.py && sleep 5 && curl -s localhost:8000 | grep Hello
11
END
Copied!
Then when we re-run our test we get:
1
+test | --> WITH DOCKER RUN docker run --rm -d --network=host server:latest python3 server.py && sleep 5 && curl -s localhost:8000 | grep Hello
2
+test | Loading images...
3
+test | Loaded image: server:latest
4
+test | ...done
5
+test | cb5299ae03cd17cfb2b528f01268ccf59761feec036cb313a3e969930d6f0815
6
+test | Hello, world!
7
+test | Target +test built successfully
8
=========================== SUCCESS ===========================
Copied!
With the use of the interactive debugger; we were able to examine the state of the embedded containerized

Demo

​​
​
​

Final tips

If you ever want to jump into an interactive debugging session at any point in your Earthfile, you can simply add a command that will fail such as:
1
RUN false
Copied!
and run earthly with the --interactive (or -i) flag.
Hopefully you won't run into failures, but if you do the interactive debugger may help you discover the root cause more easily. Happy coding.
Last modified 10mo ago