Part 6: Using Docker In Earthly

To copy the files for this example ( Part 6 ) run

earthly --artifact github.com/earthly/earthly/examples/tutorial/go:main+part6/part6 ./part6

Examples in Python, JavaScript and Java are at the bottom of this page.

The WITH DOCKER Command

You may find that you need to run Docker commands inside of a target. For those cases Earthly offers WITH DOCKER. WITH DOCKER will initialize a Docker daemon that can be used in the context of a RUN command.

Whenever you need to use WITH DOCKER we recommend (though it is not required) that you use Earthly's own Docker in Docker (dind) image: earthly/dind:alpine.

Notice WITH DOCKER creates a block of code that has an END keyword. Everything that happens within this block is going to take place within our earthly/dind:alpine container.

Pulling an Image

hello:
    FROM earthly/dind:alpine
    WITH DOCKER --pull hello-world
        RUN docker run hello-world
    END

You can see in the command above that we can pass a flag to WITH DOCKER telling it to pull an image from Docker Hub. We can pass other flags to load in artifacts built by other targets --load or even images defined by docker-compose --compose. These images will be available within the context of WITH DOCKER's docker daemon.

Loading an Image

We can load in an image created by another target with the --load flag.

my-hello-world:
    FROM ubuntu
    CMD echo "hello world"
    SAVE IMAGE my-hello:latest

hello:
    FROM earthly/dind:alpine
    WITH DOCKER --load hello:latest=+my-hello-world
        RUN docker run hello:latest
    END

A Real World Example

One common use case for WITH DOCKER is running integration tests that require other services. In this case we need to set up a redis service for our tests. For this we can user a docker-compose.yml.

docker-compose.yml

version: "3"

services:
  redis:
    container_name: local-redis
    image: redis:6.0-alpine
    ports:
      - 127.0.0.1:6379:6379
    hostname: redis
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:6379"]
      interval: 1s
      timeout: 10s
      retries: 5
    networks:
      - go/part6_default

networks:
  go/part6_default:

main.go

package main

import (
	"github.com/sirupsen/logrus"
)

var howCoolIsEarthly = "IceCool"

func main() {
	logrus.Info("hello world")
}

main_integration_test.go

package main

import (
	"context"
	"testing"

	"github.com/go-redis/redis/v8"
	"github.com/stretchr/testify/require"
)

func TestIntegration(t *testing.T) {
	ctx := context.Background()
	rdb := redis.NewClient(&redis.Options{
		Addr:     "redis:6379",
		Password: "", // no password set
		DB:       0,  // use default DB
	})

	err := rdb.Set(ctx, "howCoolIsEarthly", howCoolIsEarthly, 0).Err()
	if err != nil {
		panic(err)
	}

	resultFromDB, err := rdb.Get(ctx, "howCoolIsEarthly").Result()
	if err != nil {
		panic(err)
	}
	require.Equal(t, howCoolIsEarthly, resultFromDB)
}

VERSION 0.6
FROM golang:1.15-alpine3.13
WORKDIR /go-workdir

deps:
    COPY go.mod go.sum ./
    RUN go mod download
    SAVE ARTIFACT go.mod AS LOCAL go.mod
    SAVE ARTIFACT go.sum AS LOCAL go.sum

test-setup:
    FROM +deps
    COPY main.go .
    COPY main_integration_test.go .
    ENV CGO_ENABLED=0
    ENTRYPOINT [ "go", "test", "github.com/earthly/earthly/examples/go"]
    SAVE IMAGE test:latest

integration-tests:
    FROM earthly/dind:alpine
    COPY docker-compose.yml ./
    WITH DOCKER --compose docker-compose.yml --load tests:latest=+test-setup
        RUN docker run --network=default_go/part6_default tests:latest
    END

When we use the --compose flag, Earthly will start up the services defined in the docker-compose file for us. In this case, we built a separate image that copies in our test files and uses the command to run the tests as its ENTRYPOINT. We can then load this image into our WITH DOCKER command. Note that loading an image will not run it by default, we need to explicitly run the image after we load it.

You'll need to use --allow-privileged (or -P for short) to run this example.

earthly --allow-privileged +integration-tests

More Examples

JavaScript

To copy the files for this example ( Part 6 ) run

earthly --artifact github.com/earthly/earthly/examples/tutorial/js:main+part6/part6 ./part6

In this example, we use WITH DOCKER to run a frontend app and backend api together using Earthly.

The App

./app/package.json

{
  "name": "example-js",
  "version": "0.0.1",
  "description": "Hello world",
  "private": true,
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "MPL-2.0",
  "devDependencies": {
    "webpack": "^4.42.1",
    "webpack-cli": "^3.3.11"
  },
  "dependencies": {
    "http-server": "^0.12.1"
  }
}

./app/package-lock.json (empty)

The code of the app might look like this

./app/src/index.js

async function getUsers() {

  const response = await fetch('http://0.0.0.0:3080/api/users');
  return await response.json();

}

function component() {
  const element = document.createElement('div');
  getUsers()
    .then( users => {
      element.innerHTML = `hello world <b>${users[0].first_name} ${users[0].last_name}</b>`
    })

	return element;
}

document.body.appendChild(component());

./app/src/index.html

<!doctype html>
<html>

<head>
    <title>Getting Started</title>
</head>

<body>
    <script src="./main.js"></script>
</body>

</html>

And our api.

./api/package.json

{
  "name": "api",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "cors": "^2.8.5",
    "express": "^4.17.1",
    "http-proxy-middleware": "^1.0.4",
    "pg": "^8.7.3"
  }
}

./api/package-lock.json (empty)

./api/server.js

const express = require('express');
const path = require('path');
const cors = require("cors");
const app = express(),
bodyParser = require("body-parser");
port = 3080;

app.use(bodyParser.json());
app.use(express.static(path.join(__dirname, '../my-app/build')));

app.use(cors());

const users = [
  {
    'first_name': 'Lee',
    'last_name' : 'Earth'
  }
]

app.get('/api/users', (req, res) => {
  console.log('api/users called!')
  res.json(users);
});

app.listen(port, '0.0.0.0', () => {
  console.log(`Server listening on the port::${port}`);
});

The Earthfile is at the root of the directory.

./Earthfile

VERSION 0.6
FROM node:13.10.1-alpine3.11
WORKDIR /js-example

app-deps:
    COPY ./app/package.json ./
    COPY ./app/package-lock.json ./
    RUN npm install
    # Output these back in case npm install changes them.
    SAVE ARTIFACT package.json AS LOCAL ./app/package.json
    SAVE ARTIFACT package-lock.json AS LOCAL ./app/package-lock.json

build-app:
    FROM +app-deps
    COPY ./app/src ./app/src
    RUN mkdir -p ./app/dist && cp ./app/src/index.html ./app/dist/
    RUN cd ./app && npx webpack
    SAVE ARTIFACT ./app/dist /dist AS LOCAL ./app/dist

app-docker:
    FROM +app-deps
    ARG tag='latest'
    COPY +build-app/dist ./app/dist
    EXPOSE 8080
    ENTRYPOINT ["/js-example/node_modules/http-server/bin/http-server", "./app/dist"]
    SAVE IMAGE js-example:$tag

api-deps:
    COPY ./api/package.json ./
    COPY ./api/package-lock.json ./
    RUN npm install
    # Output these back in case npm install changes them.
    SAVE ARTIFACT package.json AS LOCAL ./api/package.json
    SAVE ARTIFACT package-lock.json AS LOCAL ./api/package-lock.json

api-docker:
    FROM +api-deps
    ARG tag='latest'
    COPY ./api/server.js .
    RUN pwd
    RUN ls
    EXPOSE 3080
    ENTRYPOINT ["node", "server.js"]
    SAVE IMAGE js-api:$tag

# Run your app and api side by side
app-with-api:
    FROM earthly/dind:alpine
    RUN apk add curl
    WITH DOCKER \
        --load app:latest=+app-docker \
        --load api:latest=+api-docker
        RUN docker run -d -p 3080:3080 api && \
            docker run -d -p 8080:8080 app  && \
            sleep 5 && \
            curl 0.0.0.0:8080 | grep 'Getting Started' && \
            curl 0.0.0.0:3080/api/users | grep 'Earth'
    END

Now you can run earthly -P +app-with-api to run the app and api side-by-side.

Java

To copy the files for this example ( Part 6 ) run

mkdir tutorial
cd tutorial
earthly --artifact github.com/earthly/earthly/examples/tutorial/java:main+part6/part6 ./part6

./Earthfile

VERSION 0.6
FROM openjdk:8-jdk-alpine
RUN apk add --update --no-cache gradle
WORKDIR /java-example

deps:
    COPY build.gradle ./
    RUN gradle build

build:
    FROM +deps
    COPY src src
    RUN gradle build
    RUN gradle install
    SAVE ARTIFACT build/install/java-example/bin AS LOCAL build/bin
    SAVE ARTIFACT build/install/java-example/lib AS LOCAL build/lib

docker:
    COPY +build/bin bin
    COPY +build/lib lib
    ARG tag='latest'
    ENTRYPOINT ["/java-example/bin/java-example"]
    SAVE IMAGE java-example:$tag

with-postgresql:
    FROM earthly/dind:alpine
	  COPY ./docker-compose.yml .
	  RUN apk update
	  RUN apk add postgresql-client
	  WITH DOCKER --compose docker-compose.yml --load app:latest=+docker
		    RUN while ! pg_isready --host=localhost --port=5432; do sleep 1; done ;\
			    docker run --network=default_java/part6_default app
	  END

docker-compose.yml

version: "3.9"
   
services:
  db:
    image: postgres
    container_name: db
    hostname: postgres
    environment:
      - POSTGRES_DB=test_db
      - POSTGRES_USER=earthly
      - POSTGRES_PASSWORD=password
    ports:
      - 127.0.0.1:5432:5432
    networks:
      - java/part6_default

networks:
  java/part6_default:

The code of the app might look like this

./src/main/java/hello/HelloWorld.java

package postgresclient;

import org.joda.time.LocalTime;
import java.sql.Connection;
import java.sql.DriverManager;


public class PostgreSQLJDBC {
   public static void main(String args[]) {
      Connection c = null;
      try {
         Class.forName("org.postgresql.Driver");
         c = DriverManager
            .getConnection("jdbc:postgresql://postgres:5432/test_db",
            "earthly", "password");
      } catch (Exception e) {
         e.printStackTrace();
         System.err.println(e.getClass().getName()+": "+e.getMessage());
         System.exit(0);
      }
      System.out.println("Opened database successfully");
   }
}

./build.gradle

apply plugin: 'java'
apply plugin: 'application'

mainClassName = 'postgresclient.PostgreSQLJDBC'

repositories {
    mavenCentral()
}

jar {
    baseName = 'hello-world'
    version = '0.0.1'
}

sourceCompatibility = 1.8
targetCompatibility = 1.8

dependencies {
    compile "joda-time:joda-time:2.2"
    compile(group: 'org.postgresql', name: 'postgresql', version: '42.3.3')
    testCompile "junit:junit:4.12"
}

Python

To copy the files for this example ( Part 7 ) run

earthly --artifact github.com/earthly/earthly/examples/tutorial/python:main+part7/part7 ./part7

./tests/test_db_connection.py

import unittest
import psycopg2

class MyIntegrationTests(unittest.TestCase):

    def test_db_connection_active(self):
        connection = psycopg2.connect(
            host="postgres",
            database="test_db",
            user="earthly",
            password="password")
        
        self.assertEqual(connection.closed, 0)

if __name__ == '__main__':
    unittest.main()

version: "3.9"
   
services:
  db:
    image: postgres
    container_name: db
    hostname: postgres
    environment:
      - POSTGRES_DB=test_db
      - POSTGRES_USER=earthly
      - POSTGRES_PASSWORD=password
    ports:
      - 5432:5432
    networks:
      - python/part6_default

networks:
  python/part6_default:

./Earthfile

VERSION 0.6
FROM python:3
WORKDIR /code

build:
  COPY ./requirements.txt .
  RUN pip install -r requirements.txt
  COPY . .

run-tests:
    FROM earthly/dind:alpine
    COPY ./docker-compose.yml .
    COPY ./tests ./tests
    RUN apk update
    RUN apk add postgresql-client
    WITH DOCKER --compose docker-compose.yml --load app:latest=+docker
        RUN while ! pg_isready --host=localhost --port=5432; do sleep 1; done ;\
          docker run --network=default_python/part6_default app python3 ./tests/test_db_connection.py
    END