Pushing Earthly to the Limit
In every project, setting up CI pipelines in a way that is easy to build, understand, and test locally is a constant struggle. Time to test a tool that promises to solve that: Earthly.
Works on My Machine... But Not in CI
Setting up a reliable CI pipeline is often more complicated than it should be. Differences in environments and inconsistencies between local development and CI make workflows difficult to maintain and debug. A common struggle is ensuring that the same checks run everywhere—locally, in CI, and across different developers’ machines—without duplicating logic or encountering subtle execution differences. Too often, something that works on one developer’s machine fails in CI, or behaves differently on another developer’s machine, leading to unnecessary debugging and frustration.
Trying Earthly with Rust
This issue was especially noticeable in a Rust crate of mine, where I needed to run checks using different Rust compiler versions—stable, beta, and nightly. Switching between them locally was cumbersome, and remembering which version was required for each check became frustrating.
I had previously used tools like Just to define common tasks in my projects, but those still ran on the host system, meaning they weren’t isolated or fully reproducible. I wanted a tool that could run these checks in a controlled environment, and that’s when I came across Earthly.
My first Earthfile was quite straightforward, implementing the key checks I typically run in my projects. Below are two of the checks that kickstarted this journey—one using the beta toolchain, the other running on nightly. By running these with Earthly, I no longer had to worry about disrupting my local development environment.
deps-latest:
FROM +sources
# Switch to beta toolchain
RUN rustup default beta
# Update the dependencies to the latest versions
RUN cargo update
# Run tests to ensure the latest versions are compatible
RUN cargo test --all-features --all-targets --locked
deps-minimal:
FROM +sources
# Switch to nightly toolchain
RUN rustup default nightly
# Set minimal versions for dependencies
RUN cargo update -Z direct-minimal-versions
# Run tests to ensure the minimal versions are compatible
RUN cargo test --all-features --all-targets --lockedThe next step was integration these checks into GitHub Actions.
jobs:
minimal:
name: Test minimal versions
runs-on: ubuntu-latest
needs: detect-changes
if: needs.detect-changes.outputs.any_changed == 'true'
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Install earthly
uses: earthly/actions-setup@v1
with:
version: 0.8
- name: Run tests with minimal versions
run: earthly --ci +deps-minimalThis felt like a massive improvement over my previous workflow. Now, I could run these checks with a single command in an isolated environment, whether locally or in CI, and always get the same result.
Moving All Checks into Earthly
After getting comfortable with Earthly and seeing success with the Rust checks, I decided to migrate all other checks in my projects into the Earthfile. Here’s an example of how I handled JSON and Markdown formatting:
all:
BUILD +json-format
BUILD +markdown-format
BUILD +markdown-lint
BUILD +rust-deps-latest
BUILD +rust-deps-minimal
BUILD +rust-doc
BUILD +rust-features
BUILD +rust-format
BUILD +rust-lint
BUILD +rust-msrv
BUILD +rust-test
BUILD +yaml-format
BUILD +yaml-lint
prettier-container:
FROM node:alpine
WORKDIR /typed-fields
# Install prettier
RUN npm install -g prettier
# Copy the source code into the container
COPY . .
json-format:
FROM +prettier-container
# Check the JSON formatting
RUN prettier --check **/*.json
markdown-format:
FROM +prettier-container
# Check the Markdown formatting
RUN prettier --check **/*.mdBy the end, the Earthfile contained 13 different checks, all of which could be executed with a single command:
earthly +allThese checks covered code formatting, linting for common file types, and various Rust-related tasks. Just like in the first step, I also integrated them into GitHub Actions, making it possible—for the first time—to fully reproduce CI locally. 🥳
Trying Earthly Satellites
While migrating checks to GitHub Actions, I ran into a major performance issue with the larger Rust checks. Previously, I had used various Actions and some caching tricks to cache intermediary Rust build artifacts between runs. Earthly, on the other hand, is blazingly fast locally due to its built-in caching—but on GitHub’s ephemeral infrastructure, it rebuilt everything from scratch every time.
Earthly's solution to this problem are Earthly Satellites, remote builders that persist a cache between builds. Earthly advertises a 2-20x speed improvement, and their free plan includes 6000 minutes, which seemed more than enough for my small project. At first, this sounded like a perfect solution. But in practice, I ran into two major problems that made Earthly Satellites unsuitable for my workflow.
The first issue was the size of the free Earthly runners, which were significantly smaller than GitHub’s free runners. If I remember correctly, Earthly’s xsmall instance—the default for free users—has only 2 vCPUs and 4GB of RAM, whereas GitHub’s free runners offer 4 vCPUs and 16GB of RAM. Rust benefits tremendously from more memory, and the limited resources of the Earthly runners meant I could never match GitHub’s performance.
The free plan does allow an upgrade to small instances, which doubles the CPU and RAM, but that also cuts the free build minutes in half. This tradeoff made it difficult to justify using Earthly Satellites, especially when GitHub’s free plan already provided better performance with no limitations.
(Earthly seems to have removed the instance size details from their website, so the hardware numbers are based on my memory. 🤷♂️)
The second issue was Earthly’s GitHub Actions integration, which is only available on paid plans. On the free plan, I could trigger an Earthly check from a GitHub Actions runner and have it execute remotely on an Earthly Satellite. But this setup introduced an extra cost: the GitHub Actions runner had to sit idle and wait for the Earthly Satellite to return a result. Since GitHub’s free plan gives me access to runners at no cost, this effectively meant I was “paying” for both a GitHub Actions runner and an Earthly Satellite just to run the same check.
- name: Run tests with test coverage
env:
EARTHLY_TOKEN: ${{ secrets.EARTHLY_TOKEN }}
run: earthly --ci --org jdno --sat small-amd64-1 +allI can imagine that Earthly Satellites are a great option for closed-source projects that already have to pay for GitHub Actions runners or for teams that need larger machines than the free plans provide. But for my needs, GitHub’s free open-source CI offering is unbeatable. It gave me faster builds at no cost without requiring any additional setup.
Caching using GitHub Container Registry
Even after switching back to GitHub’s free runners, I still had the problem of long, uncached builds. Earthly’s local caching made things fast on my machine, but every new CI run started from scratch, leading to unnecessary rebuilds.
After digging through Earthly’s documentation—and quite a few GitHub issues—I finally found the feature I needed: remote caching. Interestingly, it wasn’t in the latest documentation but in the docs for a previous version of Earthly.
As it turns out, setting up GitHub’s Container Registry (GHCR) as a remote cache for Earthly is actually quite simple. All it required was granting access to the registry within the CI job and passing the --remote-cache argument:
rust-test:
name: Run tests
runs-on: ubuntu-latest
permissions:
packages: write
contents: read
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Install earthly
uses: earthly/actions-setup@v1
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
version: 0.8
- name: Run tests with test coverage
env:
EARTHLY_TOKEN: ${{ secrets.EARTHLY_TOKEN }}
run: |
earthly \
--ci \
--push \
--remote-cache=ghcr.io/jdno/typed-fields:cache-run-tests \
+rust-testThe impact was immediate and significant. The first build—without caching—took around 4 minutes to complete. But once the cache was warm, the same build finished in just 33 seconds. Earthly was now correctly detecting which steps actually needed to be re-run, leading to huge efficiency gains.
And this wasn’t just a small-project benefit—larger projects saw an even bigger impact. Build times that previously took 15-20 minutes were reduced to around 2 minutes on average. This was the 2-20x speed improvement that Earthly had promised!
Refactoring using Earthly Functions
As my Earthfile grew, I started looking for ways to reduce duplication and keep the code manageable. Initially, I relied on different targets to share setup steps. For example:
prettier-container:
FROM node:alpine
WORKDIR /typed-fields
# Install prettier
RUN npm install -g prettier
# Copy the source code into the container
COPY . .
json-format:
FROM +prettier-container
# Check the JSON formatting
RUN prettier --check **/*.jsonIn this example, the +json-format target inherits the setup from the +prettier-container target. This pattern worked well for avoiding repeated setup steps across multiple targets, like +markdown-format or +yaml-format.
However, this approach only works for early-stage setup—it’s not well-suited for sharing steps that happen later in the pipeline. That’s where functions come in.
One great example is the COPY . . directive. Every target needs to copy the source code into the container, but for optimal caching, this should happen after the environment is prepared. Different targets might use different base images, making it impossible to share a single setup target. The solution? Extract it into a function:
COPY_SOURCES:
FUNCTION
# Copy the source code into the container
COPY . .
prettier-container:
FROM node:alpine
WORKDIR /typed-fields
# Install prettier
RUN npm install -g prettier
# Copy the source code into the container
DO +COPY_SOURCESThe COPY_SOURCES function is so simple that I technically don’t need it. However, I’m still experimenting with ways to copy only the relevant files into the container to optimize caching. Having the ability to centralize logic like this makes evolving my CI pipelines much easier.
Breaking Up the Monolithic Earthfile
At this point, my Earthfile had grown to over 200 lines, containing 2 functions and around 20 targets. I tried to keep it concise, but for what was supposed to be a “simple” project, it was becoming increasingly difficult to manage.
One of the biggest challenges I encountered with Earthly was how imports and build contexts interact. Before diving into the problem, let me first share the solution I came up with. Here’s a heavily redacted version of my repository structure:
tree -a -L 3 .
.
├── .earthly
│ ├── markdown
│ │ └── Earthfile
│ ├── prettier
│ │ └── Earthfile
│ ├── rust
│ │ └── Earthfile
│ ├── toml
│ │ └── Earthfile
│ └── yaml
│ └── Earthfile
├── .earthlyignore
└── EarthfileI ended up creating a .earthly directory, with subdirectories for the different tech stacks I was testing. Each subdirectory contained its own Earthfile, which implemented the necessary checks using functions. The top-level Earthfile became nothing more than a thin wrapper that called these functions:
format-markdown:
ARG FIX="false"
DO ./.earthly/prettier+PRETTIER --EXTENSION="md" --FIX="$FIX"While this approach felt like a bit of a hack, it significantly improved readability and maintainability. The top-level Earthfile became much shorter, making it easier to see which checks were available and what arguments they accepted. Meanwhile, the individual Earthfile allowed me to add more functionality and documentation without worrying about bloating a single file.
Beyond just improving this project, I see this as the first step toward reusing Earthly functions across multiple projects. I’m currently working on extracting these Earthfile into a separate repository, allowing me to share them across all my projects without duplicating code.
Making CI Fully Dynamic
At this point, I could have stopped and called it quits. But I had two more ideas I wanted to test—starting with reducing duplication in my GitHub Actions setup.
Previously, I split my workflows by tech stack to define granular rules for when to run each workflow. Each workflow would first check the Git diff to determine whether any relevant files had changed, and only then would it execute the necessary checks. This approach worked well and saved me both time and money over the years:
$ tree .github/workflows
.github/workflows
├── json.yml
├── markdown.yml
├── rust.yml
└── yaml.ymlHowever, Earthly’s caching made this optimization unnecessary. Since Earthly automatically skips unnecessary steps, I realized I could merge all checks into a single ci.yml workflow while removing as much duplication as possible.
I started by renaming my Earthly targets to follow a consistent naming convention:
check-*run general validation tasksformat-*run code formatterslint-*run linterstest-*run test suites
Then, I used earthly ls to list all available targets and jq to filter out only the ones that matched the above prefixes:
$ earthly ls | jq -nR '[inputs | sub("\\+"; "") | select(startswith("check-") or startswith("format-") or startswith("lint-") or startswith("test-"))]'
[
"check-docs",
"check-features",
"check-latest-deps",
"check-minimal-deps",
"check-msrv",
"format-json",
"format-markdown",
"format-rust",
"format-toml",
"format-yaml",
"lint-markdown",
"lint-rust",
"lint-yaml",
"test-rust"
]
On GitHub Actions, I then fed this list into a matrix strategy, allowing me to dynamically execute the relevant checks:
strategy:
fail-fast: false
matrix:
target: ${{ fromJson(needs.targets.outputs.matrix) }}By using a matrix strategy, I was able to eliminate all workflow duplication and create a single workflow that automatically runs any target I add to the Earthfile—without requiring manual updates.
Earthly in Pre-Commit Hooks
At this point, adopting Earthly felt like a huge improvement over my previous setup. I could now run the same checks both locally and in GitHub Actions, making it trivially simple to reproduce issues and share complex tasks with a team. However, there was still one part of my development workflow that remained outside of Earthly: pre-commit hooks.
I’ve always liked pre-commit as a tool for managing hooks and already had it set up in this project. Fortunately, there’s an integration that makes it easy to run Earthly as a pre-commit hook:
- repo: https://github.com/hongkongkiwi/earthly-precommit
rev: v0.0.5
hooks:
- id: earthly-target
name: run Earthly checks
args: ["./Earthfile", "+pre-commit", "--FIX=true"]
files: .*In my Earthfile, I created a new +pre-commit target that runs a small subset of checks before every commit:
# These targets get executed by pre-commit before every commit. Some need to be
# run sequentially to avoid overwriting each other's changes.
pre-commit:
WAIT
BUILD +prettier
END
WAIT
BUILD +format-toml
END
BUILD +format-rust
BUILD +lint-markdown
BUILD +lint-rust
BUILD +lint-yamlEarthly executes targets in an isolated environment, but it can write changes back to the local filesystem. The --FIX=true flag enables this feature, allowing pre-commit hooks to automatically fix as many issues as possible.
To prevent different formatting checks from overwriting each other’s changes, the +prettier and +format-toml targets are run sequentially using WAIT blocks.
While running pre-commit hooks with Earthly takes a few seconds longer than before, due to the overhead of spinning up a Docker container and copying project files, the tradeoff feels worth it. The consistency gained by running the exact same checks everywhere—locally, in CI, and across different machines—in my opinion far outweighs the slight performance hit.
Summary – Bringing It All Together
Migrating my CI/CD pipeline to Earthly has been a journey of continuous refinement and optimization. What started as an experiment to simplify Rust checks gradually evolved into a fully automated, highly reproducible, and maintainable system for running all my validation tasks—locally and in CI.
Adopting Earthly has dramatically improved my CI/CD workflow. Reproducibility, automation, and caching have made my builds more reliable and faster, while reducing manual work and duplication.
There are still areas to explore—such as refining caching even further or extracting my Earthly functions into a reusable repository—but for now, this setup feels like a massive leap forward from where I started.
However, I also have some issues with Earthly.
My biggest concern is its lack of offline support. Earthly simply does not work without an internet connection. A GitHub issue for offline support was opened in 2021, has received only five comments, and still shows no signs of progress. This doesn’t give me much confidence that offline support is coming anytime soon. (For the record, Dagger, a competing tool, has the same limitation.)
Beyond that, adopting Earthly required a lot of trial and error. While the official documentation is well-written, it mostly covers the “happy path”. When things go wrong, debugging can be difficult, as there aren’t many resources or community discussions available.
Finally, there’s the question of Earthly’s long-term direction. I understand that Earthly, the company, needs to monetize its open-source project. However, the approach of restricting free features to push users toward a paid product doesn’t inspire much confidence. And more recently, Earthly seems to have shifted focus towards compliance rather than CI/CD, which raises concerns about how well it will continue to align with my needs.
Would I use Earthly in other projects? Probably. The benefits outweigh the downsides for now. But I’ll also be keeping my eyes open for a similar tool that offers true offline support—because in a perfect world, a build tool should work anywhere, even without an internet connection.
If you’re curious about the implementation details, feel free to check out the code and browse through the Git history to see how it evolved over time:
jdno