No space left on device on a CI runner

The runner's disk filled. Where the space went depends heavily on whether the runner is hosted or self-hosted, so measure before changing anything.

Add this as the first step of the failing job:

- name: Disk before
  run: |
    df -h /
    docker system df || true
    du -sh /home/runner/work 2>/dev/null || true

Hosted runners: the toolchain is the problem

A GitHub-hosted ubuntu-latest runner provides roughly 14 GB free on a 75 GB disk, because the image ships with a very large set of preinstalled software. Most jobs use almost none of it.

- name: Free disk space
  run: |
    sudo rm -rf /usr/share/dotnet          # ~20 GB
    sudo rm -rf /usr/local/lib/android     # ~12 GB
    sudo rm -rf /opt/ghc                   # ~5 GB
    sudo rm -rf /usr/local/share/boost
    sudo rm -rf "$AGENT_TOOLSDIRECTORY"
    df -h /

That reclaims 30 GB or more in about twenty seconds, and it is the single most effective fix for a Docker build that runs out of space on a hosted runner.

The jlumbroso/free-disk-space action packages the same idea with flags, if you prefer not to maintain the list.

Only delete what you do not need. Removing $AGENT_TOOLSDIRECTORY breaks actions/setup-node and its siblings, so do it only in jobs that do not use them.

Docker layers

docker system df

TYPE            TOTAL   ACTIVE  SIZE      RECLAIMABLE
Images          24      3       18.2GB    14.1GB
Build Cache     412     0       9.8GB     9.8GB

Build Cache is frequently the largest item and the least visible. BuildKit keeps every intermediate layer.

On an ephemeral runner, prune freely:

docker system prune -af --volumes
docker builder prune -af

On a self-hosted runner this is destructive: it deletes caches other jobs depend on, and the next few builds are slow. Prune by age instead:

docker builder prune -f --filter until=24h
docker image prune -af --filter until=72h

Better, cap the cache so it never needs manual attention:

# /etc/gitlab-runner/config.toml
[[runners]]
  [runners.docker]
    # ...

// /etc/docker/daemon.json on the runner host
{
  "builder": { "gc": { "enabled": true, "defaultKeepStorage": "20GB" } }
}

BuildKit then garbage-collects itself and the problem stops recurring.

Build caches in the job

Package manager caches restored from a CI cache action count against the disk too, and they grow monotonically unless keyed properly:

- uses: actions/cache@v4
  with:
    path: ~/.npm
    key: npm-${{ hashFiles('package-lock.json') }}
    restore-keys: npm-

A cache keyed on a lockfile hash is bounded. One keyed on npm-${{ github.sha }} creates a new entry per commit and, on a self-hosted runner with a local cache directory, fills the disk over weeks.

GitLab has an explicit limit:

cache:
  key:
    files:
      - package-lock.json
  paths:
    - .npm/
  policy: pull-push

policy: pull on jobs that only read the cache avoids re-uploading it from every parallel job.

Self-hosted runners: what accumulates

Four things, in order:

du -sh /var/lib/docker /home/gitlab-runner/builds /home/gitlab-runner/cache /var/log

Build directories. GitLab keeps one per project per concurrent slot. Enable cleanup:

variables:
  GIT_STRATEGY: fetch
  GIT_CLEAN_FLAGS: -ffdx

The runner cache directory. Nothing expires it by default. A cron job is the usual answer:

find /home/gitlab-runner/cache -type f -mtime +7 -delete

Docker volumes from containers that were never removed.

Logs. Journald with no cap:

sudo journalctl --vacuum-size=500M

Add a monitoring alert on disk usage for self-hosted runners. Discovering it through a failed build is the expensive way.

Larger runners

If you genuinely need more, both platforms sell it. GitHub's larger runner classes come with more disk, and self-hosted runners have whatever you give them. That is worth doing for genuinely large builds and is not a substitute for bounded caches, since an unbounded cache fills any disk eventually.

A checklist

1. Add df -h / and docker system df as the first step. Measure first.
2. Hosted runner → delete /usr/share/dotnet, /usr/local/lib/android, /opt/ghc.
3. Do not delete $AGENT_TOOLSDIRECTORY in jobs that use setup-* actions.
4. docker system df and look at Build Cache specifically.
5. Ephemeral runner → docker system prune -af --volumes freely.
6. Self-hosted → prune by age, and set BuildKit defaultKeepStorage.
7. Key caches on a lockfile hash, never on the commit sha.
8. Self-hosted → cron the cache directory, vacuum journald, alert on disk usage.

---

Frequently Asked Questions

Why does a hosted runner run out of disk so quickly?

Because the image ships with a very large preinstalled toolchain, leaving only around 14 GB free on a 75 GB disk. Android SDKs, .NET, GHC, multiple language runtimes and a set of cached Docker images occupy most of it before your job starts. A Docker build pulling a few large base images can exhaust the remainder easily. Deleting the toolchains your job does not use reclaims 30 GB or more in about twenty seconds and is usually the whole fix.

Is docker system prune safe in CI?

On an ephemeral hosted runner, yes, since the machine is destroyed after the job and nothing is shared. On a self-hosted runner it is destructive: -a removes every image not currently in use and --volumes removes named volumes, so other jobs lose their caches and the next several builds are slow. There, prune by age with --filter until=24h, or better, configure BuildKit garbage collection with defaultKeepStorage so the cache stays bounded automatically.

Why does my self-hosted runner's disk fill up over weeks?

Four things accumulate and none of them expires by default: Docker images and build cache, per-project build directories, the runner's cache directory, and system logs. The cache directory in particular grows forever if your cache key includes the commit sha, since every commit creates a new entry. Cap the Docker builder, set GIT_CLEAN_FLAGS: -ffdx, add a cron job to delete cache files older than a week, and vacuum journald.

How should I key a CI cache so it does not grow without bound?

On a hash of the dependency lockfile, for example npm-${{ hashFiles('package-lock.json') }}, with a restore-keys prefix for partial matches. That produces a new cache entry only when dependencies actually change. Keying on the commit sha creates one per commit, which on hosted runners hits the platform's cache quota and on self-hosted runners fills the local disk. Also use a read-only cache policy on jobs that do not modify dependencies.

Will a larger runner fix this?

It buys headroom and does not fix an unbounded cache, which will fill any disk given enough time. Larger runner classes are genuinely worth it for builds with large images or heavy compilation, where even a clean runner is tight. Treat it as capacity planning rather than a remedy: apply the cache bounds and cleanup first, then size the runner to what the job actually needs.