Buildkite Cache

Buildkite Cache

Buildkite Cache is file-based caching built into the agent: buildkite-agent cache save and buildkite-agent cache restore archive and restore directories between builds, keyed by a structured cache key defined in .buildkite/cache.yml. This skill covers the configuration format, cache key and fallback semantics, the CLI commands, and storage setup.

Preview feature. Buildkite Cache is in preview and subject to change. On Buildkite hosted agents it works with zero storage configuration. For self-hosted agents the feature is opt-in — contact Buildkite support to enable it for the organization, and provide a cache store (see Storage below).

For the cache Buildkite plugin, the pipeline-level cache: key, and hosted agent cache volumes, see the buildkite-pipelines skill. Those are separate caching mechanisms; Buildkite Cache does not read the pipeline cache: attribute.

Quick Start

Define caches in .buildkite/cache.yml at the repository root:

caches:  - name: node    cache_key:      - node      - { agent: os }      - { agent: arch, fallback_limit: true }      - { checksum: package-lock.json }    target_paths:      - ~/.npm

Call the commands inside a job step — nothing runs automatically:

steps:  - label: ":nodejs: Test"    command: |      buildkite-agent cache restore      npm ci      buildkite-agent cache save      npm test

The cache holds npm's download cache (~/.npm), not node_modules — npm ci deletes node_modules before installing, so caching it directly is wasted work. The ~/.npm path is POSIX-only: on Windows agents npm caches to %LocalAppData%\npm-cache, so ~/.npm never exists and save fails — set npm_config_cache to a fixed path and cache that path instead. On the first build, restore reports a miss, npm ci downloads everything, and save uploads the populated ~/.npm. On later builds with an unchanged lockfile, restore is an exact hit and npm ci installs from the local cache without re-downloading. When the lockfile changes, the fallback_limit on arch lets restore fall back to the newest node + os + arch cache, so only new or changed packages are downloaded — then save uploads a fresh entry under the new checksum. The same principle applies to any tool that rebuilds its target from scratch: cache the tool's download or package cache, and cache an install directory like node_modules only when the install step preserves it.

Run cache save immediately after the step that produces the cached paths. Save checks for an existing entry first and skips the upload when the key already exists, so calling it on every build is cheap.

Cache Configuration File

The commands read .buildkite/cache.yml or .buildkite/cache.yaml (exactly one must exist — two is an error), or an explicit path from --cache-config-file / BUILDKITE_CACHE_CONFIG_FILE.

Each entry under caches: defines one named cache:

Cache key parts

cache_key is an ordered list. Each part is one of:

Checksum paths are resolved relative to the working directory and support *, **, and ? globs. All matched files are deduplicated, sorted, and hashed into a single digest, so the part changes when any matched file changes. A literal (non-glob) path must exist and be a regular file; a checksum whose patterns match no files at all is an error.

Include { agent: os } and { agent: arch } in any cache holding compiled or platform-specific content (native node modules, Go build caches, gems with extensions). Omit them only for caches that are genuinely platform-independent.

Target paths

target_paths may be relative to the working directory, absolute, or ~-prefixed (home directory). Archives store paths relative to portable anchors, so a cache saved under one checkout directory or home directory restores correctly under another. At save time every target path must exist. At restore time each target path is deleted and replaced by the cached content — never point target_paths at a directory containing uncached work.

Cache Keys and Fallback Matching

A cache entry is addressed by its target_paths set plus the resolved cache_key values, scoped to the registry. Registry policies can add further scopes (branch, build, or pipeline) to an entry's address and control which scope candidates restore considers — so identical paths and key values can intentionally miss across branches or pipelines under a scoped policy. Restore looks up entries in two stages:

1. Exact match — every key part matches. Reported as a cache hit.
2. Fallback match — only when a part sets fallback_limit: true. Every part up to and including the marked part stays mandatory; every part after it becomes optional. The newest entry matching the mandatory prefix is restored. Reported as fallback used.

With no fallback_limit, every part is mandatory and only exact matches restore. Restore reports one of three outcomes per cache: hit, fallback used, or miss. A miss is not an error and does not fail the build.

Place the volatile part (usually the checksum) last, after the fallback_limit marker:

cache_key:  - v1                                  # bump to invalidate all previous entries  - { agent: os }  - { agent: arch, fallback_limit: true }  - { checksum: Gemfile.lock }

This restores an exact match when the lockfile is unchanged, and otherwise the newest v1 + os + arch entry as a warm starting point. Because save skips existing entries, the entry saved under an unchanged key stays as-is; a changed checksum produces a new entry.

To invalidate caches after a toolchain change that the checksum does not capture, bump a literal version part (v1 → v2). Old entries expire automatically after a few days without exact-match restores.

CLI Commands

Both commands process every cache defined in the config file by default, in parallel, and print a collapsed --- :package: group in the job log.

# Restore all caches defined in .buildkite/cache.ymlbuildkite-agent cache restore
# Save only specific cachesbuildkite-agent cache save --name node --name gems

Shared flags:

Behavioral guarantees to rely on:

• Save skips existing entries (best-effort). If an entry already exists for the exact resolved key, save reports "Cache already exists" and skips the upload. New content requires a new key (normally via a checksum part). The existence check is not atomic with the write: concurrent saves under the same key can both observe a miss and both upload, and the later commit wins — do not rely on first-writer-wins across parallel builds.
• Misses and corrupt entries never fail the build. A missing entry, a missing archive, a digest mismatch, or an unreadable archive degrades to a cache miss, and stale metadata is invalidated automatically. Other failures — invalid configuration, registry/API errors, store permission or network problems — are real errors and do fail the command. Write steps so they work from an empty cache.
• A failed save does fail the command (missing target path, misconfiguration, no store access). Keep save inside the step that produced the paths so misconfiguration surfaces early.

Because nothing invokes these commands automatically, the standard placements are inline in the command step (as in Quick Start) or in repository pre-command / pre-exit hooks for pipelines with many steps sharing the same caches. A pre-exit hook also runs after failed steps, and because save skips existing entries, saving a partially built path can poison the key until it expires — guard hook saves with the command's exit status:

#!/bin/bash# .buildkite/hooks/pre-exitif [ "${BUILDKITE_COMMAND_EXIT_STATUS:-1}" -eq 0 ]; then  buildkite-agent cache savefi

Storage and Registries

Cache entry metadata lives in a cache registry in Buildkite; the archived bytes live in a cache store. Registries are scoped to a cluster, and the default registry slug ~ resolves to the cluster's default registry — most setups never need --registry.

Hosted agents: zero configuration. The registry and cache store are provided; buildkite-agent cache save/restore work out of the box.

Self-hosted agents: after Buildkite support enables the feature, provide a store with BUILDKITE_AGENT_CACHE_STORE_URL (typically as an agent environment variable or in an environment hook):

#!/bin/bash# .buildkite/hooks/environment (or agent-level environment configuration)export BUILDKITE_AGENT_CACHE_STORE_URL="s3://my-cache-bucket/buildkite?region=us-east-1"

Uploads and downloads go directly between the agent and the bucket using ambient credentials (the AWS default credential chain — instance profile, IRSA, or environment variables). Buildkite never receives the cached bytes and issues no storage credentials. Grant agents s3:GetObject and s3:PutObject on the bucket (these also authorize the self-copy the agent performs to refresh object timestamps). file:///some/path URLs work for local testing. Blob lifecycle in the bucket is the operator's responsibility — add an S3 lifecycle rule keyed on last-modified time. The agent refreshes last-modified on restore to keep hot blobs alive, but the refresh is best-effort: failures are ignored and objects over S3's 5 GB CopyObject limit cannot be refreshed, so a lifecycle rule can still expire a regularly restored large cache. Size lifecycle windows generously.

Registries carry save and restore policies controlling which jobs may write or read entries — for example, restricting saves to default-branch builds while allowing restores from any branch. Policies can also scope entries by branch, build, or pipeline, adding those dimensions to the entry address, so a scoped registry can intentionally miss on identical paths and keys saved from a different branch or pipeline. Policies are configured on the registry in Buildkite, not in .buildkite/cache.yml. Entries expire automatically after a few days without an exact-match restore; exact-match restores refresh the expiry.

Common Mistakes

Additional Resources

Reference Files

• references/configuration-reference.md — Complete key-part semantics, checksum globbing rules, store URL options (S3 query parameters), environment variables, and save/restore lifecycle details

Examples

• examples/cache.yml — Multi-language cache configuration (Node.js, Ruby, Go) with fallbacks

Further Reading

• Caching best practices — overview of all Buildkite caching mechanisms
• The Buildkite agent — agent installation and configuration
• Buildkite hosted agents — hosted agent setup
• Cache volumes — the separate hosted-agent volume caching mechanism