Run rm -rf /var/cache/apt in its own Dockerfile step and your image gets bigger, not smaller. The deleted bytes are still shipped in full, plus a marker file recording the deletion. This isn't a Docker bug — it's a direct consequence of how the OCI image format represents filesystems, and the mechanics go all the way down to a character device with device number 0/0 sitting in your container's writable layer.

A layer is a changeset that cannot subtract

An OCI image layer is a tar archive, but the spec is explicit that layers of type application/vnd.oci.image.layer.v1.tar are "applied, rather than simply extracted as tar archives." Each layer records a changeset against the stack below it: additions, modifications, and removals.

Additions and modifications are easy — the file simply appears in the tar, stored in full. Removals are the problem. A tar archive has no way to express "this path should not exist," because absence from the archive just means "unchanged." So the spec borrows a trick from older union filesystems: the whiteout file.

A whiteout is an empty regular file whose name is the deleted path's basename with a .wh. prefix. Deleting a/file2 produces an entry a/.wh.file2 in the next layer's tar. The spec attaches several rules to this:

  • A whiteout MUST only apply to resources in lower/parent layers. You cannot hide a file created in the same layer; only a later layer can do that.
  • Once applied, the whiteout itself is hidden — neither the deleted file nor the marker appears in the merged filesystem.
  • Whiteouts are regular files in the archive regardless of what they delete, including directories.
  • A .wh. entry with no corresponding path to delete is invalid, and implementations SHOULD error on it.
  • Because the prefix is reserved, a real file or directory in an image can never start with .wh..

There's also a bulk form: an opaque whiteout, a file named .wh..wh..opq placed inside a directory. It hides all children of that directory from lower layers, including subdirectories. If a layer empties out bin/, it can ship a single bin/.wh..wh..opq instead of one explicit whiteout per child. The spec says implementations SHOULD generate explicit whiteouts but MUST accept both forms, and the opaque marker is processed before any new entries in that directory, regardless of where it appears in the archive.

Now the size math is obvious. Layers, once created, are immutable — containers and derived images share them, which is the whole point of layer caching and registry deduplication. So when your Dockerfile does:

dockerfile
RUN apt-get install -y build-essential   # layer N: +300 MB of toolchain
RUN make && make install                 # layer N+1
RUN apt-get purge -y build-essential     # layer N+2: whiteout markers

layer N still contains every byte of the toolchain, stored in full, and anyone pulling the image downloads all of it. Layer N+2 adds a handful of .wh. entries on top. You paid for the files and for their tombstones.

A subtler version of the same trap: because modifications are also stored in full, a metadata-only change duplicates the file. RUN chmod 755 /app/big-binary in its own step re-ships the entire binary in a new layer — the changeset format has no "attributes changed" entry for regular files.

What the kernel actually does with a delete

The .wh. convention is the wire format. On a running host, Docker's overlay2 driver uses the kernel's overlayfs, which has its own representation. An overlayfs mount combines directories:

mount -t overlay overlay \
  -o lowerdir=/lower2:/lower1,upperdir=/upper,workdir=/work /merged

Lower directories are read-only (rightmost is the bottom of the stack), upperdir receives all writes, workdir must live on the same filesystem as upperdir, and merged is the unified view the container sees as its root. In Docker's mapping, each image layer is a lowerdir and the container's writable layer is the upperdir. The driver natively supports up to 128 lower layers, and under /var/lib/docker/overlay2 it maintains an l/ directory of short-ID symlinks to layer directories — per the docs, "these identifiers are used to avoid hitting the page size limitation on arguments to the mount command," since every lower layer path has to fit into one colon-separated mount option.

When a process in the container deletes a file that lives in an image layer, overlayfs can't touch the lower directory — it needs rm and rmdir to work without changing the lower filesystem. So it records the deletion in the upper layer as a whiteout, which the kernel docs define as "a character device with 0/0 device number" (newer kernels also accept a zero-size regular file carrying the trusted.overlay.whiteout xattr). A lower-layer entry with a matching whiteout above it is ignored, and the whiteout itself is hidden from the merged view. Deleting an entire directory sets the xattr trusted.overlay.opaque=y on the upper directory, which masks the whole lower directory at once — the runtime translates between this on-disk form and the .wh. tar convention when packing and unpacking layers.

Writes have a cost worth knowing too. Overlayfs operates at file granularity: the first write to a lower-layer file triggers a copy-up — parent directories are created in the upper layer, then metadata, file data, and extended attributes are copied — and the entire file is copied even if you change one byte. Subsequent writes hit the upper copy directly. This is why Docker's docs steer write-heavy workloads to volumes, which bypass the storage driver entirely. There's an upside to the shared lower layers, though: multiple containers reading the same image file share a single page cache entry for it.

One sharp edge: renaming a directory that lives in a lower layer isn't a rename at all. rename(2) returns EXDEV unless both source and destination are already in the top layer, and applications fall back to copy-and-unlink. (The kernel's redirect_dir feature can fix this by copying up the directory shell and recording a trusted.overlay.redirect xattr, but it's an opt-in mount option.)

Shrinking images for real

Since a layer can only add, the fix is to never let the unwanted bytes into a committed layer in the first place.

Delete in the same instruction that creates. A file created and removed within one RUN never appears in that layer's changeset — the tar records the net result:

dockerfile
RUN apt-get update \
    && apt-get install -y --no-install-recommends curl \
    && rm -rf /var/lib/apt/lists/*

Use multi-stage builds when the garbage is the whole toolchain. A multi-stage Dockerfile runs the build in one stage and copies only the artifacts into the final one:

dockerfile
FROM golang:1.26 AS build
WORKDIR /src
COPY main.go .
RUN go build -o /bin/hello ./main.go

FROM scratch
COPY --from=build /bin/hello /bin/hello
ENTRYPOINT ["/bin/hello"]

Only what you explicitly COPY --from reaches the final image — in Docker's words, "a tiny production image with nothing but the binary inside." Stages are referenced by index or by an AS name (names survive reordering), COPY --from can even pull from external images like nginx:latest, and BuildKit only builds the stages your target actually depends on, unlike the legacy builder which processed every stage up to the target.

The mental model that makes all of this stick: an image is not a filesystem, it's a log of filesystem edits, and like most append-only logs, it can mark things dead but never reclaim them. Every RUN line is a commit you'll ship forever — so make each one leave behind only what the final container needs.

References

dockeroverlayfsocicontainer-imageswhiteout