Docker Pull Failed With No Space Left on Device

Article · Interrupted execution

Docker could not unpack the new site image: no space left on device. The scheduler had found the due articles, and the build had succeeded. The release failed later, on the server. The repair removed obsolete site image tags while protecting rollback and in-use images, then checked the live publication state.

“Is the schedule working? And tested to work?”

The question came up while preparing articles for this site. The repository already had publication dates, rules to hide future posts, scheduler tests, and an hourly workflow. Those pieces made the schedule look ready.

The live site told a different story.

On October 24, 2026, a scheduled run found articles that should have been public. It built the site image and pushed it to the registry. Then the server failed to pull the image. The new pages never reached readers.

This was a real failure in this repository, with a deployment log that showed where to look next.

The scheduler had found the missing pages

The publication check compares articles due for release with the live sitemap. The failed scheduled run found thirteen missing articles and requested a deployment.

So the scheduler hadn't simply decided there was nothing to publish. It had found the missing pages and started the next step.

The build and registry push also succeeded. That left the server pull and restart to investigate. The image existed, but the server still needed to download it, unpack it, and start serving the new version.

The pull failed while unpacking a layer:

failed to register layer: write
/app/node_modules/lit/polyfill-support.d.ts:
no space left on device

The server had run out of space. The filename showed where extraction failed; it didn't show what had filled the disk.

Building the same image again wouldn't by itself free that space. The repair needed to address the server's ability to accept the release. A stale Docker build layer is a separate publication problem: there the image itself has yesterday's pages. Here the new image existed and failed to reach the running service.

What the passing tests had checked

Scheduler tests can check that a future article stays hidden, a due reviewed article becomes eligible, and a draft stays locked. Tests of the live-site checker can check that a missing URL requests deployment and a failed sitemap fetch is reported.

Those results still matter. They helped narrow the investigation to what happened after the publication decision. They just couldn't establish that the server had room for another image.

The reverse matters too. A healthy container doesn't prove that future drafts are hidden from search or the sitemap. Each result answers a different question.

An agent reporting “the schedule is tested” should say which part it means. The date rules may pass while deployment fails. A built image may be correct while the live site serves the old version. Keeping those results separate makes the next repair easier to choose.

Remove old site images before pulling the next one

The repair removes old tags from this site's image repository before the pull. It protects the rollback image and images used by existing containers. It avoids forced removal and leaves unrelated volumes alone.

There was a mistake in the first cleanup design. An image can have several tags. Docker documents that deleting one tag may leave the image's data in place, while deleting by image ID can be refused when several references remain.

Review caught that case. The revised script lists this site's tags and removes eligible ones individually, while keeping the protected image IDs. The workflow at the repaired release contains the full cleanup and rollback sequence.

This excerpt shows the removal decision after the script has collected the previous image ID, IDs used by all existing containers, and this site's image tags:

printf '%s\n' "$site_images" |
while read -r site_image site_tag; do
  case "$protected_images" in *" $site_image "*) continue ;; esac
  case "$site_tag" in
    mast2133/mehdicz-site:\<none\>) continue ;;
    mast2133/mehdicz-site:*) docker image rm "$site_tag" || true ;;
  esac
done

The protected set contains image IDs, not only tag names. That matters when the rollback image or a container's image has several tags. Removing one eligible tag is also not proof that its layers were freed: another reference can keep them alive. The later successful pull establishes that this release could proceed; the log is not a complete disk-usage measurement.

A regression test uses a fake Docker command to check the selection. Its cases include two tags on one image, an image used by a container, and a tag from another repository. It checks which tags the script tries to remove. It cannot measure how much space the real server recovers.

The real pull would have to show whether the cleanup made enough room.

The release caught another mistake

The first release attempt stopped before deployment. The new regression test assumed Bash was installed, but the Alpine build image only provided sh.

The test was changed to use the same portable shell syntax as the deployment script. After another successful CI run, the release was retried.

The repaired release succeeded. Its log shows old site tags and image layers removed before the pull. The server downloaded the new image, and the blog container became healthy at 13:06 UTC.

That was the first result that showed the release had reached the server. The next question was whether readers could see the due articles.

Check the live site

A new manual check-only option compares due articles with the live sitemap without building or deploying. It fails if due pages are missing or the sitemap cannot be checked.

The check-only run passed at 13:07 UTC. It found no missing due articles and no overdue work. Its build and deployment steps were skipped.

A separate fetch of the prompting article returned HTTP 200 with the expected title, heading, and opening paragraph. The sitemap check covered the due URLs; that fetch checked the article page itself.

For the next due article, compare three things: whether the checker requested a deployment, whether that deployment succeeded, and whether the expected URL is present in the live sitemap. If the site already contains the due article, no new deployment is needed. Otherwise, a successful checker alone is only the start of the release.

The workflow checks hourly, at minute 17. GitHub warns that scheduled runs can be delayed or dropped under high load, so the article's date isn't a promise of delivery at that exact minute.

For this release, the question had an answer: the due pages were in the live sitemap, and the original article could be read. The ten CI drafts being prepared during that repair remained locked for review.