Nexus Common Error Handling

Docker Repository Push Image with Partial Layer Errors: blob unknown:blob unknown to registry

The specific error might look like this:

xxxxxx: Layer already exists
yyyyyy: Layer already exists
errors:
blob unknown:blob unknown to registry
blob unknown:blob unknown to registry
blob unknown:blob unknown to registry

Nexus Error Symptoms

First, check the Nexus logs, and you'll find corresponding errors in log/nexus.log:

2025-09-10 01:36:22,857+0000 ERROR [qtp226372090-117]  llm org.sonatype.nexus.repository.docker.internal.datastore.recipe.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:6414378b647780fee8fd903ddb9541d134a1947ce092d08bdeb23a54cb3684ac for: ctbots/base/llamafactory/pytorch2.7.1-cu12.8-flashattn2.7.4 in repository RepositoryImpl$$EnhancerByGuice$$2e20a720{type=hosted, format=docker, name='ctbots-docke
r'}
2025-09-10 01:36:22,858+0000 ERROR [qtp226372090-117]  llm org.sonatype.nexus.repository.docker.internal.datastore.recipe.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:ad69d38804778a7dd78e609a0b3cbeb96542df5c37738a8c398ec31498a1490b for: ctbots/base/llamafactory/pytorch2.7.1-cu12.8-flashattn2.7.4 in repository RepositoryImpl$$EnhancerByGuice$$2e20a720{type=hosted, format=docker, name='ctbots-docke
r'}
2025-09-10 01:36:22,858+0000 ERROR [qtp226372090-117]  llm org.sonatype.nexus.repository.docker.internal.datastore.recipe.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:2d01ee89ef0b66169d84c496b3175d51ff2a47b2e211d7e4f4fe69868681cd31 for: ctbots/base/llamafactory/pytorch2.7.1-cu12.8-flashattn2.7.4 in repository RepositoryImpl$$EnhancerByGuice$$2e20a720{type=hosted, format=docker, name='ctbots-docke
r'}
2025-09-10 01:36:22,859+0000 ERROR [qtp226372090-117]  llm org.sonatype.nexus.repository.docker.internal.datastore.recipe.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:7d21de8cade150523dfe4c80e31ded729bb70dce5336648c5d0ef4a618ef9e3c for: ctbots/base/llamafactory/pytorch2.7.1-cu12.8-flashattn2.7.4 in repository RepositoryImpl$$EnhancerByGuice$$2e20a720{type=hosted, format=docker, name='ctbots-docke
r'}
2025-09-10 01:36:22,859+0000 ERROR [qtp226372090-117]  llm org.sonatype.nexus.repository.docker.internal.datastore.recipe.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:4b650590013c9e92d469a7b5514dbe45cd755b8d75f0d6ae09e2a3e089b0c82c for: ctbots/base/llamafactory/pytorch2.7.1-cu12.8-flashattn2.7.4 in repository RepositoryImpl$$EnhancerByGuice$$2e20a720{type=hosted, format=docker, name='ctbots-docker'}
2025-09-10 01:36:22,871+0000 WARN  [qtp226372090-117]  llm org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: PUT /v2/ctbots/base/llamafactory/manifests/pytorch2.7.1-cu12.8-flashattn2.7.4: 400 - org.sonatype.nexus.repository.docker.internal.V2Exception: Invalid Manifest

You can see that the following Layer IDs are problematic:

6414378b647780fee8fd903ddb9541d134a1947ce092d08bdeb23a54cb3684ac
ad69d38804778a7dd78e609a0b3cbeb96542df5c37738a8c398ec31498a1490b
2d01ee89ef0b66169d84c496b3175d51ff2a47b2e211d7e4f4fe69868681cd31
7d21de8cade150523dfe4c80e31ded729bb70dce5336648c5d0ef4a618ef9e3c

These layer sha256 values can actually be tested for download in Nexus, for example:

http://repo.ctbots.com:8081/repository/docker-public/v2/ctbots/testpush/blobs/sha256:6414378b647780fee8fd903ddb9541d134a1947ce092d08bdeb23a54cb3684ac

At this point, you'll find that they can actually be downloaded, but Nexus logs report that these blobs cannot be found and located.

Why? Why?

Some layers in our ctbots docker push are successful, while others fail.
Why can the failed layers be downloaded manually via blobs, but they're not missing?

Nexus Error Analysis

This is actually due to incorrect Docker mixed repository configuration.

Generally, we configure 3 Docker repo repositories:

Proxy acceleration repository (usually proxy type), let's call it docker-proxy
Private internal repository (usually host type), let's call it docker-inner
Mixed virtual repository (usually group type), let's call it docker-public

The docker-public mixed repository can bind multiple repositories and set priorities (binding docker-inner, docker-proxy). When someone does a docker pull from this repository, it first searches the docker-inner private repository. If not found, it downloads from the docker-proxy repository. If still not found, it reports an error.

Both proxy and host type repositories need to set blob storage locations. Nexus has only one by default. The error occurs when 2 conditions are met:

docker-proxy and docker-inner use the same blob storage.
Nexus uses Nginx reverse proxy, exposing port 443. If pulling images, it goes through docker-public repository; if pushing, it stores in docker-inner.

The error occurs as follows:

There's a public image, e.g., nginx:latest cached by docker-proxy. A certain docker Layer id is 233xxx, cached in blob (managed by docker-proxy repository).
A user creates a custom nginx image and docker pushes to docker-inner. At this time, the docker program queries Nexus through Nginx reverse proxy to see if a certain Layer exists. If it exists, it skips upload to save time.
Layer id 233xxx GET request reaches Nginx, reverse proxied to docker-public virtual repository. The virtual repository can't find 233xxx layer in docker-inner, but finds it in docker-proxy, returning 200.
Once the Layer is found in docker-proxy, docker program knows it doesn't need to upload this Layer and can skip it. Remaining Layers are uploaded to docker-inner (because we have virtual repository, acceleration and private repositories are separate).
Finally, after upload completion, docker-inner private repository needs to merge the entire pushed image and establish symbolic references across repos (Nexus isn't naive - it knows some Layers are user-uploaded, some are cache hits, because queries go through docker-public repository, Layers might be from its own or other repositories, requiring reference establishment).
But because docker-inner and docker-proxy use the same blob, the same Layer path processing conflicts, causing errors.

The solution is simple: separate the blob storage for docker-proxy and docker-inner to avoid mutual interference that causes Layer reference failures; or simply don't use Nginx reverse proxy for push-pull integration (push-pull integration currently seems to have many pitfalls).