* max_size now counts cached metadata as well as artifacts, and eviction drops whichever was used least recently, so metadata no longer piles up for good (#461). Metadata rows record when they were last served, an artifact not yet served counts from its download, and Arch stores a database blob again if eviction removed it under a refreshed row.
* Shorter comments on the metadata last-access migration.
* Show evicted metadata on the analytics page next to evicted artifacts, and cover the new counter in the metric surface test.
* Evicting metadata now drops the row and queues its path in one transaction, so a crash before the blob is deleted leaves it to reclaim instead of orphaning it.
* The last-access backfill now runs in the background in batches of 1000 instead of inside the migration, resuming after a restart, and until a row is backfilled eviction orders it by its write time.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
* Proxy the OCI referrers API in the container handler
GET /v2/{name}/referrers/{digest} used to fall through to a plain 404,
so clients only ever saw referrers stored under the sha256-<hex> tag
schema. Registries that keep SBOMs and signatures behind the referrers
API looked empty through the proxy.
The index is now fetched from the upstream that the repository routes
to and cached like the tag list: fresh within the metadata TTL,
revalidated with If-None-Match, and served stale when the upstream is
down. artifactType is not forwarded, so one cached row covers every
filter and clients filter the full list themselves, which they do when
OCI-Filters-Applied is absent. An upstream 404 is passed through
untouched, so clients still fall back to the tag schema for registries
like GHCR. When the upstream cannot answer and nothing is cached, the
proxy keeps answering 404 as before rather than failing the request.
Closes#144
* Cover absolute upstream links in the referrers link test
The second page now answers with an absolute Link pointing back at the
upstream host, and the test checks that it comes back rewritten to the
proxy. That was missing so far, and it also gives the upstream variable
a reason to be declared before the handler closure, which staticcheck
complained about.
* Read cached referrers through a small shared loader
loadContainerReferrers was a line-for-line copy of loadContainerTags with
the types renamed, which dupl flags in CI. The part that opens the cache
row and reads the body now lives in loadContainerMetadata next to
storeContainerMetadata, and the referrers loader only maps the row onto
its struct. The tag and manifest loaders are left as they are.
* Make the manifest routing test notice which handler answered
For foo/manifests/referrers/<digest> the manifest handler and the
referrers handler send the very same upstream path, so the test passed
even with the referrers case moved in front of the manifest case. The
fake upstream now rejects requests that only accept an image index,
which is what the referrers handler sends.
* Check that a cached referrers row links back to whoever reads it
A repository can be reached as upstream/<name>/<repo> and, when the
default registry is the same host, as plain <repo>. Both end up in the
same cache row, so the pagination Link has to be rewritten when the row
is served, not when it is stored. The link test now reads the row
through both paths; with a store-time rewrite the second path would get
the first one's link.
* Make the referrers ETag test prove the revalidation
Without If-None-Match the fake upstream just sends the full index again,
so the old assertions held either way. The test now counts the 304s the
upstream sent and expects exactly one.
* Reword which tools use the referrers API in the README
The cosign mention was too broad. cosign reaches the referrers API for
its newer signature bundles, while the old .sig tags keep going through
the normal manifest path. Also rewraps the paragraph, one line had run
past the others.
* Check that a 304 restarts the referrers cache TTL
The ETag test now backdates the cached row instead of running with a
zero TTL. After the upstream answers 304, a third request has to come
from the cache. Before, the test would also have passed if the 304
branch forgot to bump fetchedAt and every request kept revalidating.
* Test the referrers Content-Type fallback
None of the fake upstreams left out Content-Type, so the default to the
image index type was never exercised. oras-go rejects anything else, so
it is worth pinning, both for the first fetch and for the cached copy.
* Test referrers digests with extra characters around them
The invalid-request table only had digests that were too short or used
an unknown algorithm, so it would not notice if the digest pattern lost
its anchors. Two cases now put junk before and after an otherwise valid
sha256 digest.
* Reuse the manifest cache key for referrers
The referrers handler had its own key function, but it did exactly what
containerManifestCacheKey already does with four strings. Referrers rows
sit in their own oci-referrers ecosystem anyway, so they can't collide
with manifests.
* Drop the separate struct for cached referrers
A cached referrers response carries the same fields as a cached tag
list, and the freshness check was a straight copy too. So the referrers
code just uses cachedContainerTags and containerTagsFresh now. Loading,
storing and writing stay on their own, since the ecosystem and the Link
handling are different.
* Route referrers requests by ns like the other OCI endpoints
Now that containerd ns support is on main, referrers requests have to
go through registryForRequest too, otherwise a _default mirror would
look every subject up on Docker Hub. ns only picks the registry, so it
is dropped from the forwarded query and the cache key, and the next-page
Link keeps it, the same way tag lists do. This also follows the new
rewriteContainerTagsLink signature, which takes the namespace.
* Without a cached index, referrers now pass upstream errors through and answer 502 when the upstream can't be reached or sends garbage, so clients no longer read an outage as a registry without the referrers API. Stale cache is still served when there is one.
* The tag list and manifest routes now only match paths that end in their endpoint, so a repository with tags/list or manifests in its name can still reach the referrers endpoint.
* Add Homebrew JSON API and bottle proxy support
* Fix Homebrew HEAD offline fallback and non-sha256 OCI manifest handling
Route Homebrew API HEAD requests through ProxyCached so a warm cache
answers without an upstream call and stale entries are served when the
upstream is unreachable. HEAD still reaches upstream as HEAD when
metadata caching is disabled.
Limit OCI manifest digest verification to sha256 references and
Docker-Content-Digest headers so other digest algorithms are proxied
instead of rejected, and log the failing expected value.
* Reconcile with #280 and #301 after rebase
Compute real manifest digests in #280's fixture upstreams so the new
verification accepts them, and add headerETag / headerLastModified to
* Send fixed Accept for Homebrew API and match If-None-Match properly
The Homebrew API cache key does not include Accept, so replaying the
client header could serve one representation under another; the API
does not negotiate anyway. Compare If-None-Match with weak comparison,
list splitting and "*" per RFC 7232 instead of string equality, and
apply the same helper to the metadata and swift responders.
* Reconcile with #298 and #304 after rebase
Move the configureScanning doc comment back to its function after the
auto-merge stacked it on mountProtocolHandlers, and drop the second
ETag/Last-Modified set in writeMetadataCachedResponse now that the
pre-304 set covers both response paths.
