Cached metadata responses took their ETag, Last-Modified and stale
flag from the cache row as it was after the fetch, not from the
document actually sent. When a refetch got a new document but could
not store it, the client got the new body with the old ETag and a
Warning header, and a client still holding the old document got a
304 for it. With metadata_ttl 0 the stale fallback had no Warning at
all, since staleness was inferred from the row's age.
The lookup result now carries the validators of its body (upstream's
for a fetched document, the row's for a stored one) and whether it is
a stale fallback, and the response is written from those.
A 304 also trusted whatever was stored at the row's path. Another
instance sharing the store may have replaced it meanwhile, and the
new bytes were then served and stored under the old ETag. The stored
bytes are now checked against the row's digest, and a 304 whose row
can no longer be refreshed is fetched again without validators
instead of storing the old document back over the new one.
Debian, RPM, APK, PyPI, CRAN and Maven build upstream URLs from the
decoded request path, where a '?' or '#' would end the path and cache
one file under another's name. Only the server middleware refused them;
Arch and FreeBSD also check in the handler. Check in these handlers
too, so they stay safe when mounted without the middleware.
Conan, CRAN, Conda, RPM, the main Debian archive and the default Maven
repository keyed cached metadata by the request path with '/' replaced
by '_'. Names and versions may contain '_', so foo/bar_1.0 and
foo_bar/1.0 shared an entry, and with cache_metadata one reference's
recipe files were served for the other. Conan also appended the query
with '_', so a path without a query could collide with a shorter path
plus one.
Hash the upstream URL, the path and, for Conan, the query instead, as
the APK, Arch, generic and Helm handlers already do. Each part is
length-prefixed so no two inputs share a key.
Clients resolve base_url and shards_base_url against the proxy's URL. A
scheme or host sends them around the proxy, and a leading slash leaves
its mount, so neither layout is served.
The shard index (repodata_shards.msgpack.zst) and the shards of CEP 16
fell through to the uncached passthrough, so clients that use them, such
as pixi, got no cooldown filtering and no offline copy.
Without cooldown, the index and shards are now served through the
metadata cache. With cooldown, shards have to be filtered, but clients
cache them by the hash in the index and never fetch a hash twice. The
proxy rewrites the index instead: each hash is XORed with a mask derived
from channel, subdir and the current hour, and shards_base_url points at
cooldown-shards/{hour}/. That route undoes the mask, fetches the
upstream shard and drops records still in cooldown. The unmasked shard
paths answer 404 while cooldown is on.
When the index puts packages or shards somewhere the proxy doesn't
route, or has an unknown version, the proxy answers 404 and clients fall
back to repodata.json, which is filtered as before.
Fixes#529
