Retention evicts every file that has not been served for the configured
time, also the newest version of a package. keep_latest: N holds back
the files of the N highest cached versions of each package from
eviction by age. Versions are ordered by the ecosystem's version rules.
It applies to cargo, golang, npm and pypi, where the cache stores a
release under its version. Other ecosystems ignore it for now.
A path value is decoded, so a revision of R%2Fpackages%2FP%2FQ gave a
recipe file the cache key of a package file. The recipe and package file
routes now answer 400 when user, channel, a revision, the package
reference or the filename contains a slash.
The cooldown read a registration with encoding/json into a struct and
the copy read it again token by token. The two disagreed on documents
with member names in another case and with repeated members: the struct
matched Items and Published, and kept the first or merged where the copy
took what it saw. Such a document could be served with releases that are
in cooldown, by the registration endpoints, the version list and the
download check, or fail after part of an index was sent.
The cooldown now reads with the copy's decoder and its rules: exact
names, as main matched them. A node that repeats @id, items,
catalogEntry, version or published is refused.
Cooldown profiles and the unprefixed URLs share a packument's cache row,
but each chose its format by its own policy: abbreviated without
cooldown, full with it. With cooldown on for only some of them, a request
without cooldown that revalidated the row stored the abbreviated
packument. The next request under cooldown rejected it and fetched the
full one again, even within metadata_ttl, and had no stale copy to serve
when the upstream failed.
Cooldown in the global policy or in any profile now makes all of them
request the full packument, as upstream.npm_full_metadata does. URLs
without cooldown are then served full packuments too.
The registration endpoints decoded the index and, with cooldown on, all
of its pages into generic maps, filtered and rewrote the tree and
encoded it again. For AWSSDK.Core that allocated about 80 MB per
request for a 6.8 MB response.
The document is now copied token by token. Links are rewritten on the
way, leaves in cooldown and pages left empty are cut out of the output
buffer, and count, lower and upper are written once a page's leaves are
known. The index and its linked pages are still loaded and checked
before the response starts, so a registration that can't be filtered
stays an error response.
The output keeps the upstream's member order and bytes, so it is the
same JSON as before but no longer sorted by key.
With cooldown on, the version list reads the package's registration
index and all of its pages to find the versions in cooldown. Each
document was decoded into a generic map, which for a package like
AWSSDK.Core (6.8 MB of pages) allocated 26 MB per request, also when
everything was cached. The registration is now decoded into a struct
that names only the version and publish time, so the rest is skipped.
The download check reads its leaf the same way.
With cooldown on, every simple index request and every download reads
the package's JSON API document for its upload times. It was decoded
into a generic map, which for a package like botocore (4.4 MB) allocated
about 17 MB per request, also when everything was cached. The times are
now decoded into a struct that names only that field, so the rest of
the document is skipped.
A metadata fetch always ran on a context detached from its first
caller, so a client could request many uncached documents under
different names and hang up at once, leaving one upstream fetch per
request running to the HTTP client's timeout, each buffering up to
metadata_max_size. Artifact fetches only detach while they hold one of
maxDetachedFetches slots. Metadata fetches now do the same with their
own 64 slots; beyond that, a fetch ends with its first caller.
A Debian package was cached under its repository and file name, but
fetched from its full path. Docker's apt repository has a pool under each
suite with the same file names, so a bookworm client could get the
bullseye build that was fetched first, and fail its hash check. Packages
are now cached under their path.
Conan joined revision, package reference, package revision and file name
with underscores, which revisions can contain, so a recipe file and a
package file could share a cache entry. The parts are now joined with
slashes.
Entries cached under the old names are still served, but only for the
URL they were fetched from, so warm caches keep working.
Arch wrote databases and signatures with the validators of the cache row
read at write time, which may describe other bytes once another instance
rewrote it. Its results now carry validators: fetchedResult for a fetched
database, describedBy for a stored one, and stale on the fallback after an
upstream failure, which also marks it with metadata_ttl 0. Nothing calls
writeMetadataCachedResponseWithEncoding or lookupCachedMeta any more.
A file named {artifact}-{version} under a dotted group directory, such as
org.foo/bar/1.0/bar-1.0-linux.exe, was not recognised as a coordinate file
and so was proxied past the checks. isCoordinateFile no longer checks the
group, so handleDownload answers 404 for it. The docs say which files get
the 404, and the Range test covers a stale If-Range.
A file whose group directories hold a dot or colon was proxied without
caching, which skipped the denylist, scan and cooldown. Such paths aren't
in the Maven layout, so they now get 404. Directory listings and catalogs
still go through proxyUpstream, which shares its error handling with
Proxy.ProxyUpstream in relayUpstream.
The fallback served replaced bytes with the old row's content type and
encoding, so a blob swapped from identity to gzip went out without its
Content-Encoding. It now reads the row once more and serves the blob
under the row that describes it, and treats the copy as unusable if
none does yet.
CopyObject takes the reader and decodes it with jsontext, accepting
repeated names and invalid UTF-8 as encoding/json did. Keys are written
with the bytes they were read with instead of being re-encoded. Member
gets Skip, which reads past a value without keeping it, and npm's
RewriteStream skips dropped versions and duplicates with it.
conda-forge noarch repodata.json (195 MB): filtering 0.89 to 0.62 s and
379 to 77 MB allocated, the timestamp index 0.84 to 0.61 s. The next
packument (32 MB): RewriteStream 69 to 34 ms and 38 to 3.5 MB.
Each match searched the rest of the buffer for every replacement. The
root index has no single-quoted links, so after every double-quoted one
the other pattern scanned to the end again, making each chunk quadratic
in its links. Each replacement's next match is now kept and only
searched for again once a replacement passes it (50,000 links: 18.6 to
818 MB/s).
The release listing read the cached InRelease, Release or repomd.xml
without comparing it with the row's digest, so bytes another instance had
replaced could decide which index is served. Such a file is now ignored,
and the index falls back to its own cache entry.
Paths the proxy doesn't cache, such as ones with a dotted group
directory, went to the primary upstream only, so a file available only on
the Gradle Plugin Portal returned 404. They now try the Plugin Portal on a
404, as cached files do.
A file the refresh mirrored was fetched just now and never accessed,
which counted as recent use, so the refresh kept every package it had
refreshed in the most used list forever. Fetches under the new
handler.MirrorFetch context set mirrored_at, and a file whose last fetch
was a mirror's counts only once a client downloads it.
Under cooldown, a tarball for a version the packument doesn't list is
decided on a fresh fetch of the packument. That check had gaps:
- A stale copy served because upstream failed counted as fetched, so
the version went through with no publish time. It now triggers the
refetch like a copy within its TTL.
- A packument just stored from upstream counted as cached, so every
request for an unlisted version fetched it twice. Only a stored row
read within its TTL counts now.
- A stored refetch that joined another caller's fetch could get that
caller's stale fallback and take it as upstream's answer. The stored
lookup now reports stale results, and a stale refetch is refused.
- A refused refetch dropped the upstream error, so a rate-limit hold
answered 502 instead of 503 with Retry-After, and a 404 was lost.
Stale results now carry the upstream error, and the refusal wraps it.
The index lookup always used a cached InRelease over Release, so a
stale InRelease decided which index was served to a client that had
just fetched a fresh Release, and it stopped at the nearest cached
release file even when that file, such as the legacy per-component
Release, listed nothing for the path.
Skip release files that don't list the path, and where both InRelease
and Release do, use the one fetched last: apt reads InRelease and
debootstrap Release, and a client asking for an index has most likely
just fetched one of them.
Offline mode serves the cached copy without asking the database once
it is known to be unreachable, but derived staleness from the row's
age. With metadata_ttl 0 that copy had no Warning header. It is
always stale, since upstream was not asked.
