Cooldown profiles and the unprefixed URLs share a packument's cache row,
but each chose its format by its own policy: abbreviated without
cooldown, full with it. With cooldown on for only some of them, a request
without cooldown that revalidated the row stored the abbreviated
packument. The next request under cooldown rejected it and fetched the
full one again, even within metadata_ttl, and had no stale copy to serve
when the upstream failed.
Cooldown in the global policy or in any profile now makes all of them
request the full packument, as upstream.npm_full_metadata does. URLs
without cooldown are then served full packuments too.
With cooldown on, every simple index request and every download reads
the package's JSON API document for its upload times. It was decoded
into a generic map, which for a package like botocore (4.4 MB) allocated
about 17 MB per request, also when everything was cached. The times are
now decoded into a struct that names only that field, so the rest of
the document is skipped.
Arch wrote databases and signatures with the validators of the cache row
read at write time, which may describe other bytes once another instance
rewrote it. Its results now carry validators: fetchedResult for a fetched
database, describedBy for a stored one, and stale on the fallback after an
upstream failure, which also marks it with metadata_ttl 0. Nothing calls
writeMetadataCachedResponseWithEncoding or lookupCachedMeta any more.
A file named {artifact}-{version} under a dotted group directory, such as
org.foo/bar/1.0/bar-1.0-linux.exe, was not recognised as a coordinate file
and so was proxied past the checks. isCoordinateFile no longer checks the
group, so handleDownload answers 404 for it. The docs say which files get
the 404, and the Range test covers a stale If-Range.
A file whose group directories hold a dot or colon was proxied without
caching, which skipped the denylist, scan and cooldown. Such paths aren't
in the Maven layout, so they now get 404. Directory listings and catalogs
still go through proxyUpstream, which shares its error handling with
Proxy.ProxyUpstream in relayUpstream.
The fallback served replaced bytes with the old row's content type and
encoding, so a blob swapped from identity to gzip went out without its
Content-Encoding. It now reads the row once more and serves the blob
under the row that describes it, and treats the copy as unusable if
none does yet.
AppendUnquote reports invalid UTF-8 after replacing it, which failed the
whole copy although the decoder allows it. Key now holds U+FFFD in its
place, as with encoding/json and ScanPackument, and the key is still
written with its original bytes.
CopyObject takes the reader and decodes it with jsontext, accepting
repeated names and invalid UTF-8 as encoding/json did. Keys are written
with the bytes they were read with instead of being re-encoded. Member
gets Skip, which reads past a value without keeping it, and npm's
RewriteStream skips dropped versions and duplicates with it.
conda-forge noarch repodata.json (195 MB): filtering 0.89 to 0.62 s and
379 to 77 MB allocated, the timestamp index 0.84 to 0.61 s. The next
packument (32 MB): RewriteStream 69 to 34 ms and 38 to 3.5 MB.
The repeated dist-tags name in the jsontext test now has a different
first value, so the test checks that the last one counts. Comments
that repeat the code go, and the upload-pack paragraph states its
check rule once.
Each match searched the rest of the buffer for every replacement. The
root index has no single-quoted links, so after every double-quoted one
the other pattern scanned to the end again, making each chunk quadratic
in its links. Each replacement's next match is now kept and only
searched for again once a replacement passes it (50,000 links: 18.6 to
818 MB/s).
The release listing read the cached InRelease, Release or repomd.xml
without comparing it with the row's digest, so bytes another instance had
replaced could decide which index is served. Such a file is now ignored,
and the index falls back to its own cache entry.
Paths the proxy doesn't cache, such as ones with a dotted group
directory, went to the primary upstream only, so a file available only on
the Gradle Plugin Portal returned 404. They now try the Plugin Portal on a
404, as cached files do.
The docs say refreshing Debian, Alpine and RPM packages needs
cache_metadata, but nothing said so when it was off. Rejecting the
setting would also stop the refresh of every other ecosystem, so the
server logs a warning from the new Config.Warnings at startup.
A version's first number was taken as its major line, so certifi
2024.8.30 and tzdata 2024.2 never got the next year's release. A first
number of 1000 or more is a year, and all such versions are one line.
Refresh called mirrorOne on the scheduler's goroutine without the
recover Run puts around each item, so a panic took the proxy down.
Both now go through mirrorRecovered, which counts it as a failure and
carries on.
The refresh read InRelease, APKINDEX and repomd.xml again through the
metadata cache, which marked them accessed, so every suite looked
fetched within since and never aged out. Read them straight from
upstream, as the Packages and primary indexes already are.
A file the refresh mirrored was fetched just now and never accessed,
which counted as recent use, so the refresh kept every package it had
refreshed in the most used list forever. Fetches under the new
handler.MirrorFetch context set mirrored_at, and a file whose last fetch
was a mirror's counts only once a client downloads it.
security.debian.org keeps files under pool/updates/main while its
InRelease lists main/binary-*, so taking the first segment after pool/
as the component found no index and security suites were never
refreshed. Take the first or second segment, whichever the release
file lists indexes for.
A tag found in the cache after the refresh failed was reported as
mirrored with no sign it might be stale, though tags can move. Keep
serving it, as a cached commit is, but log a warning naming the tag,
the commit and the refresh error.
