Retention evicts every file that has not been served for the configured
time, also the newest version of a package. keep_latest: N holds back
the files of the N highest cached versions of each package from
eviction by age. Versions are ordered by the ecosystem's version rules.
It applies to cargo, golang, npm and pypi, where the cache stores a
release under its version. Other ecosystems ignore it for now.
Cooldown profiles and the unprefixed URLs share a packument's cache row,
but each chose its format by its own policy: abbreviated without
cooldown, full with it. With cooldown on for only some of them, a request
without cooldown that revalidated the row stored the abbreviated
packument. The next request under cooldown rejected it and fetched the
full one again, even within metadata_ttl, and had no stale copy to serve
when the upstream failed.
Cooldown in the global policy or in any profile now makes all of them
request the full packument, as upstream.npm_full_metadata does. URLs
without cooldown are then served full packuments too.
A file named {artifact}-{version} under a dotted group directory, such as
org.foo/bar/1.0/bar-1.0-linux.exe, was not recognised as a coordinate file
and so was proxied past the checks. isCoordinateFile no longer checks the
group, so handleDownload answers 404 for it. The docs say which files get
the 404, and the Range test covers a stale If-Range.
The repeated dist-tags name in the jsontext test now has a different
first value, so the test checks that the last one counts. Comments
that repeat the code go, and the upload-pack paragraph states its
check rule once.
The docs say refreshing Debian, Alpine and RPM packages needs
cache_metadata, but nothing said so when it was off. Rejecting the
setting would also stop the refresh of every other ecosystem, so the
server logs a warning from the new Config.Warnings at startup.
A version's first number was taken as its major line, so certifi
2024.8.30 and tzdata 2024.2 never got the next year's release. A first
number of 1000 or more is a year, and all such versions are one line.
The refresh read InRelease, APKINDEX and repomd.xml again through the
metadata cache, which marked them accessed, so every suite looked
fetched within since and never aged out. Read them straight from
upstream, as the Packages and primary indexes already are.
A file the refresh mirrored was fetched just now and never accessed,
which counted as recent use, so the refresh kept every package it had
refreshed in the most used list forever. Fetches under the new
handler.MirrorFetch context set mirrored_at, and a file whose last fetch
was a mirror's counts only once a client downloads it.
A tag found in the cache after the refresh failed was reported as
mirrored with no sign it might be stale, though tags can move. Keep
serving it, as a cached commit is, but log a warning naming the tag,
the commit and the refresh error.
Document that a mutable SNAPSHOT file falls back to the cached copy
and that HEAD is answered from cache, and that the most recently
fetched of InRelease and Release decides which index is served.
A version that wasn't a lowercase full-length hex ID was taken for a
tag, so an uppercase commit ID failed as a missing tag. Lowercase a
full-length hex version first. A shorter hex version that names no tag
now fails saying it is neither a full commit ID nor a tag.
Looking for .gitmodules took any error from git cat-file as the file
not being there, so a mirror whose context was cancelled reported
success with no submodules. Use rev-parse --verify --quiet, whose exit
status 1 means the path is missing, and return other errors.
A .gitmodules git config can't parse failed the mirror of a commit that
is cached all the same. Log a warning and skip its submodules instead,
as an unserved submodule host is skipped.
An upload-pack request returned a stored refusal without checking the
upstream, so a client that never asks for info/refs, such as a protocol
v2 client sending ls-refs and fetch directly, stayed refused after a
passing 403 or 404. Refresh first while a refusal stands, as for a copy
past stale_max_age, and refuse only if the refusal remains.
Before the strict parser, PROXY_TRUST_FORWARDED_FOR=no meant false.
With only strconv.ParseBool's forms accepted, no was ignored and a
setting enabled in the config file stayed on. Accept yes/no/on/off in
any case for every boolean variable, the Gradle read-only one included.
Composer reads "notification-url": null as missing and reports installs
to notify-batch, so a pushed manifest with null now gets an empty one too.
The docs say a pushed manifest's own notification-url is kept.
Git LFS objects are scanned under the git key, which the new ecosystem
check rejected, so a scanner restricted to ecosystems: [git] would have
stopped the proxy from starting.
Non-timestamped SNAPSHOT files went straight to upstream, past the
denylist, cooldown, scanning and the Plugin Portal fallback. They now take
the normal download path, with the cached copy dropped first so each
request still gets the current build.
A database error while refreshing a verified 304 no longer forces a full
download: the bytes were checked against the row and upstream confirmed
them.
Without cooldown, maven-metadata.xml and its .md5/.sha1/.sha256/.sha512
files were cached under separate keys with separate TTLs, so a deploy
between the two fetches left the checksum describing another document
and strict builds failed. The checksum files are now computed from the
cached metadata, as the cooldown path already did. The metadata is
also written with the validators of the document served.
SNAPSHOT files named with SNAPSHOT rather than a build timestamp
(lib-1.0-SNAPSHOT.jar, as JitPack and non-unique snapshot repositories
publish them) matched the artifact rules and were cached for good, so
the first build was served forever. They are now passed through
uncached. Timestamped SNAPSHOT files never change and stay cached.
InRelease/Release and repomd.xml had their own metadata cache entries
and TTLs, independent of the Packages, Sources or plainly named
primary.xml.gz files they list. Within a TTL a client could get the
cached release file and a freshly fetched index, or a fresh release
file and a cached index, and apt then failed with "Hash Sum mismatch"
(repositories without by-hash, or RPM repositories built with
--simple-md-filenames).
An index file that the cached release file lists is now served by
that listing: the cached copy while its digest matches the listed
sha256, whatever its age, and otherwise a copy fetched from upstream.
If upstream's copy does not match either, the repository has moved on
since the release file was cached; that copy is served and the
release file is expired so the next update gets a matching pair.
The denylist kept the PURL namespace, but the Debian and RPM handlers
build pkg:deb/{name} and pkg:rpm/{name} without a vendor, while Alpine
and Arch build pkg:apk/alpine/{name} and pkg:alpm/arch/{name} with one.
So standard entries such as pkg:deb/debian/curl@... or
pkg:rpm/fedora/curl@..., and pkg:apk/busybox@... or pkg:alpm/glibc@...,
never matched. Drop the namespace of distro types on both sides, as
retention and the inventory already do, using packageurl's list of
those types.
Debian and RPM handlers read versions from filenames, which leave out
the epoch, so an entry such as pkg:deb/foo@1:2.3-1 never matched
foo_2.3-1_amd64.deb. An entry with an epoch now also covers the
version without it.
The Conda handler divided every repodata timestamp by 1000, assuming
milliseconds. Some entries (a few in conda-forge noarch) use seconds,
so their publish time came out in 1970 and they were never held back
by cooldown. Read a timestamp as milliseconds only when it is above
253402300799, as conda does, in one helper used by the repodata
filter, the shard filter and the download index.
Only GET info/refs checked the upstream. A POST git-upload-pack was served
from the cached copy unchecked, so a client posting ls-refs and fetch
directly (protocol v2 allows it) kept getting a repository that was
deleted or made private upstream, past stale_max_age, indefinitely.
Remember when a refresh finds a repository missing or refused upstream,
and refuse upload-pack requests for it until a refresh succeeds. Refresh
first when the copy was last checked more than a minute past
stale_max_age ago; the minute keeps a normal clone (info/refs, then
upload-pack) at one upstream check even with stale_max_age 0. Mirror
refreshes on its own and skips the check.
Composer reports installs to the repository's notify-batch URL unless a
version has its own notification-url, and the proxy's packages.json sets
notify-batch to the upstream. Routed packages get an empty
notification-url for this reason, but pushed and hosted-only versions
did not, so installs of private packages were reported to packagist.org.
Give pushed versions the same empty notification-url.
On a case-insensitive file system, as macOS and Windows use by default,
acme/WIDGETS and acme/widgets name the same cache directory but were
tracked as two repositories. Users of one spelling didn't keep the other
from being evicted, so eviction could move the directory away while it
was being read, and each entry had its own write lock, so two refreshes
could fetch into the directory at once.
Check at startup whether the cache directory ignores case and, if so,
track repositories under their lowercased key. The directory and the
upstream URL keep the spelling requested. Case-sensitive file systems,
and so existing caches on Linux, are unchanged.
parsePath joins the group directories with dots, and the cache filename
is the file's base name, so org.foo/bar/1.0/bar-1.0.jar and
org/foo/bar/1.0/bar-1.0.jar shared one cache entry. Only upstreams that
serve content at such directories, such as raw or hosted repositories,
are affected.
The Maven layout splits a group at its dots, so a group directory never
contains a dot or is empty. Treat such paths as outside the layout and
proxy them uncached, which keeps every existing cache entry valid.
Cached metadata responses took their ETag, Last-Modified and stale
flag from the cache row as it was after the fetch, not from the
document actually sent. When a refetch got a new document but could
not store it, the client got the new body with the old ETag and a
Warning header, and a client still holding the old document got a
304 for it. With metadata_ttl 0 the stale fallback had no Warning at
all, since staleness was inferred from the row's age.
The lookup result now carries the validators of its body (upstream's
for a fetched document, the row's for a stored one) and whether it is
a stale fallback, and the response is written from those.
A 304 also trusted whatever was stored at the row's path. Another
instance sharing the store may have replaced it meanwhile, and the
new bytes were then served and stored under the old ETag. The stored
bytes are now checked against the row's digest, and a 304 whose row
can no longer be refreshed is fetched again without validators
instead of storing the old document back over the new one.
npm: a tarball request for a version the cached packument did not list
yet (cached within metadata_ttl before the release) found no publish
time and was let through cooldown, and fetched without integrity. When
cooldown applies and the cached packument lacks the version, fetch it
again and decide on what upstream lists now; if it can't be fetched,
fail as when a publish time can't be read.
npm: cooldown only moved dist-tags.latest. Other tags kept naming
versions in cooldown, while the denylist removes such tags. Cooldown
now removes them too, and drops latest when no version is left, in
both the in-memory and streamed rewrites.
PyPI: the simple JSON rewrite removed filtered files but left their
versions in the PEP 700 versions list. Remove them there too.
pub: latest was never rewritten, so its archive_url pointed at pub.dev,
it kept naming a version in cooldown when no version was left, and the
replacement was the last list entry, which may be a prerelease. Point
latest's archive_url at the proxy, replace a held back latest with the
newest stable version left (or the newest prerelease), and drop it
when nothing is left.
ParseSize accepted "-5GB", "NaNGB", "InfGB" and "1e30GB", the last
two clamping to the largest int64. A negative or NaN storage.max_size
passed validation, and eviction treats a limit of zero or less as
unlimited.
Reject negative, NaN, infinite and overflowing sizes in ParseSize, and
reject negative metadata_ttl and storage.direct_serve_ttl at startup.
scanning.scanners[].ecosystems was never checked, and the scanner group
compared it exactly against the keys handlers pass. Natural names such
as go, docker and rubygems passed validation but never matched the
handler keys golang, oci and gem, so a block-mode scanner restricted to
them silently never ran.
Map common aliases to the handler keys, reject unknown names at
startup, and normalise the names again when building the group.
Every boolean PROXY_* variable was true only for exactly "true" or
"1". Anything else, including "True", "TRUE" or a typo, silently
meant false, so PROXY_SCANNING_ENABLED=True turned scanning off and
validation still passed.
Parse them with strconv.ParseBool like
PROXY_GRADLE_BUILD_CACHE_READ_ONLY already does. An unrecognised value
now keeps the current setting and is logged as a warning at startup.
RPM packages were cached under their bare filename. With upstream.rpm
pointing at a mirror root that serves several distributions, Rocky and
Alma both publish bash-5.1.8-9.el9.x86_64.rpm with different bytes, and
whichever was fetched first was served for both, so dnf failed the
checksum or signature check.
Use the path under the upstream as the cache filename, as the APK and
Arch handlers do. Packages cached before this change are no longer
found and are fetched again once; retention still matches them by
package name.
A pkg:git version can now be a tag as well as a full commit ID. With a
version, the submodules the commit records are mirrored at the commits it
pins, from the upstream.git hosts serving them; one no host serves is
skipped with a warning.
