The release listing read the cached InRelease, Release or repomd.xml
without comparing it with the row's digest, so bytes another instance had
replaced could decide which index is served. Such a file is now ignored,
and the index falls back to its own cache entry.
Paths the proxy doesn't cache, such as ones with a dotted group
directory, went to the primary upstream only, so a file available only on
the Gradle Plugin Portal returned 404. They now try the Plugin Portal on a
404, as cached files do.
The docs say refreshing Debian, Alpine and RPM packages needs
cache_metadata, but nothing said so when it was off. Rejecting the
setting would also stop the refresh of every other ecosystem, so the
server logs a warning from the new Config.Warnings at startup.
A version's first number was taken as its major line, so certifi
2024.8.30 and tzdata 2024.2 never got the next year's release. A first
number of 1000 or more is a year, and all such versions are one line.
Refresh called mirrorOne on the scheduler's goroutine without the
recover Run puts around each item, so a panic took the proxy down.
Both now go through mirrorRecovered, which counts it as a failure and
carries on.
The refresh read InRelease, APKINDEX and repomd.xml again through the
metadata cache, which marked them accessed, so every suite looked
fetched within since and never aged out. Read them straight from
upstream, as the Packages and primary indexes already are.
A file the refresh mirrored was fetched just now and never accessed,
which counted as recent use, so the refresh kept every package it had
refreshed in the most used list forever. Fetches under the new
handler.MirrorFetch context set mirrored_at, and a file whose last fetch
was a mirror's counts only once a client downloads it.
security.debian.org keeps files under pool/updates/main while its
InRelease lists main/binary-*, so taking the first segment after pool/
as the component found no index and security suites were never
refreshed. Take the first or second segment, whichever the release
file lists indexes for.
A tag found in the cache after the refresh failed was reported as
mirrored with no sign it might be stale, though tags can move. Keep
serving it, as a cached commit is, but log a warning naming the tag,
the commit and the refresh error.
Under cooldown, a tarball for a version the packument doesn't list is
decided on a fresh fetch of the packument. That check had gaps:
- A stale copy served because upstream failed counted as fetched, so
the version went through with no publish time. It now triggers the
refetch like a copy within its TTL.
- A packument just stored from upstream counted as cached, so every
request for an unlisted version fetched it twice. Only a stored row
read within its TTL counts now.
- A stored refetch that joined another caller's fetch could get that
caller's stale fallback and take it as upstream's answer. The stored
lookup now reports stale results, and a stale refetch is refused.
- A refused refetch dropped the upstream error, so a rate-limit hold
answered 502 instead of 503 with Retry-After, and a 404 was lost.
Stale results now carry the upstream error, and the refusal wraps it.
The relay turned a redirect into a failed fetch, which counted as the
upstream failing rather than refusing, so the cached copy was still
served and no refusal was recorded. Report it as a refusal, as for a
redirected ref listing.
stale_max_age plus the upload-pack grace overflowed for a stale_max_age
near the largest duration, which made every upload-pack request check
the upstream. Cap the sum instead.
A refusal was cleared only by a refresh that succeeded as a whole. When
the upstream listed the refs again but the fetch that followed failed,
say because the cache was full or the fetch timed out, the old refusal
stayed and the repository was still refused, though the upstream had
just confirmed access. Clear it as soon as the ref check succeeds.
The index lookup always used a cached InRelease over Release, so a
stale InRelease decided which index was served to a client that had
just fetched a fresh Release, and it stopped at the nearest cached
release file even when that file, such as the legacy per-component
Release, listed nothing for the path.
Skip release files that don't list the path, and where both InRelease
and Release do, use the one fetched last: apt reads InRelease and
debootstrap Release, and a client asking for an index has most likely
just fetched one of them.
Offline mode serves the cached copy without asking the database once
it is known to be unreachable, but derived staleness from the row's
age. With metadata_ttl 0 that copy had no Warning header. It is
always stale, since upstream was not asked.
A version that wasn't a lowercase full-length hex ID was taken for a
tag, so an uppercase commit ID failed as a missing tag. Lowercase a
full-length hex version first. A shorter hex version that names no tag
now fails saying it is neither a full commit ID nor a tag.
With metadata_ttl 0 another instance can replace the blob between
reading the row and reading the blob, and the fallback then served
the new bytes under the old row's ETag, so a client holding that ETag
got a wrong 304. Serve the blob without the row's validators when it
doesn't match the row's digest.
The refresh of a row confirmed by a 304 had moved ahead of validate,
so a stored document validate then rejected was still recorded as
fetched now. Validate first, as before.
Mirroring img:v1@sha512:<X> compared the sha512 digest with the sha256
digest the tag resolved to, so an unmoved tag was reported as moved
and its manifest walked twice. Key seen manifests and the returned
digest by sha256, as tags are.
applicable compared the caller's ecosystem raw against the normalised
configured keys, so a caller passing an alias such as go or rubygems
skipped a scanner restricted to golang or gem. Normalise it first.
NewGroup now also rejects unknown names the way Validate does, through
a shared ScannerEcosystemKey, instead of building a scanner that can
never match.
Looking for .gitmodules took any error from git cat-file as the file
not being there, so a mirror whose context was cancelled reported
success with no submodules. Use rev-parse --verify --quiet, whose exit
status 1 means the path is missing, and return other errors.
A .gitmodules git config can't parse failed the mirror of a commit that
is cached all the same. Log a warning and skip its submodules instead,
as an unserved submodule host is skipped.
ParseSize truncated fractional sizes such as 0.5B to 0, and a
storage.max_size of 0 means unlimited. Reject a non-zero size that
would round down to 0 bytes.
A SNAPSHOT file named without a timestamp was dropped from the cache
before it was fetched again, so an unreachable upstream gave a 502
instead of the last build, concurrent requests could drop each
other's fresh copy, and HEAD downloaded the file.
Fetch it first under the coalescing key and let the new copy replace
the cached one once stored. When the fetch fails for any reason but
the file being missing or blocked, serve the cached copy. HEAD is
answered from cache.
The existing test only runs where the file system ignores case, so it
was skipped on Linux, and it passed if the second spelling had no entry
at all. Add a test that sets foldCase directly and checks both
spellings share one entry, require the entry in the existing test, and
work out the repository key once in acquire.
An upstream such as Nexus served from /repository/pypi links projects
as /repository/pypi/simple/<name>/, which the rewrite missed. Match
links under the configured upstream's path, in single-quoted hrefs as
well as double-quoted ones, and drop the upstream's ETag, which names
the body before the rewrite.
An upload-pack request returned a stored refusal without checking the
upstream, so a client that never asks for info/refs, such as a protocol
v2 client sending ls-refs and fetch directly, stayed refused after a
passing 403 or 404. Refresh first while a refusal stands, as for a copy
past stale_max_age, and refuse only if the refusal remains.
Composer falls back to the default repository's notify-batch when a
version's notification-url is null, which would report installs of a
routed private package to Packagist. Replace null with an empty string
as is already done for pushed versions.
Also drop the pushed-version check's local url variable, which shadowed
net/url, and test that a pushed manifest's own URL is kept.
The cache key for an RPM package is its request path, so a path with
an empty or "." segment made a second entry for the same upstream
file. The path is now cleaned before it is used as the key and the
upstream URL.
Before the strict parser, PROXY_TRUST_FORWARDED_FOR=no meant false.
With only strconv.ParseBool's forms accepted, no was ignored and a
setting enabled in the config file stayed on. Accept yes/no/on/off in
any case for every boolean variable, the Gradle read-only one included.
Escaping _ in the search pattern stopped typing_extensions from finding
typing-extensions, which worked before because _ matched any single
character. Escape only % and the escape character.
The cache keys Maven artifacts as group:artifact, so org:foo/bar and
org/foo:bar both came out as org:foo:bar and shared one entry. Such
paths are now proxied without caching, like dotted group directories.
This reverts commit fc0e94b. The cleanup stopped at the first segment
that looked like a fetch id, but named Maven repositories put a hex
upstream hash in the filename, so it stopped early. For objects in the
old layout, with no fetch directory, the walk could remove version or
package directories under a concurrent fetch. Leftover empty
directories will be handled separately.
APK and Arch packages, and now RPM ones, have filenames with directories
in them, which put the object below its fetch directory. Delete only
removed the object's own directory when it was the fetch directory, so
those directories stayed behind. It now removes them up to and including
the fetch directory while they are empty.
A check that ran past refresh_wait or failed served the cached copy, also
right after the upstream had refused the repository. With protocol v0 the
refs are in the info/refs response itself, so they were advertised. The
stored refusal now refuses those too, until a check succeeds.
Composer reads "notification-url": null as missing and reports installs
to notify-batch, so a pushed manifest with null now gets an empty one too.
The docs say a pushed manifest's own notification-url is kept.
Git LFS objects are scanned under the git key, which the new ecosystem
check rejected, so a scanner restricted to ecosystems: [git] would have
stopped the proxy from starting.
Non-timestamped SNAPSHOT files went straight to upstream, past the
denylist, cooldown, scanning and the Plugin Portal fallback. They now take
the normal download path, with the cached copy dropped first so each
request still gets the current build.
A database error while refreshing a verified 304 no longer forces a full
download: the bytes were checked against the row and upstream confirmed
them.
Without cooldown, maven-metadata.xml and its .md5/.sha1/.sha256/.sha512
files were cached under separate keys with separate TTLs, so a deploy
between the two fetches left the checksum describing another document
and strict builds failed. The checksum files are now computed from the
cached metadata, as the cooldown path already did. The metadata is
also written with the validators of the document served.
SNAPSHOT files named with SNAPSHOT rather than a build timestamp
(lib-1.0-SNAPSHOT.jar, as JitPack and non-unique snapshot repositories
publish them) matched the artifact rules and were cached for good, so
the first build was served forever. They are now passed through
uncached. Timestamped SNAPSHOT files never change and stay cached.
InRelease/Release and repomd.xml had their own metadata cache entries
and TTLs, independent of the Packages, Sources or plainly named
primary.xml.gz files they list. Within a TTL a client could get the
cached release file and a freshly fetched index, or a fresh release
file and a cached index, and apt then failed with "Hash Sum mismatch"
(repositories without by-hash, or RPM repositories built with
--simple-md-filenames).
An index file that the cached release file lists is now served by
that listing: the cached copy while its digest matches the listed
sha256, whatever its age, and otherwise a copy fetched from upstream.
If upstream's copy does not match either, the repository has moved on
since the release file was cached; that copy is served and the
release file is expired so the next update gets a matching pair.
The denylist kept the PURL namespace, but the Debian and RPM handlers
build pkg:deb/{name} and pkg:rpm/{name} without a vendor, while Alpine
and Arch build pkg:apk/alpine/{name} and pkg:alpm/arch/{name} with one.
So standard entries such as pkg:deb/debian/curl@... or
pkg:rpm/fedora/curl@..., and pkg:apk/busybox@... or pkg:alpm/glibc@...,
never matched. Drop the namespace of distro types on both sides, as
retention and the inventory already do, using packageurl's list of
those types.
Debian and RPM handlers read versions from filenames, which leave out
the epoch, so an entry such as pkg:deb/foo@1:2.3-1 never matched
foo_2.3-1_amd64.deb. An entry with an epoch now also covers the
version without it.
The Conda handler divided every repodata timestamp by 1000, assuming
milliseconds. Some entries (a few in conda-forge noarch) use seconds,
so their publish time came out in 1970 and they were never held back
by cooldown. Read a timestamp as milliseconds only when it is above
253402300799, as conda does, in one helper used by the repodata
filter, the shard filter and the download index.
exec.CommandContext kills only the git parent. Its git-remote-http helper
kept its relay connection and went on downloading for up to
http.lowSpeedTime after a client disconnect, Close or fetchTimeout. Each
read it relayed called cover, which reserved the relayed total again on a
reservation already settled or released, so c.reserved grew for good and
could evict repositories, and the helpers outlived the fetch cap.
Tie each relay route to the fetch's context and cancel it when the fetch
returns, so the upstream request ends and the helper's connection is cut.
Make cover refuse a reservation that has been given back.
Only GET info/refs checked the upstream. A POST git-upload-pack was served
from the cached copy unchecked, so a client posting ls-refs and fetch
directly (protocol v2 allows it) kept getting a repository that was
deleted or made private upstream, past stale_max_age, indefinitely.
Remember when a refresh finds a repository missing or refused upstream,
and refuse upload-pack requests for it until a refresh succeeds. Refresh
first when the copy was last checked more than a minute past
stale_max_age ago; the minute keeps a normal clone (info/refs, then
upload-pack) at one upstream check even with stale_max_age 0. Mirror
refreshes on its own and skips the check.
SearchPackages and CountSearchResults matched names with
p.name LIKE '%query%'. LIKE ignores ASCII case on SQLite but not on
Postgres, so the same search found different packages depending on the
database. The % and _ in a query were also taken as wildcards. Compare
LOWER(name) with LOWER(pattern) and escape the wildcards.
Composer reports installs to the repository's notify-batch URL unless a
version has its own notification-url, and the proxy's packages.json sets
notify-batch to the upstream. Routed packages get an empty
notification-url for this reason, but pushed and hosted-only versions
did not, so installs of private packages were reported to packagist.org.
Give pushed versions the same empty notification-url.
