Each match searched the rest of the buffer for every replacement. The
root index has no single-quoted links, so after every double-quoted one
the other pattern scanned to the end again, making each chunk quadratic
in its links. Each replacement's next match is now kept and only
searched for again once a replacement passes it (50,000 links: 18.6 to
818 MB/s).
A tag found in the cache after the refresh failed was reported as
mirrored with no sign it might be stale, though tags can move. Keep
serving it, as a cached commit is, but log a warning naming the tag,
the commit and the refresh error.
Under cooldown, a tarball for a version the packument doesn't list is
decided on a fresh fetch of the packument. That check had gaps:
- A stale copy served because upstream failed counted as fetched, so
the version went through with no publish time. It now triggers the
refetch like a copy within its TTL.
- A packument just stored from upstream counted as cached, so every
request for an unlisted version fetched it twice. Only a stored row
read within its TTL counts now.
- A stored refetch that joined another caller's fetch could get that
caller's stale fallback and take it as upstream's answer. The stored
lookup now reports stale results, and a stale refetch is refused.
- A refused refetch dropped the upstream error, so a rate-limit hold
answered 502 instead of 503 with Retry-After, and a 404 was lost.
Stale results now carry the upstream error, and the refusal wraps it.
The relay turned a redirect into a failed fetch, which counted as the
upstream failing rather than refusing, so the cached copy was still
served and no refusal was recorded. Report it as a refusal, as for a
redirected ref listing.
stale_max_age plus the upload-pack grace overflowed for a stale_max_age
near the largest duration, which made every upload-pack request check
the upstream. Cap the sum instead.
A refusal was cleared only by a refresh that succeeded as a whole. When
the upstream listed the refs again but the fetch that followed failed,
say because the cache was full or the fetch timed out, the old refusal
stayed and the repository was still refused, though the upstream had
just confirmed access. Clear it as soon as the ref check succeeds.
A version that wasn't a lowercase full-length hex ID was taken for a
tag, so an uppercase commit ID failed as a missing tag. Lowercase a
full-length hex version first. A shorter hex version that names no tag
now fails saying it is neither a full commit ID nor a tag.
Mirroring img:v1@sha512:<X> compared the sha512 digest with the sha256
digest the tag resolved to, so an unmoved tag was reported as moved
and its manifest walked twice. Key seen manifests and the returned
digest by sha256, as tags are.
applicable compared the caller's ecosystem raw against the normalised
configured keys, so a caller passing an alias such as go or rubygems
skipped a scanner restricted to golang or gem. Normalise it first.
NewGroup now also rejects unknown names the way Validate does, through
a shared ScannerEcosystemKey, instead of building a scanner that can
never match.
Looking for .gitmodules took any error from git cat-file as the file
not being there, so a mirror whose context was cancelled reported
success with no submodules. Use rev-parse --verify --quiet, whose exit
status 1 means the path is missing, and return other errors.
A .gitmodules git config can't parse failed the mirror of a commit that
is cached all the same. Log a warning and skip its submodules instead,
as an unserved submodule host is skipped.
ParseSize truncated fractional sizes such as 0.5B to 0, and a
storage.max_size of 0 means unlimited. Reject a non-zero size that
would round down to 0 bytes.
server.Start logs the warnings LoadFromEnv collects, but mirror never
calls it, so an unparseable boolean there was dropped without a word.
Print them to stderr before validating the config. stats and hosted
don't load the config file and aren't affected.
An upstream such as Nexus served from /repository/pypi links projects
as /repository/pypi/simple/<name>/, which the rewrite missed. Match
links under the configured upstream's path, in single-quoted hrefs as
well as double-quoted ones, and drop the upstream's ETag, which names
the body before the rewrite.
An upload-pack request returned a stored refusal without checking the
upstream, so a client that never asks for info/refs, such as a protocol
v2 client sending ls-refs and fetch directly, stayed refused after a
passing 403 or 404. Refresh first while a refusal stands, as for a copy
past stale_max_age, and refuse only if the refusal remains.
Composer falls back to the default repository's notify-batch when a
version's notification-url is null, which would report installs of a
routed private package to Packagist. Replace null with an empty string
as is already done for pushed versions.
Also drop the pushed-version check's local url variable, which shadowed
net/url, and test that a pushed manifest's own URL is kept.
Before the strict parser, PROXY_TRUST_FORWARDED_FOR=no meant false.
With only strconv.ParseBool's forms accepted, no was ignored and a
setting enabled in the config file stayed on. Accept yes/no/on/off in
any case for every boolean variable, the Gradle read-only one included.
Escaping _ in the search pattern stopped typing_extensions from finding
typing-extensions, which worked before because _ matched any single
character. Escape only % and the escape character.
A check that ran past refresh_wait or failed served the cached copy, also
right after the upstream had refused the repository. With protocol v0 the
refs are in the info/refs response itself, so they were advertised. The
stored refusal now refuses those too, until a check succeeds.
Composer reads "notification-url": null as missing and reports installs
to notify-batch, so a pushed manifest with null now gets an empty one too.
The docs say a pushed manifest's own notification-url is kept.
Git LFS objects are scanned under the git key, which the new ecosystem
check rejected, so a scanner restricted to ecosystems: [git] would have
stopped the proxy from starting.
The denylist kept the PURL namespace, but the Debian and RPM handlers
build pkg:deb/{name} and pkg:rpm/{name} without a vendor, while Alpine
and Arch build pkg:apk/alpine/{name} and pkg:alpm/arch/{name} with one.
So standard entries such as pkg:deb/debian/curl@... or
pkg:rpm/fedora/curl@..., and pkg:apk/busybox@... or pkg:alpm/glibc@...,
never matched. Drop the namespace of distro types on both sides, as
retention and the inventory already do, using packageurl's list of
those types.
Debian and RPM handlers read versions from filenames, which leave out
the epoch, so an entry such as pkg:deb/foo@1:2.3-1 never matched
foo_2.3-1_amd64.deb. An entry with an epoch now also covers the
version without it.
The Conda handler divided every repodata timestamp by 1000, assuming
milliseconds. Some entries (a few in conda-forge noarch) use seconds,
so their publish time came out in 1970 and they were never held back
by cooldown. Read a timestamp as milliseconds only when it is above
253402300799, as conda does, in one helper used by the repodata
filter, the shard filter and the download index.
exec.CommandContext kills only the git parent. Its git-remote-http helper
kept its relay connection and went on downloading for up to
http.lowSpeedTime after a client disconnect, Close or fetchTimeout. Each
read it relayed called cover, which reserved the relayed total again on a
reservation already settled or released, so c.reserved grew for good and
could evict repositories, and the helpers outlived the fetch cap.
Tie each relay route to the fetch's context and cancel it when the fetch
returns, so the upstream request ends and the helper's connection is cut.
Make cover refuse a reservation that has been given back.
Only GET info/refs checked the upstream. A POST git-upload-pack was served
from the cached copy unchecked, so a client posting ls-refs and fetch
directly (protocol v2 allows it) kept getting a repository that was
deleted or made private upstream, past stale_max_age, indefinitely.
Remember when a refresh finds a repository missing or refused upstream,
and refuse upload-pack requests for it until a refresh succeeds. Refresh
first when the copy was last checked more than a minute past
stale_max_age ago; the minute keeps a normal clone (info/refs, then
upload-pack) at one upstream check even with stale_max_age 0. Mirror
refreshes on its own and skips the check.
SearchPackages and CountSearchResults matched names with
p.name LIKE '%query%'. LIKE ignores ASCII case on SQLite but not on
Postgres, so the same search found different packages depending on the
database. The % and _ in a query were also taken as wildcards. Compare
LOWER(name) with LOWER(pattern) and escape the wildcards.
Composer reports installs to the repository's notify-batch URL unless a
version has its own notification-url, and the proxy's packages.json sets
notify-batch to the upstream. Routed packages get an empty
notification-url for this reason, but pushed and hosted-only versions
did not, so installs of private packages were reported to packagist.org.
Give pushed versions the same empty notification-url.
pypi.org's HTML root index links each project as /simple/<name>/. The
proxy relayed it unchanged, but it is mounted at /pypi, so following a
link went to /simple/<name>/ on the proxy and got 404. Rewrite those
hrefs to the proxy's /pypi/simple/ while streaming the response, since
the index is tens of megabytes. The JSON index has no links and is left
alone.
npm: a tarball request for a version the cached packument did not list
yet (cached within metadata_ttl before the release) found no publish
time and was let through cooldown, and fetched without integrity. When
cooldown applies and the cached packument lacks the version, fetch it
again and decide on what upstream lists now; if it can't be fetched,
fail as when a publish time can't be read.
npm: cooldown only moved dist-tags.latest. Other tags kept naming
versions in cooldown, while the denylist removes such tags. Cooldown
now removes them too, and drops latest when no version is left, in
both the in-memory and streamed rewrites.
PyPI: the simple JSON rewrite removed filtered files but left their
versions in the PEP 700 versions list. Remove them there too.
pub: latest was never rewritten, so its archive_url pointed at pub.dev,
it kept naming a version in cooldown when no version was left, and the
replacement was the last list entry, which may be a prerelease. Point
latest's archive_url at the proxy, replace a held back latest with the
newest stable version left (or the newest prerelease), and drop it
when nothing is left.
ParseSize accepted "-5GB", "NaNGB", "InfGB" and "1e30GB", the last
two clamping to the largest int64. A negative or NaN storage.max_size
passed validation, and eviction treats a limit of zero or less as
unlimited.
Reject negative, NaN, infinite and overflowing sizes in ParseSize, and
reject negative metadata_ttl and storage.direct_serve_ttl at startup.
scanning.scanners[].ecosystems was never checked, and the scanner group
compared it exactly against the keys handlers pass. Natural names such
as go, docker and rubygems passed validation but never matched the
handler keys golang, oci and gem, so a block-mode scanner restricted to
them silently never ran.
Map common aliases to the handler keys, reject unknown names at
startup, and normalise the names again when building the group.
The blob route and blobPathPattern only matched sha256 digests, so a
sha512 blob got 404 although the fetch path can verify sha512. The image
mirror rejected sha512 digests in index entries, configs and layers for
the same reason. Accept sha512 in both, with exact digest lengths.
A HEAD for an uncached manifest by digest passed on whatever digest the
upstream named. GET checks the body against the requested digest; HEAD
now answers DIGEST_INVALID when the upstream names a different digest
of the same algorithm.
Every boolean PROXY_* variable was true only for exactly "true" or
"1". Anything else, including "True", "TRUE" or a typo, silently
meant false, so PROXY_SCANNING_ENABLED=True turned scanning off and
validation still passed.
Parse them with strconv.ParseBool like
PROXY_GRADLE_BUILD_CACHE_READ_ONLY already does. An unrecognised value
now keeps the current setting and is logged as a warning at startup.
R 4.3 and later on macOS fetch binaries from a two-segment platform path
such as bin/macosx/big-sur-arm64/contrib/4.4/. The router only matched a
single platform segment, so both the PACKAGES indexes and the .tgz
downloads returned 404. Add routes for that layout and include the arch
in the stored version so arm64 and x86_64 builds stay apart.
src/contrib/Meta/archive.rds and current.rds, which remotes and pak use
to install archived versions, had no route either. Serve .rds files under
Meta as cached metadata.
ScanPackument and LookupVersion read with a jsontext.Decoder, skipping
values they don't keep with SkipValue and reading the one LookupVersion
returns with ReadValue, instead of decoding each into a json.RawMessage.
Repeated names and invalid UTF-8 are still accepted as encoding/json
accepts them. Go 1.27 is now required, for encoding/json/jsontext.
Benchmarks for both use the 5,000-version packument the handler benchmarks
build, now shared in internal/metadata/npm/npmbench.
git config --get-regexp exits 1 when nothing matches, which failed the
mirror of a commit with an empty .gitmodules. The submodules are now read
from --list.
