A file named {artifact}-{version} under a dotted group directory, such as
org.foo/bar/1.0/bar-1.0-linux.exe, was not recognised as a coordinate file
and so was proxied past the checks. isCoordinateFile no longer checks the
group, so handleDownload answers 404 for it. The docs say which files get
the 404, and the Range test covers a stale If-Range.
A file whose group directories hold a dot or colon was proxied without
caching, which skipped the denylist, scan and cooldown. Such paths aren't
in the Maven layout, so they now get 404. Directory listings and catalogs
still go through proxyUpstream, which shares its error handling with
Proxy.ProxyUpstream in relayUpstream.
CopyObject takes the reader and decodes it with jsontext, accepting
repeated names and invalid UTF-8 as encoding/json did. Keys are written
with the bytes they were read with instead of being re-encoded. Member
gets Skip, which reads past a value without keeping it, and npm's
RewriteStream skips dropped versions and duplicates with it.
conda-forge noarch repodata.json (195 MB): filtering 0.89 to 0.62 s and
379 to 77 MB allocated, the timestamp index 0.84 to 0.61 s. The next
packument (32 MB): RewriteStream 69 to 34 ms and 38 to 3.5 MB.
Each match searched the rest of the buffer for every replacement. The
root index has no single-quoted links, so after every double-quoted one
the other pattern scanned to the end again, making each chunk quadratic
in its links. Each replacement's next match is now kept and only
searched for again once a replacement passes it (50,000 links: 18.6 to
818 MB/s).
Paths the proxy doesn't cache, such as ones with a dotted group
directory, went to the primary upstream only, so a file available only on
the Gradle Plugin Portal returned 404. They now try the Plugin Portal on a
404, as cached files do.
Under cooldown, a tarball for a version the packument doesn't list is
decided on a fresh fetch of the packument. That check had gaps:
- A stale copy served because upstream failed counted as fetched, so
the version went through with no publish time. It now triggers the
refetch like a copy within its TTL.
- A packument just stored from upstream counted as cached, so every
request for an unlisted version fetched it twice. Only a stored row
read within its TTL counts now.
- A stored refetch that joined another caller's fetch could get that
caller's stale fallback and take it as upstream's answer. The stored
lookup now reports stale results, and a stale refetch is refused.
- A refused refetch dropped the upstream error, so a rate-limit hold
answered 502 instead of 503 with Retry-After, and a 404 was lost.
Stale results now carry the upstream error, and the refusal wraps it.
Mirroring img:v1@sha512:<X> compared the sha512 digest with the sha256
digest the tag resolved to, so an unmoved tag was reported as moved
and its manifest walked twice. Key seen manifests and the returned
digest by sha256, as tags are.
An upstream such as Nexus served from /repository/pypi links projects
as /repository/pypi/simple/<name>/, which the rewrite missed. Match
links under the configured upstream's path, in single-quoted hrefs as
well as double-quoted ones, and drop the upstream's ETag, which names
the body before the rewrite.
Composer falls back to the default repository's notify-batch when a
version's notification-url is null, which would report installs of a
routed private package to Packagist. Replace null with an empty string
as is already done for pushed versions.
Also drop the pushed-version check's local url variable, which shadowed
net/url, and test that a pushed manifest's own URL is kept.
The cache key for an RPM package is its request path, so a path with
an empty or "." segment made a second entry for the same upstream
file. The path is now cleaned before it is used as the key and the
upstream URL.
The cache keys Maven artifacts as group:artifact, so org:foo/bar and
org/foo:bar both came out as org:foo:bar and shared one entry. Such
paths are now proxied without caching, like dotted group directories.
Composer reads "notification-url": null as missing and reports installs
to notify-batch, so a pushed manifest with null now gets an empty one too.
The docs say a pushed manifest's own notification-url is kept.
The Conda handler divided every repodata timestamp by 1000, assuming
milliseconds. Some entries (a few in conda-forge noarch) use seconds,
so their publish time came out in 1970 and they were never held back
by cooldown. Read a timestamp as milliseconds only when it is above
253402300799, as conda does, in one helper used by the repodata
filter, the shard filter and the download index.
Composer reports installs to the repository's notify-batch URL unless a
version has its own notification-url, and the proxy's packages.json sets
notify-batch to the upstream. Routed packages get an empty
notification-url for this reason, but pushed and hosted-only versions
did not, so installs of private packages were reported to packagist.org.
Give pushed versions the same empty notification-url.
pypi.org's HTML root index links each project as /simple/<name>/. The
proxy relayed it unchanged, but it is mounted at /pypi, so following a
link went to /simple/<name>/ on the proxy and got 404. Rewrite those
hrefs to the proxy's /pypi/simple/ while streaming the response, since
the index is tens of megabytes. The JSON index has no links and is left
alone.
parsePath joins the group directories with dots, and the cache filename
is the file's base name, so org.foo/bar/1.0/bar-1.0.jar and
org/foo/bar/1.0/bar-1.0.jar shared one cache entry. Only upstreams that
serve content at such directories, such as raw or hosted repositories,
are affected.
The Maven layout splits a group at its dots, so a group directory never
contains a dot or is empty. Treat such paths as outside the layout and
proxy them uncached, which keeps every existing cache entry valid.
npm: a tarball request for a version the cached packument did not list
yet (cached within metadata_ttl before the release) found no publish
time and was let through cooldown, and fetched without integrity. When
cooldown applies and the cached packument lacks the version, fetch it
again and decide on what upstream lists now; if it can't be fetched,
fail as when a publish time can't be read.
npm: cooldown only moved dist-tags.latest. Other tags kept naming
versions in cooldown, while the denylist removes such tags. Cooldown
now removes them too, and drops latest when no version is left, in
both the in-memory and streamed rewrites.
PyPI: the simple JSON rewrite removed filtered files but left their
versions in the PEP 700 versions list. Remove them there too.
pub: latest was never rewritten, so its archive_url pointed at pub.dev,
it kept naming a version in cooldown when no version was left, and the
replacement was the last list entry, which may be a prerelease. Point
latest's archive_url at the proxy, replace a held back latest with the
newest stable version left (or the newest prerelease), and drop it
when nothing is left.
The blob route and blobPathPattern only matched sha256 digests, so a
sha512 blob got 404 although the fetch path can verify sha512. The image
mirror rejected sha512 digests in index entries, configs and layers for
the same reason. Accept sha512 in both, with exact digest lengths.
A HEAD for an uncached manifest by digest passed on whatever digest the
upstream named. GET checks the body against the requested digest; HEAD
now answers DIGEST_INVALID when the upstream names a different digest
of the same algorithm.
RPM packages were cached under their bare filename. With upstream.rpm
pointing at a mirror root that serves several distributions, Rocky and
Alma both publish bash-5.1.8-9.el9.x86_64.rpm with different bytes, and
whichever was fetched first was served for both, so dnf failed the
checksum or signature check.
Use the path under the upstream as the cache filename, as the APK and
Arch handlers do. Packages cached before this change are no longer
found and are fetched again once; retention still matches them by
package name.
R 4.3 and later on macOS fetch binaries from a two-segment platform path
such as bin/macosx/big-sur-arm64/contrib/4.4/. The router only matched a
single platform segment, so both the PACKAGES indexes and the .tgz
downloads returned 404. Add routes for that layout and include the arch
in the stored version so arm64 and x86_64 builds stay apart.
src/contrib/Meta/archive.rds and current.rds, which remotes and pak use
to install archived versions, had no route either. Serve .rds files under
Meta as cached metadata.
ScanPackument and LookupVersion read with a jsontext.Decoder, skipping
values they don't keep with SkipValue and reading the one LookupVersion
returns with ReadValue, instead of decoding each into a json.RawMessage.
Repeated names and invalid UTF-8 are still accepted as encoding/json
accepts them. Go 1.27 is now required, for encoding/json/jsontext.
Benchmarks for both use the 5,000-version packument the handler benchmarks
build, now shared in internal/metadata/npm/npmbench.
A pkg:git version can now be a tag as well as a full commit ID. With a
version, the submodules the commit records are mirrored at the commits it
pins, from the upstream.git hosts serving them; one no host serves is
skipped with a warning.
Mirroring a commit on no branch or tag again writes a fresh wanted ref, so
its 7 days count from the last mirror run. A cached repository whose refresh
fetched new commits is no longer reported as cached.
proxy mirror, mirror API jobs and SBOMs take pkg:git/{host}/{path}@{commit}
and cache the repository and commit in the cache /git serves, so a later
clone is a cache hit, also with the upstream down. Part of #403.
A mirror run that finds a blob cached records it in mirrored_at, which
retention counts as use. LRU eviction keeps using last_accessed_at, so
#508's decision not to count mirror runs as cache hits stands.
Fixes#648.
The PURL namespace names the upstream.apk repository, and the distro and
arch qualifiers syft writes give the release and architecture. The package
is looked up in the main and community indexes of that release and cached
under the path the APK handler serves it from.
The lookup took any non-200 metadata response for a version that isn't
listed, so a Packagist outage or rate limit answered the archive request
with 404. Composer treats that as the version being gone.
Only 404 and 410 now mean the document is missing. A 429 is passed on as
503 with Retry-After while the upstream is held, and any other status
answers 502.
