RPM packages were cached under their bare filename. With upstream.rpm
pointing at a mirror root that serves several distributions, Rocky and
Alma both publish bash-5.1.8-9.el9.x86_64.rpm with different bytes, and
whichever was fetched first was served for both, so dnf failed the
checksum or signature check.
Use the path under the upstream as the cache filename, as the APK and
Arch handlers do. Packages cached before this change are no longer
found and are fetched again once; retention still matches them by
package name.
proxy mirror, mirror API jobs and SBOMs take pkg:git/{host}/{path}@{commit}
and cache the repository and commit in the cache /git serves, so a later
clone is a cache hit, also with the upstream down. Part of #403.
The PURL namespace names the upstream.apk repository, and the distro and
arch qualifiers syft writes give the release and architecture. The package
is looked up in the main and community indexes of that release and cached
under the path the APK handler serves it from.
The API handler takes the enrichment calls it uses as an interface, so
the test can answer them without reaching a registry.
Remove FillVersionPublishedAt until the shared cooldown filter calls it.
metadata_max_size bounds what is read into memory, but a packument
streamed from upstream into storage had no limit, and with cooldown was
stored twice. It is now capped by metadata_stream_max_size, 1 GB by
default. A document over the cap fails as an upstream failure: nothing is
stored, and the stored copy, if any, is served instead.
The metadata settings move into configureMetadata to keep serve under
the maintidx limit.
* Accept Swift source repository PURLs in the denylist and retention rules
Closes#613. pkg:swift/github.com/apple/swift-nio, as SBOM tools and GHSA
write Swift packages, is resolved to its registry identity through the
upstream's /identifiers:
- Denylist entries are resolved at startup, and the proxy refuses to start
when one can't be, since a skipped entry would let the version through.
- Retention keys are accepted at validation and kept pending. The
retention loop resolves them before each sweep, and Swift isn't swept
while any is pending, so a rule that keeps a package is never skipped.
The mirror's source URL helper moves to the handler as SwiftSourceURL.
* Resolve Swift source denylist entries in proxy mirror too
serve resolved source-form Swift denylist entries through the upstream,
but proxy mirror built its denylist directly, so such an entry never
matched a mirror download. Both now use server.ConfigureDenylist, and
neither runs when an entry can't be resolved.
* Apply Swift source policies to every listed identity
A Swift registry may list several identities for one source repository,
but a denylist entry or retention rule naming the repository only
covered the first. Denying pkg:swift/github.com/acme/widget@1.0.0 let
acme.widget through when the registry listed other.widget first, and a
never-evict rule cleared the cached apple.swift-nio archive when
other.swift-nio came first.
Both now cover every identity the registry lists. A retention rule that
conflicts with another rule for any of them stays pending, with none
applied. Mirroring a source PURL still downloads the first identity.
Part of #617. The package page marks versions in cooldown with when they
become available, and lists a version only seen in metadata (no artifact
row) only while it is held back. /api/package returns the package's
cooldown and available_at on a held-back version.
FillVersionPublishedAt stores a publish time without overwriting one, for
the cooldown filters to call once they record the versions they hold back.
Two requests recording the same index version in the same millisecond
stored to one path, and the one whose row was not added deleted the
blob the other's row points at. The path now ends in a random suffix.
Also trims comments that restated the PR, notes in the docs that turning
snapshots off leaves them in place, and moves the git cache setup out of
serve, which lint found too long after the merge.
The feed readers move to registries' changes package. The proxy keeps the
cursor and moves it past the changes only once their metadata is expired,
so a failed expiry reads them again.
An hour is long next to a cooldown of a few hours. The TTL is now a 24th
of the shortest cooldown configured, between a minute and an hour, and
the RubyGems and Hex section no longer says a minute.
The package page showed the default cooldown for every ecosystem, also
for Debian, Alpine, OCI and generic files, which have no publish time to
check. It now shows none there, and the dashboard marks an ecosystem
override for one as not supported. The docs list Helm, which applies
cooldown from the index's created times.
pkg:brew PURLs failed with "unsupported ecosystem: brew", although listing their versions worked. Homebrew publishes a formula's bottles as an image on its artifact upstream, which proxy mirror can already fetch, so a brew PURL is now resolved to that image and mirrored like one: the formula's JSON API gives the tag, which carries the revision and the bottles' rebuild, and --platform selects bottles like image platforms.
Listing a formula's versions returned the versioned siblings the Homebrew client adds (openssl@3.6 for openssl@3) as versions of it; they are now left out.
PACKAGES.rds comes from the same host as the tarball, so checking against it
doesn't justify a decoder for R serialization. The duplicate archived
versions are tracked in git-pkgs/registries#119.
proxy mirror pkg:cran/<name>@<version> now fetches the source tarball
from upstream.cran and caches it under the key the CRAN handler reads,
("cran", name, version, name_version.tar.gz), which serves both
/src/contrib/{file} and /src/contrib/Archive/{name}/{file}.
A current version comes from src/contrib and is checked against the
MD5sum in src/contrib/PACKAGES.gz, read once per five minutes. Other
versions come from src/contrib/Archive/{name}/, checked against the
MD5sum in that directory's PACKAGES.rds, an R-serialized character
matrix read by a small decoder. Without a PACKAGES.rds the archived
tarball is fetched unchecked, as the handler fetches it.
Listing versions dropped duplicates: the CRAN client lists each archived
version twice.
