TestMirrorImageKeepsTagMovedByPullDuringWalk moves a tag while a blob is
served. With blobs fetched concurrently, another blob request reads the
manifest map at the same time, which the race detector caught on
Windows.
The image mirror fetched the config and layer blobs of a manifest one
after another, so a run took as long as all layers together. It now
fetches up to three at once, as many as a Docker pull does by default.
For a Home Assistant image (550 MB in ten blobs) that would leave
mostly the time of its largest layer.
packageurl.HandlerPackage returns the ecosystem and name the handlers store
a package URL under. The mirror and the inventory both use it, which
settles where they disagreed: distribution namespaces of apk, alpm, bsd,
deb and rpm are dropped, PyPI and NuGet names use purl's comparison form,
and Maven names are group:artifact as the handler stores them, where the
mirror used group/artifact.
A snapshot was recorded after the response by reading the metadata cache
again, so a refresh that replaced the cached copy in between lost the
version the request served. Record it from the served bytes before
responding instead, and compare a new snapshot with the version current
at its own time, so one recorded late keeps its place in the history.
parseComposerUpstreamIndex now returns an error for a packages.json that
is not a JSON object or has the advisories and filter fields in the wrong
shape, and fetchUpstreamIndex passes it to the metadata cache as validate.
A malformed 200 is no longer stored: with metadata caching the stored
copy is served as when upstream fails, and without it the error keeps
the copy in memory.
AddMetadataSnapshot takes a transaction-scoped advisory lock on the
document before its check and insert, so two replicas recording the same
version can't both read the previous digest and both insert. SQLite runs
one write at a time, so it keeps the single statement.
A fetched packument is now stored under a staging path and checked before
it replaces the cached copy, so an abbreviated one no longer overwrites a
full copy; the full copy is served as when upstream fails. A failure to
store falls back to the in-memory path, as caching is best effort there.
Requests for a single version or dist-tag and the tarball download's
cooldown and integrity lookup read the stored packument as a stream too,
keeping only the one version entry they need.
A cached rewrite made with cooldown on stays right until a version it held
back leaves cooldown, so it now expires then instead of after a fixed
time. Handlers ask cooldown through Proxy.cooldown, which reports each
withheld version to the rewrites running at that moment. A rewrite that
held nothing back is kept for the shortest cooldown, at most a day.
* Add --latest to mirror only the newest version
An unversioned PURL mirrors every version the upstream lists. With
--latest, or "latest": true in a mirror API job, only the newest is
mirrored: the highest by the ecosystem's version ordering, without
yanked or retracted versions and, unless there is nothing else,
pre-releases.
* Pick the latest version with registries.FetchLatestVersion
--latest now uses the shared policy from git-pkgs/registries: the version
the registry advertises as latest where it has one, otherwise the newest
active version by publish time. Deprecated versions are left out too, and
newestVersion and the direct vers dependency go away.
Closes#525. With metadata caching and the rewrite cache on, a packument
is streamed from upstream into storage and rewritten from there in two
passes: the first collects the time map, dist-tags and version names, the
second copies each kept version and rewrites its tarball URL. Neither the
packument nor its rewrite is held in memory uncompressed, so
metadata_max_size no longer limits packuments.
Both passes check the stored bytes against the digest in the cache row, so
a document replaced in between is not rewritten from mixed bytes. Documents
the stream can't handle (abbreviated, no versions object) and rewrite
errors fall back to the in-memory rewrite.
Part of #617. The package page marks versions in cooldown with when they
become available, and lists a version only seen in metadata (no artifact
row) only while it is held back. /api/package returns the package's
cooldown and available_at on a held-back version.
FillVersionPublishedAt stores a publish time without overwriting one, for
the cooldown filters to call once they record the versions they hold back.
Two requests recording the same index version in the same millisecond
stored to one path, and the one whose row was not added deleted the
blob the other's row points at. The path now ends in a random suffix.
Also trims comments that restated the PR, notes in the docs that turning
snapshots off leaves them in place, and moves the git cache setup out of
serve, which lint found too long after the merge.
The feed readers move to registries' changes package. The proxy keeps the
cursor and moves it past the changes only once their metadata is expired,
so a failed expiry reads them again.
A cached rewrite made with cooldown on stays right until a version it held
back leaves cooldown, so it now expires then instead of after a fixed
time. Handlers ask cooldown through Proxy.cooldown, which reports each
withheld version to the rewrites running at that moment. A rewrite that
held nothing back is kept for the shortest cooldown, at most a day.
Packuments list each version's provenance at dist.attestations.url on
registry.npmjs.org. npm takes only the path of that URL and resolves it
against the configured registry, so behind the proxy `npm audit
signatures` asked for /-/npm/v1/attestations/... at the proxy's root,
outside /npm, and could not verify attestations.
The rewrite now points dist.attestations.url at
/npm/-/npm/v1/attestations/{name}@{version}, and the handler serves that
path. It reads the version's attestations URL from the packument and
fetches it from the package's registry, the route's for routed packages;
a URL on another host is not followed and the registry's standard path
is asked instead. Attestations never change once published, so they are
cached as immutable metadata.
Versions blocked by the denylist get 403 and versions held back by
cooldown 404, as their tarballs do. Pushed versions have no upstream
attestations and get 404.
An hour is long next to a cooldown of a few hours. The TTL is now a 24th
of the shortest cooldown configured, between a minute and an hour, and
the RubyGems and Hex section no longer says a minute.
pnpm 12 checks every lockfile entry against the registry on each install
unless trustLockfile is set, so each install reads the packument of every
package in the lockfile. The rewrite cache kept each document twice, plain
and gzipped, and with cooldown on for a minute only, so these packuments
were rewritten and compressed again on every install.
Keep only the gzip copy of a cached rewrite, about an eighth of the
document, and decompress it for clients and handlers that need the plain
bytes. Keep rewrites made with cooldown on for an hour. Answer HEAD for
packuments, which pnpm 12 sends before each GET.
The package page showed the default cooldown for every ecosystem, also
for Debian, Alpine, OCI and generic files, which have no publish time to
check. It now shows none there, and the dashboard marks an ecosystem
override for one as not supported. The docs list Helm, which applies
cooldown from the index's created times.
pkg:brew PURLs failed with "unsupported ecosystem: brew", although listing their versions worked. Homebrew publishes a formula's bottles as an image on its artifact upstream, which proxy mirror can already fetch, so a brew PURL is now resolved to that image and mirrored like one: the formula's JSON API gives the tag, which carries the revision and the bottles' rebuild, and --platform selects bottles like image platforms.
Listing a formula's versions returned the versioned siblings the Homebrew client adds (openssl@3.6 for openssl@3) as versions of it; they are now left out.
PACKAGES.rds comes from the same host as the tarball, so checking against it
doesn't justify a decoder for R serialization. The duplicate archived
versions are tracked in git-pkgs/registries#119.
proxy mirror pkg:cran/<name>@<version> now fetches the source tarball
from upstream.cran and caches it under the key the CRAN handler reads,
("cran", name, version, name_version.tar.gz), which serves both
/src/contrib/{file} and /src/contrib/Archive/{name}/{file}.
A current version comes from src/contrib and is checked against the
MD5sum in src/contrib/PACKAGES.gz, read once per five minutes. Other
versions come from src/contrib/Archive/{name}/, checked against the
MD5sum in that directory's PACKAGES.rds, an R-serialized character
matrix read by a small decoder. Without a PACKAGES.rds the archived
tarball is fetched unchecked, as the handler fetches it.
Listing versions dropped duplicates: the CRAN client lists each archived
version twice.
Serving packages.json fetched the upstream's on every request with
cache_metadata off, and after each TTL with it on, so every Composer
command waited on the upstream and on http_timeout when it was down.
The parsed index is now kept for the metadata TTL, or an hour when it
is 0, and a failed fetch is not retried within that time.
