A file named {artifact}-{version} under a dotted group directory, such as
org.foo/bar/1.0/bar-1.0-linux.exe, was not recognised as a coordinate file
and so was proxied past the checks. isCoordinateFile no longer checks the
group, so handleDownload answers 404 for it. The docs say which files get
the 404, and the Range test covers a stale If-Range.
A file whose group directories hold a dot or colon was proxied without
caching, which skipped the denylist, scan and cooldown. Such paths aren't
in the Maven layout, so they now get 404. Directory listings and catalogs
still go through proxyUpstream, which shares its error handling with
Proxy.ProxyUpstream in relayUpstream.
AppendUnquote reports invalid UTF-8 after replacing it, which failed the
whole copy although the decoder allows it. Key now holds U+FFFD in its
place, as with encoding/json and ScanPackument, and the key is still
written with its original bytes.
CopyObject takes the reader and decodes it with jsontext, accepting
repeated names and invalid UTF-8 as encoding/json did. Keys are written
with the bytes they were read with instead of being re-encoded. Member
gets Skip, which reads past a value without keeping it, and npm's
RewriteStream skips dropped versions and duplicates with it.
conda-forge noarch repodata.json (195 MB): filtering 0.89 to 0.62 s and
379 to 77 MB allocated, the timestamp index 0.84 to 0.61 s. The next
packument (32 MB): RewriteStream 69 to 34 ms and 38 to 3.5 MB.
Each match searched the rest of the buffer for every replacement. The
root index has no single-quoted links, so after every double-quoted one
the other pattern scanned to the end again, making each chunk quadratic
in its links. Each replacement's next match is now kept and only
searched for again once a replacement passes it (50,000 links: 18.6 to
818 MB/s).
Paths the proxy doesn't cache, such as ones with a dotted group
directory, went to the primary upstream only, so a file available only on
the Gradle Plugin Portal returned 404. They now try the Plugin Portal on a
404, as cached files do.
A tag found in the cache after the refresh failed was reported as
mirrored with no sign it might be stale, though tags can move. Keep
serving it, as a cached commit is, but log a warning naming the tag,
the commit and the refresh error.
Under cooldown, a tarball for a version the packument doesn't list is
decided on a fresh fetch of the packument. That check had gaps:
- A stale copy served because upstream failed counted as fetched, so
the version went through with no publish time. It now triggers the
refetch like a copy within its TTL.
- A packument just stored from upstream counted as cached, so every
request for an unlisted version fetched it twice. Only a stored row
read within its TTL counts now.
- A stored refetch that joined another caller's fetch could get that
caller's stale fallback and take it as upstream's answer. The stored
lookup now reports stale results, and a stale refetch is refused.
- A refused refetch dropped the upstream error, so a rate-limit hold
answered 502 instead of 503 with Retry-After, and a 404 was lost.
Stale results now carry the upstream error, and the refusal wraps it.
The relay turned a redirect into a failed fetch, which counted as the
upstream failing rather than refusing, so the cached copy was still
served and no refusal was recorded. Report it as a refusal, as for a
redirected ref listing.
stale_max_age plus the upload-pack grace overflowed for a stale_max_age
near the largest duration, which made every upload-pack request check
the upstream. Cap the sum instead.
A refusal was cleared only by a refresh that succeeded as a whole. When
the upstream listed the refs again but the fetch that followed failed,
say because the cache was full or the fetch timed out, the old refusal
stayed and the repository was still refused, though the upstream had
just confirmed access. Clear it as soon as the ref check succeeds.
A version that wasn't a lowercase full-length hex ID was taken for a
tag, so an uppercase commit ID failed as a missing tag. Lowercase a
full-length hex version first. A shorter hex version that names no tag
now fails saying it is neither a full commit ID nor a tag.
Mirroring img:v1@sha512:<X> compared the sha512 digest with the sha256
digest the tag resolved to, so an unmoved tag was reported as moved
and its manifest walked twice. Key seen manifests and the returned
digest by sha256, as tags are.
applicable compared the caller's ecosystem raw against the normalised
configured keys, so a caller passing an alias such as go or rubygems
skipped a scanner restricted to golang or gem. Normalise it first.
NewGroup now also rejects unknown names the way Validate does, through
a shared ScannerEcosystemKey, instead of building a scanner that can
never match.
Looking for .gitmodules took any error from git cat-file as the file
not being there, so a mirror whose context was cancelled reported
success with no submodules. Use rev-parse --verify --quiet, whose exit
status 1 means the path is missing, and return other errors.
A .gitmodules git config can't parse failed the mirror of a commit that
is cached all the same. Log a warning and skip its submodules instead,
as an unserved submodule host is skipped.
ParseSize truncated fractional sizes such as 0.5B to 0, and a
storage.max_size of 0 means unlimited. Reject a non-zero size that
would round down to 0 bytes.
server.Start logs the warnings LoadFromEnv collects, but mirror never
calls it, so an unparseable boolean there was dropped without a word.
Print them to stderr before validating the config. stats and hosted
don't load the config file and aren't affected.
The existing test only runs where the file system ignores case, so it
was skipped on Linux, and it passed if the second spelling had no entry
at all. Add a test that sets foldCase directly and checks both
spellings share one entry, require the entry in the existing test, and
work out the repository key once in acquire.
An upstream such as Nexus served from /repository/pypi links projects
as /repository/pypi/simple/<name>/, which the rewrite missed. Match
links under the configured upstream's path, in single-quoted hrefs as
well as double-quoted ones, and drop the upstream's ETag, which names
the body before the rewrite.
An upload-pack request returned a stored refusal without checking the
upstream, so a client that never asks for info/refs, such as a protocol
v2 client sending ls-refs and fetch directly, stayed refused after a
passing 403 or 404. Refresh first while a refusal stands, as for a copy
past stale_max_age, and refuse only if the refusal remains.
Composer falls back to the default repository's notify-batch when a
version's notification-url is null, which would report installs of a
routed private package to Packagist. Replace null with an empty string
as is already done for pushed versions.
Also drop the pushed-version check's local url variable, which shadowed
net/url, and test that a pushed manifest's own URL is kept.
The cache key for an RPM package is its request path, so a path with
an empty or "." segment made a second entry for the same upstream
file. The path is now cleaned before it is used as the key and the
upstream URL.
Before the strict parser, PROXY_TRUST_FORWARDED_FOR=no meant false.
With only strconv.ParseBool's forms accepted, no was ignored and a
setting enabled in the config file stayed on. Accept yes/no/on/off in
any case for every boolean variable, the Gradle read-only one included.
Escaping _ in the search pattern stopped typing_extensions from finding
typing-extensions, which worked before because _ matched any single
character. Escape only % and the escape character.
The cache keys Maven artifacts as group:artifact, so org:foo/bar and
org/foo:bar both came out as org:foo:bar and shared one entry. Such
paths are now proxied without caching, like dotted group directories.
This reverts commit fc0e94b. The cleanup stopped at the first segment
that looked like a fetch id, but named Maven repositories put a hex
upstream hash in the filename, so it stopped early. For objects in the
old layout, with no fetch directory, the walk could remove version or
package directories under a concurrent fetch. Leftover empty
directories will be handled separately.
APK and Arch packages, and now RPM ones, have filenames with directories
in them, which put the object below its fetch directory. Delete only
removed the object's own directory when it was the fetch directory, so
those directories stayed behind. It now removes them up to and including
the fetch directory while they are empty.
A check that ran past refresh_wait or failed served the cached copy, also
right after the upstream had refused the repository. With protocol v0 the
refs are in the info/refs response itself, so they were advertised. The
stored refusal now refuses those too, until a check succeeds.
Composer reads "notification-url": null as missing and reports installs
to notify-batch, so a pushed manifest with null now gets an empty one too.
The docs say a pushed manifest's own notification-url is kept.
Git LFS objects are scanned under the git key, which the new ecosystem
check rejected, so a scanner restricted to ecosystems: [git] would have
stopped the proxy from starting.
The denylist kept the PURL namespace, but the Debian and RPM handlers
build pkg:deb/{name} and pkg:rpm/{name} without a vendor, while Alpine
and Arch build pkg:apk/alpine/{name} and pkg:alpm/arch/{name} with one.
So standard entries such as pkg:deb/debian/curl@... or
pkg:rpm/fedora/curl@..., and pkg:apk/busybox@... or pkg:alpm/glibc@...,
never matched. Drop the namespace of distro types on both sides, as
retention and the inventory already do, using packageurl's list of
those types.
Debian and RPM handlers read versions from filenames, which leave out
the epoch, so an entry such as pkg:deb/foo@1:2.3-1 never matched
foo_2.3-1_amd64.deb. An entry with an epoch now also covers the
version without it.
The Conda handler divided every repodata timestamp by 1000, assuming
milliseconds. Some entries (a few in conda-forge noarch) use seconds,
so their publish time came out in 1970 and they were never held back
by cooldown. Read a timestamp as milliseconds only when it is above
253402300799, as conda does, in one helper used by the repodata
filter, the shard filter and the download index.
exec.CommandContext kills only the git parent. Its git-remote-http helper
kept its relay connection and went on downloading for up to
http.lowSpeedTime after a client disconnect, Close or fetchTimeout. Each
read it relayed called cover, which reserved the relayed total again on a
reservation already settled or released, so c.reserved grew for good and
could evict repositories, and the helpers outlived the fetch cap.
Tie each relay route to the fetch's context and cancel it when the fetch
returns, so the upstream request ends and the helper's connection is cut.
Make cover refuse a reservation that has been given back.
Only GET info/refs checked the upstream. A POST git-upload-pack was served
from the cached copy unchecked, so a client posting ls-refs and fetch
directly (protocol v2 allows it) kept getting a repository that was
deleted or made private upstream, past stale_max_age, indefinitely.
Remember when a refresh finds a repository missing or refused upstream,
and refuse upload-pack requests for it until a refresh succeeds. Refresh
first when the copy was last checked more than a minute past
stale_max_age ago; the minute keeps a normal clone (info/refs, then
upload-pack) at one upstream check even with stale_max_age 0. Mirror
refreshes on its own and skips the check.
SearchPackages and CountSearchResults matched names with
p.name LIKE '%query%'. LIKE ignores ASCII case on SQLite but not on
Postgres, so the same search found different packages depending on the
database. The % and _ in a query were also taken as wildcards. Compare
LOWER(name) with LOWER(pattern) and escape the wildcards.
Composer reports installs to the repository's notify-batch URL unless a
version has its own notification-url, and the proxy's packages.json sets
notify-batch to the upstream. Routed packages get an empty
notification-url for this reason, but pushed and hosted-only versions
did not, so installs of private packages were reported to packagist.org.
Give pushed versions the same empty notification-url.
pypi.org's HTML root index links each project as /simple/<name>/. The
proxy relayed it unchanged, but it is mounted at /pypi, so following a
link went to /simple/<name>/ on the proxy and got 404. Rewrite those
hrefs to the proxy's /pypi/simple/ while streaming the response, since
the index is tens of megabytes. The JSON index has no links and is left
alone.
On a case-insensitive file system, as macOS and Windows use by default,
acme/WIDGETS and acme/widgets name the same cache directory but were
tracked as two repositories. Users of one spelling didn't keep the other
from being evicted, so eviction could move the directory away while it
was being read, and each entry had its own write lock, so two refreshes
could fetch into the directory at once.
Check at startup whether the cache directory ignores case and, if so,
track repositories under their lowercased key. The directory and the
upstream URL keep the spelling requested. Case-sensitive file systems,
and so existing caches on Linux, are unchanged.
parsePath joins the group directories with dots, and the cache filename
is the file's base name, so org.foo/bar/1.0/bar-1.0.jar and
org/foo/bar/1.0/bar-1.0.jar shared one cache entry. Only upstreams that
serve content at such directories, such as raw or hosted repositories,
are affected.
The Maven layout splits a group at its dots, so a group directory never
contains a dot or is empty. Treat such paths as outside the layout and
proxy them uncached, which keeps every existing cache entry valid.
npm: a tarball request for a version the cached packument did not list
yet (cached within metadata_ttl before the release) found no publish
time and was let through cooldown, and fetched without integrity. When
cooldown applies and the cached packument lacks the version, fetch it
again and decide on what upstream lists now; if it can't be fetched,
fail as when a publish time can't be read.
npm: cooldown only moved dist-tags.latest. Other tags kept naming
versions in cooldown, while the denylist removes such tags. Cooldown
now removes them too, and drops latest when no version is left, in
both the in-memory and streamed rewrites.
PyPI: the simple JSON rewrite removed filtered files but left their
versions in the PEP 700 versions list. Remove them there too.
pub: latest was never rewritten, so its archive_url pointed at pub.dev,
it kept naming a version in cooldown when no version was left, and the
replacement was the last list entry, which may be a prerelease. Point
latest's archive_url at the proxy, replace a held back latest with the
newest stable version left (or the newest prerelease), and drop it
when nothing is left.
ParseSize accepted "-5GB", "NaNGB", "InfGB" and "1e30GB", the last
two clamping to the largest int64. A negative or NaN storage.max_size
passed validation, and eviction treats a limit of zero or less as
unlimited.
Reject negative, NaN, infinite and overflowing sizes in ParseSize, and
reject negative metadata_ttl and storage.direct_serve_ttl at startup.
scanning.scanners[].ecosystems was never checked, and the scanner group
compared it exactly against the keys handlers pass. Natural names such
as go, docker and rubygems passed validation but never matched the
handler keys golang, oci and gem, so a block-mode scanner restricted to
them silently never ran.
Map common aliases to the handler keys, reject unknown names at
startup, and normalise the names again when building the group.
The blob route and blobPathPattern only matched sha256 digests, so a
sha512 blob got 404 although the fetch path can verify sha512. The image
mirror rejected sha512 digests in index entries, configs and layers for
the same reason. Accept sha512 in both, with exact digest lengths.
A HEAD for an uncached manifest by digest passed on whatever digest the
upstream named. GET checks the body against the requested digest; HEAD
now answers DIGEST_INVALID when the upstream names a different digest
of the same algorithm.
Every boolean PROXY_* variable was true only for exactly "true" or
"1". Anything else, including "True", "TRUE" or a typo, silently
meant false, so PROXY_SCANNING_ENABLED=True turned scanning off and
validation still passed.
Parse them with strconv.ParseBool like
PROXY_GRADLE_BUILD_CACHE_READ_ONLY already does. An unrecognised value
now keeps the current setting and is logged as a warning at startup.
RPM packages were cached under their bare filename. With upstream.rpm
pointing at a mirror root that serves several distributions, Rocky and
Alma both publish bash-5.1.8-9.el9.x86_64.rpm with different bytes, and
whichever was fetched first was served for both, so dnf failed the
checksum or signature check.
Use the path under the upstream as the cache filename, as the APK and
Arch handlers do. Packages cached before this change are no longer
found and are fetched again once; retention still matches them by
package name.
R 4.3 and later on macOS fetch binaries from a two-segment platform path
such as bin/macosx/big-sur-arm64/contrib/4.4/. The router only matched a
single platform segment, so both the PACKAGES indexes and the .tgz
downloads returned 404. Add routes for that layout and include the arch
in the stored version so arm64 and x86_64 builds stay apart.
src/contrib/Meta/archive.rds and current.rds, which remotes and pak use
to install archived versions, had no route either. Serve .rds files under
Meta as cached metadata.
ScanPackument and LookupVersion read with a jsontext.Decoder, skipping
values they don't keep with SkipValue and reading the one LookupVersion
returns with ReadValue, instead of decoding each into a json.RawMessage.
Repeated names and invalid UTF-8 are still accepted as encoding/json
accepts them. Go 1.27 is now required, for encoding/json/jsontext.
Benchmarks for both use the 5,000-version packument the handler benchmarks
build, now shared in internal/metadata/npm/npmbench.
git config --get-regexp exits 1 when nothing matches, which failed the
mirror of a commit with an empty .gitmodules. The submodules are now read
from --list.
A pkg:git version can now be a tag as well as a full commit ID. With a
version, the submodules the commit records are mirrored at the commits it
pins, from the upstream.git hosts serving them; one no host serves is
skipped with a warning.
Mirroring a commit on no branch or tag again writes a fresh wanted ref, so
its 7 days count from the last mirror run. A cached repository whose refresh
fetched new commits is no longer reported as cached.
proxy mirror, mirror API jobs and SBOMs take pkg:git/{host}/{path}@{commit}
and cache the repository and commit in the cache /git serves, so a later
clone is a cache hit, also with the upstream down. Part of #403.
A mirror run that finds a blob cached records it in mirrored_at, which
retention counts as use. LRU eviction keeps using last_accessed_at, so
#508's decision not to count mirror runs as cache hits stands.
Fixes#648.
Drop projectRoots and readFromLockFile; sbom now decides which inferred
roots are the project. sbom is on the merge commit of git-pkgs/sbom#27
until it is tagged.
The PURL namespace names the upstream.apk repository, and the distro and
arch qualifiers syft writes give the release and architecture. The package
is looked up in the main and community indexes of that release and cached
under the path the APK handler serves it from.
Bump sbom to v0.1.8 and replace lockFileRoots with sbom.Roots. Inferred
roots count as the project only when read from a lock file, so an image's
packages that nothing depends on are kept.
The lookup took any non-200 metadata response for a version that isn't
listed, so a Packagist outage or rate limit answered the archive request
with 404. Composer treats that as the version being gone.
Only 404 and 410 now mean the document is missing. A 429 is passed on as
503 with Retry-After while the upstream is held, and any other status
answers 502.
The API handler takes the enrichment calls it uses as an interface, so
the test can answer them without reaching a registry.
Remove FillVersionPublishedAt until the shared cooldown filter calls it.
metadata_max_size bounds what is read into memory, but a packument
streamed from upstream into storage had no limit, and with cooldown was
stored twice. It is now capped by metadata_stream_max_size, 1 GB by
default. A document over the cap fails as an upstream failure: nothing is
stored, and the stored copy, if any, is served instead.
The metadata settings move into configureMetadata to keep serve under
the maintidx limit.
The change to ReadMetadataSized and fetchUpstreamMetadata, with its tests,
is now #649. upstreamBodyReader stays for storeUpstreamMetadata, in a file
of its own that #649 adds with the same content.
TestAPKSnapshots_CacheHit covers the cache-hit branch with a metadata TTL
above 0. proxyCachedWithEncoding takes optional hooks for a cache hit and a
fetched body, so the APK handler no longer keeps a copy of it.
ReadMetadataSized took the transport's io.ErrUnexpectedEOF for the end of
a body shorter than its size hint. A document upstream closed before its
Content-Length was served with a 200 and cached, or failed the cooldown
filter with a 502, instead of the cached copy being served. Upstream
bodies are now read through a reader that marks their read errors, and
such a body fails the read as an upstream failure.
FuzzParse checks that an uploaded body never makes the inventory panic
and that every component it records has a package URL, ecosystem and
version. TestSaveProjectConcurrentUploads saves one project from eight
uploads at once and checks that each succeeds and the components of
exactly one remain.
errgroup.WithContext cancelled the blobs being fetched as soon as one
failed. A client pull sharing one of those fetches failed with it, and the
next run had to fetch them again. A failure now only stops further blobs
from starting.
Covers namespaced npm and Maven PURLs, a publish time under an alias of
a package row's ecosystem, and a PURL starting with the orphan prefix,
and checks the per-ecosystem versions against GetCacheStats' total.
A body that ended before its Content-Length was read as complete in
memory: ReadMetadataSized took the transport's io.ErrUnexpectedEOF for
the end of a body shorter than its size hint. The half document was
served and cached, or failed the filter with a 502, instead of the stored
copy being served. Upstream bodies are now read through a reader that
marks their read errors, and such a failure is an upstream failure on
both paths: streamed, it no longer counts as a failure to store, which
had sent the request to the bounded in-memory path.
A stored packument whose bytes no longer match its row's digest is now
fetched again like a missing one.
Tests also cover concurrent requests sharing one fetch, client
revalidation and upstream failures with nothing cached. The stream tests
build cooldown through cooldown.NewPolicy as the server does, since a
bare cooldown.Config parses itself on first use without locking.
A row within its TTL was served without checking its blob, so a missing
blob failed the streamed read and the request fell back to the bounded
in-memory path. A read that finds the stored copy missing now fetches the
packument again into storage, ignoring that row, and reads it once more,
for packument, version and tarball requests.
Cooldown stores the publish time of every version it looks up, also of
packages nobody downloaded, which have no package row. The ecosystem stats
counted those versions as unattributed. They now count under the type of
their package PURL; only a PURL that can't be parsed stays unattributed.
* Accept Swift source repository PURLs in the denylist and retention rules
Closes#613. pkg:swift/github.com/apple/swift-nio, as SBOM tools and GHSA
write Swift packages, is resolved to its registry identity through the
upstream's /identifiers:
- Denylist entries are resolved at startup, and the proxy refuses to start
when one can't be, since a skipped entry would let the version through.
- Retention keys are accepted at validation and kept pending. The
retention loop resolves them before each sweep, and Swift isn't swept
while any is pending, so a rule that keeps a package is never skipped.
The mirror's source URL helper moves to the handler as SwiftSourceURL.
* Resolve Swift source denylist entries in proxy mirror too
serve resolved source-form Swift denylist entries through the upstream,
but proxy mirror built its denylist directly, so such an entry never
matched a mirror download. Both now use server.ConfigureDenylist, and
neither runs when an entry can't be resolved.
* Apply Swift source policies to every listed identity
A Swift registry may list several identities for one source repository,
but a denylist entry or retention rule naming the repository only
covered the first. Denying pkg:swift/github.com/acme/widget@1.0.0 let
acme.widget through when the registry listed other.widget first, and a
never-evict rule cleared the cached apple.swift-nio archive when
other.swift-nio came first.
Both now cover every identity the registry lists. A retention rule that
conflicts with another rule for any of them stays pending, with none
applied. Mirroring a source PURL still downloads the first identity.
purl v0.2.1 keeps the case of Go module paths, so the inventory no
longer lowercases syft's mainModule. Module paths are case-sensitive:
github.com/azure/foo is a dependency of github.com/Azure/foo, not part
of it.
With purl v0.2.1, FullName returns a Swift package as its registry
identity, scope.name. The mirror splits namespace/name again to find
the scope or the source repository, so HandlerPackage keeps that form,
as mirrorTarget did before the two mappings were shared.
TestMirrorImageKeepsTagMovedByPullDuringWalk moves a tag while a blob is
served. With blobs fetched concurrently, another blob request reads the
manifest map at the same time, which the race detector caught on
Windows.
The image mirror fetched the config and layer blobs of a manifest one
after another, so a run took as long as all layers together. It now
fetches up to three at once, as many as a Docker pull does by default.
For a Home Assistant image (550 MB in ten blobs) that would leave
mostly the time of its largest layer.
packageurl.HandlerPackage returns the ecosystem and name the handlers store
a package URL under. The mirror and the inventory both use it, which
settles where they disagreed: distribution namespaces of apk, alpm, bsd,
deb and rpm are dropped, PyPI and NuGet names use purl's comparison form,
and Maven names are group:artifact as the handler stores them, where the
mirror used group/artifact.
A snapshot was recorded after the response by reading the metadata cache
again, so a refresh that replaced the cached copy in between lost the
version the request served. Record it from the served bytes before
responding instead, and compare a new snapshot with the version current
at its own time, so one recorded late keeps its place in the history.
parseComposerUpstreamIndex now returns an error for a packages.json that
is not a JSON object or has the advisories and filter fields in the wrong
shape, and fetchUpstreamIndex passes it to the metadata cache as validate.
A malformed 200 is no longer stored: with metadata caching the stored
copy is served as when upstream fails, and without it the error keeps
the copy in memory.
AddMetadataSnapshot takes a transaction-scoped advisory lock on the
document before its check and insert, so two replicas recording the same
version can't both read the previous digest and both insert. SQLite runs
one write at a time, so it keeps the single statement.
A fetched packument is now stored under a staging path and checked before
it replaces the cached copy, so an abbreviated one no longer overwrites a
full copy; the full copy is served as when upstream fails. A failure to
store falls back to the in-memory path, as caching is best effort there.
Requests for a single version or dist-tag and the tarball download's
cooldown and integrity lookup read the stored packument as a stream too,
keeping only the one version entry they need.
A cached rewrite made with cooldown on stays right until a version it held
back leaves cooldown, so it now expires then instead of after a fixed
time. Handlers ask cooldown through Proxy.cooldown, which reports each
withheld version to the rewrites running at that moment. A rewrite that
held nothing back is kept for the shortest cooldown, at most a day.
* Add --latest to mirror only the newest version
An unversioned PURL mirrors every version the upstream lists. With
--latest, or "latest": true in a mirror API job, only the newest is
mirrored: the highest by the ecosystem's version ordering, without
yanked or retracted versions and, unless there is nothing else,
pre-releases.
* Pick the latest version with registries.FetchLatestVersion
--latest now uses the shared policy from git-pkgs/registries: the version
the registry advertises as latest where it has one, otherwise the newest
active version by publish time. Deprecated versions are left out too, and
newestVersion and the direct vers dependency go away.
Closes#525. With metadata caching and the rewrite cache on, a packument
is streamed from upstream into storage and rewritten from there in two
passes: the first collects the time map, dist-tags and version names, the
second copies each kept version and rewrites its tarball URL. Neither the
packument nor its rewrite is held in memory uncompressed, so
metadata_max_size no longer limits packuments.
Both passes check the stored bytes against the digest in the cache row, so
a document replaced in between is not rewritten from mixed bytes. Documents
the stream can't handle (abbreviated, no versions object) and rewrite
errors fall back to the in-memory rewrite.
Part of #617. The package page marks versions in cooldown with when they
become available, and lists a version only seen in metadata (no artifact
row) only while it is held back. /api/package returns the package's
cooldown and available_at on a held-back version.
FillVersionPublishedAt stores a publish time without overwriting one, for
the cooldown filters to call once they record the versions they hold back.
Two requests recording the same index version in the same millisecond
stored to one path, and the one whose row was not added deleted the
blob the other's row points at. The path now ends in a random suffix.
Also trims comments that restated the PR, notes in the docs that turning
snapshots off leaves them in place, and moves the git cache setup out of
serve, which lint found too long after the merge.
The feed readers move to registries' changes package. The proxy keeps the
cursor and moves it past the changes only once their metadata is expired,
so a failed expiry reads them again.
A cached rewrite made with cooldown on stays right until a version it held
back leaves cooldown, so it now expires then instead of after a fixed
time. Handlers ask cooldown through Proxy.cooldown, which reports each
withheld version to the rewrites running at that moment. A rewrite that
held nothing back is kept for the shortest cooldown, at most a day.
Packuments list each version's provenance at dist.attestations.url on
registry.npmjs.org. npm takes only the path of that URL and resolves it
against the configured registry, so behind the proxy `npm audit
signatures` asked for /-/npm/v1/attestations/... at the proxy's root,
outside /npm, and could not verify attestations.
The rewrite now points dist.attestations.url at
/npm/-/npm/v1/attestations/{name}@{version}, and the handler serves that
path. It reads the version's attestations URL from the packument and
fetches it from the package's registry, the route's for routed packages;
a URL on another host is not followed and the registry's standard path
is asked instead. Attestations never change once published, so they are
cached as immutable metadata.
Versions blocked by the denylist get 403 and versions held back by
cooldown 404, as their tarballs do. Pushed versions have no upstream
attestations and get 404.
An hour is long next to a cooldown of a few hours. The TTL is now a 24th
of the shortest cooldown configured, between a minute and an hour, and
the RubyGems and Hex section no longer says a minute.
pnpm 12 checks every lockfile entry against the registry on each install
unless trustLockfile is set, so each install reads the packument of every
package in the lockfile. The rewrite cache kept each document twice, plain
and gzipped, and with cooldown on for a minute only, so these packuments
were rewritten and compressed again on every install.
Keep only the gzip copy of a cached rewrite, about an eighth of the
document, and decompress it for clients and handlers that need the plain
bytes. Keep rewrites made with cooldown on for an hour. Answer HEAD for
packuments, which pnpm 12 sends before each GET.
The package page showed the default cooldown for every ecosystem, also
for Debian, Alpine, OCI and generic files, which have no publish time to
check. It now shows none there, and the dashboard marks an ecosystem
override for one as not supported. The docs list Helm, which applies
cooldown from the index's created times.
pkg:brew PURLs failed with "unsupported ecosystem: brew", although listing their versions worked. Homebrew publishes a formula's bottles as an image on its artifact upstream, which proxy mirror can already fetch, so a brew PURL is now resolved to that image and mirrored like one: the formula's JSON API gives the tag, which carries the revision and the bottles' rebuild, and --platform selects bottles like image platforms.
Listing a formula's versions returned the versioned siblings the Homebrew client adds (openssl@3.6 for openssl@3) as versions of it; they are now left out.
PACKAGES.rds comes from the same host as the tarball, so checking against it
doesn't justify a decoder for R serialization. The duplicate archived
versions are tracked in git-pkgs/registries#119.
proxy mirror pkg:cran/<name>@<version> now fetches the source tarball
from upstream.cran and caches it under the key the CRAN handler reads,
("cran", name, version, name_version.tar.gz), which serves both
/src/contrib/{file} and /src/contrib/Archive/{name}/{file}.
A current version comes from src/contrib and is checked against the
MD5sum in src/contrib/PACKAGES.gz, read once per five minutes. Other
versions come from src/contrib/Archive/{name}/, checked against the
MD5sum in that directory's PACKAGES.rds, an R-serialized character
matrix read by a small decoder. Without a PACKAGES.rds the archived
tarball is fetched unchecked, as the handler fetches it.
Listing versions dropped duplicates: the CRAN client lists each archived
version twice.
Serving packages.json fetched the upstream's on every request with
cache_metadata off, and after each TTL with it on, so every Composer
command waited on the upstream and on http_timeout when it was down.
The parsed index is now kept for the metadata TTL, or an hour when it
is 0, and a failed fetch is not retried within that time.
- GetProjectsUsingPackage no longer counts each project's components
per row; the package page only needs names and versions. The ?purl=
listing leaves component_count out accordingly.
- The delete button sends to DELETE /ui/api/projects/{project}, the same
handler as the API and still behind the token, so it sits under /ui
like the cache deletes.
- The project page answers 500 when its components can't be read,
instead of showing an empty project.
- DELETE of a path ending in /sbom that names no project answers 400
with a hint, since it's most likely the upload path.
- The upload endpoint lists JSON, XML and plain text (SPDX tag-value)
as accepted.
- The docs say that versions are compared as the SBOM spells them.
FromPURL now names packages the way their handlers store them:
- NuGet IDs are lowercased, as the NuGet handler stores them.
- PyPI names are normalized as in PEP 503. The PyPI handler takes the
name from the file name, which spells "." as "_", so
pkg:pypi/zope.interface has to match the cached zope-interface.
- Conda package URLs without a channel qualifier are skipped: the conda
handler records packages as channel/name, so they can never match.
- The Go main-module check ignores case. syft keeps the module path's
case in its mainModule property, while golang package URLs are
lowercased.
A table test checks every mapped PURL type against the name its handler
passes when caching.
The vulnerabilities table is only written by tests: nothing in the proxy
calls UpsertVulnerability, so the dashboard card, the per-project count
and the per-component markers were always empty. Remove them along with
GetComponentVulnerabilities and inventory.Affected, and keep the
outdated marker, which uses the latest version the handlers record.
The page asks for the upload token and sends it to DELETE
/api/projects/{project}, so /ui gains no unauthenticated way to delete.
The token is kept in session storage for the next delete.
A syft SBOM of a repository lists the repository itself: the module in
go.mod and the root of package-lock.json, both without a version or with
the version from package.json, and the module its Go binaries were built
from. The project page showed them as dependencies.
Entries without a version are now skipped, as are lock-file roots (lock
entries that depend on others while nothing depends on them) and modules
syft marks as a binary's main module.
Repositories upload an SBOM of their default branch, for example from
syft in CI, to PUT /api/projects/{project}/sbom. The proxy records the
package versions it lists, keyed by the same unversioned PURL the
handlers record packages under, so the inventory joins with what the
proxy has cached and the vulnerabilities it holds.
That answers which repository uses which version of a package: the new
/ui/projects pages list each project's package versions with outdated
and vulnerability markers, package pages show the projects per version,
the dashboard lists vulnerable versions in use, and
GET /api/projects?purl= does the same for scripts.
The feature is off by default. Uploads and deletes need projects.token;
an upload replaces the project's components in one transaction.
projects.max_age removes projects that stopped uploading.
Within metadata_ttl cached npm and Composer metadata is served without
asking upstream, so a new release can take that long to show up. Both
registries publish a feed of the packages that changed.
upstream.npm_change_feed and upstream.composer_change_feed (off by
default, require cache_metadata) start a reader per feed that runs every
minute and clears fetched_at on the cached rows of listed packages that
were fetched before the read. The rows keep their document and ETag, so
the next request revalidates them upstream. Routed packages are cached
under other keys and never match. A Packagist resync expires all cached
Composer metadata. Failed reads back off up to 15 minutes. The feed
position is kept in memory, so after a restart the reader starts from the
feed's current position.
`npm dist-tag ls` reads /-/package/{name}/dist-tags. The proxy had no
route for it, so the request ended up in the metadata path and failed.
The tags are taken from the packument the proxy serves rather than
relayed from upstream, so latest never names a version cooldown holds
back, routed packages use their route, and pushed versions and
hosted-only scopes count as they do for the packument. Changing tags
stays unsupported and still gets 405.
Alpine mirrors keep only the current build of each package, so once a
package is upgraded `apk add curl=8.5.0-r0` fails: the current index no
longer lists the old build, even when the proxy still has it cached.
With upstream.apk_snapshots set (e.g. "90d"), each version of an index or
detached signature the metadata cache stores is copied to
_snapshots/alpine/ and recorded in a new metadata_snapshots table with the
time it was first served. /apk/{name}@{YYYY-MM-DD}/ serves the version that
was current at the end of that day; packages below it are the plain path's
requests. The kept bytes are the signed upstream bytes, so apk verifies
them unchanged.
An hourly job deletes versions replaced longer ago than the configured
duration and queues their blobs on the pending-delete queue; the newest
version of each index stays. Packages listed by kept snapshots are not
protected from eviction or retention; the docs say so.
govulncheck reads vuln.go.dev directly, so on networks that only reach
the proxy it cannot run. upstream.go_vulndb (default https://vuln.go.dev)
is now served at /go/vulndb for `govulncheck -db`.
Only the files the database API defines are passed on (index/db,
index/modules, index/vulns and ID/{id}, as .json and .json.gz), with IDs
limited to letters, digits, '-' and '_'. They are stored and served
byte-for-byte; .json.gz files are gzip data without Content-Encoding.
All files, entries included, follow the metadata TTL since entries change
when their modified date does, and fall back to the cached copy when the
database is unreachable. Non-JSON responses are refused so a block page
does not replace the copy kept for an outage. HEAD is allowed under
/vulndb/, because govulncheck checks a non-default database with HEAD on
index/modules.json.gz.
The proxy's packages.json listed only metadata, search and notify URLs,
so a client using the proxy as its only Composer repository lost
`composer audit`, `audit.block-insecure` and Packagist's malware filter
list without any warning.
The service index now carries the security-advisories and filter entries
of the upstream repository's own packages.json, cached for the metadata
TTL, with URLs pointing at the proxy. An upstream that does not offer
them gets nothing advertised, so a custom upstream.composer_repository
without the API keeps working as before. Advisories and filter entries
embedded in p2 metadata were already passed on unchanged.
Advisories queries and the filter summary are relayed uncached. Names of
routed packages and hosted-only vendors are removed from advisories
queries first, so private package names are not sent to the public API.
The dashboard shows the default and the ecosystem overrides, and counts
package overrides and patterns without naming them, since the UI has no
auth. A package page shows the cooldown that applies to the package and
the kind of rule it comes from.
Also add FillVersionPublishedAt, which stores a publish time only where
none is stored, for recording the versions cooldown holds back.
A client that fell behind is switched to the stored copy, which for an
unrecorded refetch is deleted once the waiters close. It now takes its
own hold, released when it closes.
A breaking release gets upgrade notes before it goes out. release.yml
creates it as a draft and publish.yml isn't started; publishing the
draft by hand starts publish.yml for the images and the chart.
The pod only ever got the config, data and tmp volumes, so there was no
way to mount a private CA certificate, file-based credentials or a
projected token, or to run init containers. All three values default to
empty, leaving the rendered Deployment unchanged.
Render the new blocks in CI and document the private CA setup in the
README.
Closes#614
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The monthly release skipped the month when a pull request labelled
breaking had merged, leaving the release to be cut by hand. It now
cuts a new minor version instead (a new major version from 1.0 on).
The upstream read fed the client through a pipe, so it ran at the client's
speed. The client is now fed from a bounded buffer and, once it falls
behind, continues from the stored copy where it left off.
A caller handed the stream takes no hold on an unrecorded object.
Release lists and manifests record their sha256 digest and are read back
through storedMetadataEntry and openMetadataEntry, so a corrupted copy is
fetched again. pkg:swift PURLs naming a source repository are looked up
through the registry's /identifiers.
Whether a packument is the abbreviated format is a property of the
document, so it sits with the rest of the npm metadata code; the
handler keeps only the error it turns that into. Add a test that keeps
the package's imports to the portable set, give jsonscan a package
comment, and describe the new packages in the architecture docs.
npm metadata rewriting depended on handler.Proxy, which pulls SQL
drivers, cloud SDKs and metrics into its import graph. Move the
transformation into internal/metadata/npm with explicit policy,
evaluation time, public URL and hosted-version inputs, and share the
JSON scanner between npm and Composer through internal/jsonscan.
Rebased onto main: rewritten tarball URLs are still logged once per
document, now through an OnTarballsRewritten callback that reports the
count.
proxy mirror and mirror jobs now take pkg:swift PURLs whose namespace
is the registry scope, such as pkg:swift/apple/swift-nio@2.62.0 for
apple.swift-nio. The source archive is fetched from upstream.swift,
checked against the SHA-256 checksum in the release metadata and cached
under the key the archive route serves. PURLs whose namespace is a
source repository URL don't name a registry identity and are refused.
Unversioned PURLs list releases from upstream.swift.
SwiftPM reads the release list and Package.swift before it downloads
an archive, and the handler passed both through uncached, so a mirrored
release still couldn't be resolved with the upstream down. With
cache_metadata the handler now keeps them in the metadata cache along
with their Link header, which carries release list pages and the
version-specific manifests, and serves them when the upstream fails.
Manifests are keyed by tools version, so the 5.8 of an alternate link
and the 5.8.0 SwiftPM requests share a key. The mirror stores every
release list page and each manifest a release links.
On a cache miss the proxy stored the whole artifact before sending any of
it, so clients with an idle timeout gave up on large artifacts: git-lfs
after 30 seconds, pacman after 10. The caller that starts a shared fetch
now gets the bytes as they arrive while they are stored; requests joining
the fetch, HEAD and range requests still wait for the stored copy.
The client's copy is checked against the upstream's digest as it is read,
and a mismatch aborts the response before it completes. A client going
away leaves the copy to storage running. A storage error leaves the
client's copy running: the client gets the artifact, and the failure is
logged at error level and counted in proxy_artifacts_served_uncached_total.
A block-mode scanner has to see the whole artifact first, so with one
configured for the ecosystem a miss is handled as before.
proxy mirror and mirror jobs now take pkg:julia PURLs with the uuid
qualifier. The tree hash of the version comes from the General registry,
after checking the uuid is the one General lists for the name, and the
tarball is fetched from upstream.julia at /package/{uuid}/{tree-hash} and
cached under the name and tree hash the Julia handler uses.
The handler names a package from the registry's Registry.toml, which it
loads from the upstream. Without the upstream it fell back to caching and
looking up by uuid, so a package cached under its name, mirrored or
fetched before a restart, was not served offline. It now looks the
tarball up by its tree hash in that case.
Julia versions are listed from General whatever upstream.julia is, as
packages are identified by uuid and tree hash on every Pkg server serving
General.
Mirroring pkg:helm PURLs failed with an unsupported ecosystem error. The
namespace now names the upstream.helm repository, as the Helm handler
caches charts per repository: pkg:helm/{repository}/{chart}@{version}.
The chart is found in the repository's index through the handler's own
index lookup, fetched from the first URL the release lists, as helm does,
checked against the listed digest and cached under the repository,
digest and filename the handler serves it from. The index goes through
the metadata cache, so helm finds the chart there while the repository is
down. Unversioned PURLs list every version in the index.
Charts without a repository, from an unconfigured one, or listed in an
OCI registry are refused; the last are pulled through /v2, and the error
names the pkg:oci PURL that mirrors them.
- Keep the download URLs of a batch response in memory until shortly
before they expire, so the GET that follows needs no second batch
request. A kept URL that fails is replaced by a fresh one.
- Cut off a download that isn't the object's size.
- Copy an object whose content is already stored, such as for a fork,
instead of downloading it, once the upstream has confirmed the
repository has it. Artifacts are indexed by content hash for this.
- Keep signed download URLs out of logs and the database: the fetch uses
the signed URL, everything else the URL without its query.
- Answer a batch from the cache when the upstream can't be reached or
fails, so git-lfs can use cached objects during an outage. Refusals
are still passed on.
- Answer a HEAD for an uncached object from the batch response instead
of downloading it.
git-lfs finds the LFS API under the rewritten remote, so clones through
/git/ already asked the proxy for LFS objects and got 404s. The proxy now
serves the batch API for downloads: it forwards the batch to the upstream,
which decides which objects the repository has, and points each download
at the proxy. Objects are served from the artifact cache. A miss asks the
upstream batch API for a fresh download URL, fetches it and checks it
against its sha256 object ID before caching it.
Download URLs often name another host than the Git server, such as object
storage. The proxy fetches from whatever host the upstream names, with the
usual address checks; upstream.auth credentials stay scoped to their
hosts. Objects are cached per repository. Uploads and locking are refused.
Closes#517.
isMemoryPath missed URIs such as file::memory:?cache=shared, whose
readers would each have got an empty database. New tests open a database
left in rollback-journal mode and run writes, hit flushes, lookups and
stats scans concurrently.
The proxy filters repodata.json and current_repodata.json, but
repodata.json.bz2 went through the metadata cache unfiltered, and
repodata.json.zst and repodata.jlap reached the download handler and were
passed through. conda tries repodata.json.zst first, so with cooldown on it
got versions still in cooldown.
With cooldown on these now answer 404, and conda falls back to
repodata.json. repodata_from_packages.json is filtered like repodata.json.
With cooldown on, the RubyGems compact index and versions API, the Hex
package and Hex API entry, and conda's repodata.json were fetched with
plain HTTP requests on every client request: no TTL, no revalidation,
no coalescing and no stale fallback, even with cache_metadata.
Gem and Hex now read both documents through the metadata cache and
filter the cached copies. The filtered result goes through the rewrite
cache and is served with an ETag. When the times document is older than
the index and misses a version, it is fetched again; if that fails the
version is left out. Conda filters a fresh stored repodata.json as it
reads it from storage, uses the metadata cache otherwise, and remembers
the ETag of the filtered copy until a withheld entry leaves cooldown.
SQLite used one connection for everything, so request lookups queued
behind the per-minute stats queries, dashboard pages and eviction scans.
Writes keep the single connection. Get, Select, Query and QueryRow now go
to a pool of query_only connections (GOMAXPROCS, 2 to 8), which WAL mode
lets run alongside the writer. A write sent to that pool fails.
Pragmas move into the DSN so every connection gets them. Postgres and
in-memory SQLite keep one pool for both.
Holds placed by the rate limit gate are written to a new upstream_holds
table, and each instance reads that table at most every two seconds and
refuses requests to a host any instance holds. Before, every replica had
to get its own 429 before it stopped sending.
Instances waiting on a fetch lease reread the holds when the fetch fails,
so when the holder was rate limited they answer 503 with its Retry-After
instead of each sending one more request to the limited host.
The Helm index.yaml and PyPI simple pages and JSON API documents were
parsed and rewritten on every request, cache hits and conditional
requests included. They now go through the rewrite cache like npm and
Composer, so a repeat request is served from memory with an ETag of
its own, a dated Last-Modified, a gzip copy and 304 when it matches.
PyPI simple pages are keyed by their content type and the versions
filtered out as well as the upstream page. Download hrefs are now
rewritten with a literal prefix search that makes the same changes as
the old regular expression, and the link filter's expressions are
compiled once.
A chart cache miss no longer parses the whole Helm index again while
the stored index is unchanged: the chart downloads it lists are kept
per repository, keyed by the index digest.
A stale tag was revalidated by repeating the client's GET with
If-None-Match. Docker Hub counts manifest GETs against its pull limit,
so every TTL expiry of a tag could cost a pull. The tag is now resolved
with a HEAD and the Docker-Content-Digest compared with the cached one;
the manifest is fetched by digest only when that digest isn't cached.
A HEAD without a digest falls back to the conditional GET.
A client HEAD whose digest is cached now points the tag at it, so the
GET that follows is a cache hit. Concurrent requests for the same
manifest share one upstream exchange, and instances sharing Postgres
take turns through the fetch lease.
Every npm and Composer download ran GetLocalArtifact, a full artifact
row lookup, even with hosted packages disabled. localVersions already
returns early in that case, so metadata does not list pushes, and
/generic/hosted/ is only served while hosting is enabled.
GetLocalArtifact now returns nil without a lookup when hosting is off.
Downloads then take the normal proxy path. An archive pushed earlier is
still answered from storage by that path and is never replaced by the
upstream copy; only cooldown now applies to it, as metadata no longer
lists it either. The hosted docs say so.
Every PyPI download with cooldown on read the JSON API first, cached
files included. Without cache_metadata that was a full /pypi/<name>/json
per file.
A download that looks up the JSON API now stores the release's newest
upload time on its versions row. A cached file is then checked against
the stored time without asking upstream. A file that isn't cached is
still looked up, since files can be added to a release later, and the
stored time is refreshed. A stored time inside the window withholds the
release without a lookup.
A Go module's .mod and .info for a canonical version, and RPM repodata
files named after their checksum, never change once published. They
went through the metadata TTL, so each cost a revalidation every TTL with
cache_metadata on and a full fetch on every request with it off.
They are now kept in the metadata cache with or without cache_metadata
and served whatever their age. A stored copy is checked against its
digest and fetched again if it doesn't match. Go queries, @v/list,
@latest and repomd.xml stay on the TTL.
Nothing set GOMEMLIMIT, so in a container the heap could grow until the
container was OOM-killed while much of it was garbage, for example while
large metadata documents were being rewritten. At startup on Linux, read
the cgroup v2 or v1 memory limit and set the runtime's soft limit to 80%
of it, leaving room for git subprocesses. An explicit GOMEMLIMIT wins.
A HEAD on a cache hit opened the stored object and closed it unread.
On S3 and Azure that is a GET whose body is thrown away, which can also
drop the pooled connection; on GCS it is a GET as well.
The server now marks HEAD requests in the request context, and readStored
answers a marked cache hit from the database record. Size, content type
and ETag already came from the record. Accept-Ranges depended on the
opened reader being seekable, so backends now say whether their readers
are (Blob does, GCS does not) and the HEAD response matches the GET.
A HEAD still counts as a cache hit, as before. DirectServe redirects are
unchanged since they are decided before the object would be opened.
ReadMetadata grew its buffer with io.ReadAll, which allocates about 2.5
times the document size in total. Cached documents have a recorded size
and most upstream responses a Content-Length, so read those into one
buffer of that size. The size is a hint: a missing, wrong or over-limit
one still reads the body correctly and enforces the limit.
Store passed empty writer options, so the S3 driver allocated six 8 MiB
part buffers up front for every object over 16 MiB, on top of reading
the first 16 MiB whole, and Azure used five 8 MiB blocks. Use 5 MiB
parts, the S3 minimum, with two uploads in flight.
A 32 MiB store against a fake S3 endpoint allocates 54 MB instead of
88 MB.
A cache hit hashed the stored object with SHA-256 for content_hash and
again with every native algorithm in versions.integrity, SHA-512 for npm.
The native integrity was already checked when the bytes were stored and
content_hash was computed from the same bytes, so the second hash adds
CPU per byte served without catching anything more. Records without a
content hash still fall back to the native integrity.
Each download and metadata handler logged its own request line at info,
next to the request line LoggerMiddleware already writes for every
request with path, status and duration. That doubled the info log volume
on the hot path. The handler lines only repeat values parsed from the
path, so they move to debug.
Kept at info: the npm audit line (it carries the request body size), the
julia package line (it carries the name resolved from the UUID) and the
gem and hex cooldown lines.
Mirroring pkg:composer failed with unsupported ecosystem. A tagged release
is now mirrored with the dist archive upstream.composer lists, under the
file name the Composer handler serves it as and checked against the
listed shasum. The release's publish time is stored as the handler stores
it, so the archive is served from the cache without the metadata, with
cooldown on as well.
Branches are refused, since the handler reads the metadata for every
branch download, and so are archive URLs with a query and packages in
upstream.composer_routes.
The mirror now finds the whole version rather than its files, for the
dist type and publish time.
Part of #414.
A conda version is mirrored with every build anaconda.org lists for it,
each cached under the channel, name, version and subdir/filename the conda
download handler reads from its path. The channel comes from the namespace
or the channel qualifier; a package without one is refused, as retention
keys without one are. The subdir, build and type qualifiers select builds.
Which files a version has, which qualifiers select them and where the
handler caches them is now described per ecosystem, so PyPI and conda share
mirrorFiles.
Part of #414.
Mirroring a PyPI version passed each file's sha256 as the integrity
GetOrFetchArtifactFromURLWithIntegrity records with the version. Wheels of
one version share the name and version they are cached under, so the last
file's checksum replaced the others', and serving the other wheels from
the cache failed the integrity check.
The mirror now passes the sha256 as a digest, which is checked against
that file's content hash alone.
A streamed cache hit whose bytes don't match the row's digest now
records an integrity failure and deletes the row, queueing its blob for
deletion, as a corrupt cached artifact is handled. The next request
fetches the document from upstream instead of aborting until the TTL
runs out.
proxy mirror and mirror jobs failed for pypi with "unsupported
ecosystem", since the resolver they use has no registry for it. A PyPI
version has several files, so the mirror now lists them from
upstream.pypi's JSON API, through the same client as version listing,
and fetches each one, verified against its sha256. The file_name
qualifier limits it to one file.
Files are cached under the name and version the download handler reads
from the filename, which can differ from the PURL (zope.interface's
wheels are zope_interface), so pip is served them from the cache.
PyPI and npm downloads look up the publish time before the cache. When
that lookup fails, a file that is already cached is now served, and only
a cache miss fails with 502 or 503. The proxy keeps serving what it has
while the upstream is unreachable, and nothing new is fetched unchecked.
Also drops the unattached comment block in cooldown_fail_closed.go.
The case-variant lookup filtered on lower(name), which no index covers,
so a legacy row's first lookup scanned every metadata_cache row in its
ecosystem. Reclaim already skips a queued path a row still points at,
so each row now queues just its own legacy blob once it has moved off
it. On case-insensitive storage a case variant still at its legacy path
can lose its blob that way, and is then fetched again.
With cache_metadata on, every metadata cache hit read the whole stored
document into its own buffer before any coalescing, since
coalesceMetadata only covers upstream misses. metadata_max_size bounds
one request, not the sum: 30 concurrent hits on a 20 MB document held
about 820 MB.
The handlers that go through ProxyCached serve the stored document
unchanged, so a hit there no longer needs the bytes in memory.
serveCachedMetadata streams the blob from storage to the response
instead, with the row's Content-Type, Content-Encoding, size, ETag and
Last-Modified, and answers If-None-Match and If-Modified-Since with 304
as before. The bytes are checked against the row's content digest as
they are sent, and a mismatch aborts the response like a corrupt cached
artifact. A row without a digest or size, a document over the size
limit, or a blob that can't be opened takes the buffered path as before.
Handlers that rewrite or filter metadata keep reading it into memory,
and the rewrite cache stays in front of them. writeMetadataValidators
is split out of writeMetadataCachedResponseWithEncoding so both paths
share the conditional request handling.
registries main streams npm packuments (git-pkgs/registries#109), so
listing the versions of packages whose packument is over 10 MiB, such as
typescript, no longer fails with "unexpected end of JSON input". It also
reads Maven versions from maven-metadata.xml (#114).
go.mod points at a pseudo-version of registries main until those are
released.
registries' safehttp now checks the destination of requests sent through
a forward proxy (git-pkgs/registries#116), so the proxy's own guard is
removed and newUpstreamClient is back to its earlier form.
go.mod points at a pseudo-version of registries main until #116 is
released.
The per-ecosystem rules for when two spellings name the same package
(PEP 503, NuGet case, Composer's leading v, conda channel, OCI paths,
Swift registry case) now live in purl. The denylist keeps only the
proxy's own pkg:generic/swift-registry spelling and the gem platform
match. Keys, cache paths and stored PURLs are unchanged.
internal/enrichment now adapts the shared module instead of calling
registries and OSV itself. License categories in the API and the web UI
come from enrichment.CategorizeLicense, which normalizes the license
first. Vulnerability fixed versions come from the affected range of the
queried package.
Ignoring every row still at its unhashed path dropped the whole metadata
cache on upgrade, including on Linux and S3 where keys never collided,
and left nothing to serve while the upstream was down.
A legacy row's blob is now hashed once and compared with the row's
content_digest. When it matches, the blob is this key's document: it is
copied to the hashed path and the row moved there, so it is served
fresh or as a stale fallback as before. Only a mismatch, where another
key's document overwrote the shared file, or a row without a digest is
dropped and fetched again.
The legacy blobs are queued for deletion once no key differing only in
case still reads its legacy path, which on a case-insensitive
filesystem is the same file.
Debian, RPM, APK, PyPI, CRAN and Maven build upstream URLs from the
decoded request path, where a '?' or '#' would end the path and cache
one file under another's name. Only the server middleware refused them;
Arch and FreeBSD also check in the handler. Check in these handlers
too, so they stay safe when mounted without the middleware.
With cooldown on, the Hex handler read the /packages response, its decompressed payload and the Hex API's JSON without a limit, and the gem handler decoded the versions API's JSON the same way. A misbehaving upstream could make the proxy hold any amount in memory, and a small gzip payload could expand without bound.
These reads now go through ReadMetadata, so metadata_max_size applies to each, and to the Hex payload after decompression as the NuGet handler does. A response over the limit fails the request with 502.
RubyGems, Hex and PyPI keep publish times outside the metadata clients
read, so cooldown fetches them from a second API. When that fetch
failed, was rate limited, or returned something that didn't parse, the
compact index, Hex package or simple page was served unfiltered, and a
Hex package that couldn't be re-encoded was passed on as upstream sent
it. A flaky or rate-limited timestamp API let every version in
cooldown through.
These now fail with 502, or with 503 and Retry-After when the rate
limit gate held the request, as the Go handler already does. PyPI file
downloads check the same way. A PyPI package the JSON API answers 404
for still has no times and is served unfiltered, as before.
npm, Composer, pub.dev and PyPI JSON metadata that couldn't be
rewritten was also passed on unfiltered unless the package had denied
versions. It now fails with 502 whenever cooldown applies too. npm
tarball downloads whose packument couldn't be fetched skipped the
cooldown check and downloaded straight from upstream; they now fail.
All of this only applies when cooldown covers the package: packages
exempted with a duration of 0 skip the timestamp lookups, and the gem
and Hex handlers serve them through the metadata cache as when cooldown
is off.
upload-pack was started with its stdin attached to the request body, after
the read deadline had been extended to two hours, and ls-refs requests took
no slot. A client that sent part of a request and then nothing kept an
upload-pack process waiting, and any number of them could.
Stateless requests are complete once sent, so the handler now reads the
body in full first, within the server's read timeout, keeping bodies past
1 MiB in a temporary file. Only then are the deadlines extended for the
response. info/refs and ls-refs share their own slots, four per CPU, so they
still never wait behind clones. Each response write is bounded to two
minutes, so a client that stops reading a pack frees its slot instead of
holding it until the request timeout.
Conan, CRAN, Conda, RPM, the main Debian archive and the default Maven
repository keyed cached metadata by the request path with '/' replaced
by '_'. Names and versions may contain '_', so foo/bar_1.0 and
foo_bar/1.0 shared an entry, and with cache_metadata one reference's
recipe files were served for the other. Conan also appended the query
with '_', so a path without a query could collide with a shorter path
plus one.
Hash the upstream URL, the path and, for Conan, the query instead, as
the APK, Arch, generic and Helm handlers already do. Each part is
length-prefixed so no two inputs share a key.
Without artifact streaming, a chart was fetched with no expected
digest, committed to the cache, and only then compared with the
index's digest by serveChart, which cleared it again. Until then the
unverified chart was a cache entry like any other.
The index digest is now always passed to the fetch, so a mismatching
chart is discarded before it is committed, as it already was when
streaming. serveChart's check stays for charts that were cached before.
Manifests and digest-addressed artifacts were only checked against
sha256 digests. A sha512 digest, which the OCI spec also allows, or a
digest with any other algorithm was fetched and cached unchecked: a
manifest requested as sha512:<hex>, or a tag whose Docker-Content-Digest
header named one, was stored under that digest whatever the body was.
Manifests are now checked against any digest in the reference or the
Docker-Content-Digest header with the digest's own algorithm, and one
the proxy can't compute is answered with DIGEST_INVALID and not cached.
GetOrFetchArtifactFromURLWithDigest verifies sha512 (and sha384)
digests through the declared-integrity check, so a mismatch is never
committed, and refuses other algorithms with ErrUnsupportedDigest
instead of fetching unverified. Blob routes still only accept sha256.
Cached metadata was stored at _metadata/<ecosystem>/<key>/metadata. Keys
are case-sensitive, but on a case-insensitive filesystem such as APFS or
NTFS, file:// storage put keys that differ only in case in one file:
github.com/Sirupsen/logrus and github.com/sirupsen/logrus each had a
row, and whichever was cached last was served for both. Keys may also
hold names Windows reserves or end in a dot.
The path now carries a SHA-256 of the exact key. Keys that already are
SHA-256 hex digests, as most hashed keys are, keep their path.
A row still pointing at an unhashed path may hold another key's bytes,
so it is neither served nor revalidated; the document is fetched again
and stored at the new path, and the old blob is queued for deletion.
The upstream client refuses loopback, private and link-local addresses
when it dials, but with HTTP_PROXY or HTTPS_PROXY set it dials the
forward proxy, so only the forward proxy's address was checked. A URL
taken from upstream metadata, such as a Helm chart URL, a Composer dist
URL, a token realm or a redirect, could then reach 169.254.169.254 or
an internal host through the forward proxy.
The transport's proxy function now resolves the destination of every
request it sends to a forward proxy and applies the same rules,
including upstream.allow_loopback and upstream.allow_private_hosts,
before handing it over. A name that cannot be resolved locally is still
passed on, since some networks only resolve public names at the
forward proxy. Direct requests are unchanged.
Git fetches and token requests use the same transport, so they are
covered too.
An uncached tarball is checked against the integrity its packument
declares. When the packument couldn't be fetched or was refused, for
example because it is larger than metadata_max_size or the registry
answered 503, the download went ahead with no integrity at all, and
the unverified tarball was cached and served to every later client.
Without the packument the download now fails like the Composer one
does: 502, or 404 when the registry says the package doesn't exist.
Nothing is fetched or cached. Tarballs already in the cache are still
served, since they were verified when they were stored, and a packument
that lists the version without a usable checksum still falls back to
an unverified fetch as before.
With cooldown on, /info/{name} is fetched from upstream and versions still in cooldown are removed line by line. The fetch passed on the client's Accept-Encoding, and setting that header turns off the transport's transparent decompression, so for any client that accepts gzip, Bundler included, the filter scanned gzip bytes, matched no versions and sent the compressed index on with upstream's Content-Encoding. Cooldown filtered nothing for those clients.
The fetch now passes on only Accept, so the transport decompresses the index before it is filtered. If-None-Match and If-Modified-Since are no longer passed on either: the client's validators came from a filtered copy, not from upstream's file.
The filtered index no longer carries upstream's ETag, Last-Modified, Digest, Repr-Digest or range headers, which describe the unfiltered file; Bundler checks the digest against the body it gets.
The index is now read through ReadMetadata before anything is written. bufio.Scanner stopped at a line over 64 KB and the handler sent what it had so far as a complete 200; a read error or an index over the metadata size limit now answers 502.
A generic upstream forwards any path below /generic/{name}/. The
documented mise setup puts a GITHUB_TOKEN on api.github.com to raise
the rate limit, so anyone who could reach the proxy could make any GET
request with that token, including for private repositories it can
read, and the answers went into the shared metadata cache. upstream.git
already refuses an upstream with credentials unless it lists its repos.
upstream.generic_paths now lists, per generic upstream, the paths it
may serve, with the same pattern syntax as git repos. Other paths
answer 404 without reaching the upstream, and so does a path holding
'%', '?', '#' or a backslash, which the upstream could read as a
different path than the one that matched. Config validation refuses a
generic upstream that upstream.auth has credentials for unless it has
generic_paths, which also covers a github.com token reaching the
generic github upstream. Upstreams without credentials keep serving
every path.
The docs and example config show the paths mise needs and recommend a
fine-grained token limited to public repositories.
The transport remembered a challenge for every /v2/<repository> it was
ever challenged on and never forgot one, so a long-running proxy, or a
metadata document naming many registries, grew the map without limit.
Tokens were only pruned of expired entries, and the expiry came from
the token service.
Challenges are now kept per registry host, holding only the realm and
service, and the pull scope for a request is built from its path, so
pulling many repositories from one registry keeps one entry. Both
caches have a size limit, evicting an arbitrary challenge or the token
that expires soonest, and a token is reused for at most an hour,
whatever lifetime the token service claims.
After a 401 the transport fetches a token from the challenge's realm,
using credentials configured for that realm, and sends it to the host
that issued the challenge. It trusted that host if it had any configured
credentials, even unrelated ones, or shared the realm's registrable
domain. The second rule makes every tenant on a shared domain such as
jfrog.io, azurecr.io or pkg.dev the same site, and the first hands a
token minted from one registry's credentials to another registry the
proxy happens to have a key for.
A credentialed token now goes only to a registry on the realm's own
host and port, to one configured with the same credentials as the
realm, or to Docker Hub and GitLab.com, whose registries use a token
service on another host. Anonymous tokens are unaffected.
The scope in the token request also came from the challenge, so a
registry could ask for push access or for other repositories. The proxy
only reads, so it now always asks for pull access to the repository in
the request path.
parseGemFilename tries every "-<digit>" in the filename as the start of the version and matches the rest against two patterns each time, so its cost grows with the square of the filename's length. A path of a few hundred kilobytes kept a request busy for minutes, and the parse doesn't watch the request context.
Gem filenames longer than 512 bytes are now answered with 400 before parsing. Real ones are far shorter.
Without X-Frame-Options or a frame-ancestors policy another site could
frame a version page and trick a signed-in user into clicking Delete.
Every /ui response now sends both; browsed archive files keep their
sandbox policy alongside frame-ancestors.
A page on another site could make a visitor's browser POST a text/plain
body that parses as JSON and start mirror jobs. Browsers only send
application/json cross-origin after a CORS preflight, which the proxy
never answers, so HandleCreate now refuses other content types with 415.
- Name the proxy in the User-Agent of publish-time HEAD requests, as its
artifact downloads do, rather than Go's default, which Maven Central
asks tools not to send.
- Take a version's time from its Gradle .module as well as its .pom, so
either one fetched first carries it.
- Serve filtered metadata through the shared rewrite path, with gzip and
the stale warning like the other ecosystems.
- Say in the docs that a refused .pom or .module is still cached.
A build fetching pinned versions never reads maven-metadata.xml, so each
version's time was read with a HEAD before its download, roughly
doubling the requests to Maven Central, which answered a fresh build
with 429 and a 30-minute Retry-After. The .pom's own response now gives
the time; Maven fetches the .pom before a version's other files, so
their check reads the stored time. A HEAD is only sent for a .pom served
from the cache or a file requested before its .pom.
A 429 on that HEAD now answers 503 with the upstream's Retry-After, as
other held upstream requests do, instead of 502.
A version's publish time is the Last-Modified of its .pom, read with a
HEAD request once per version and stored in the versions table. Named
repositories store theirs under a repository_url qualifier.
Versions in cooldown are left out of maven-metadata.xml, <latest> and
<release> are moved to the last versions left, and the metadata's
checksum files are computed from what is served. Files of a version in
cooldown get 403. Only versions that can still be in cooldown are
checked: none when <lastUpdated> is older than the window, otherwise
from the end of the list back to the first old enough version.
Part of #132.
TestSlowFetchAfterPushIsWaitedFor relied on the ref check finishing
within two seconds, which a slow Windows runner missed. The test now
holds the fetch and ends RefreshWait itself. When the check and the wait
end together, ensureFresh now waits for the fetch rather than picking
either at random. Failing gitcache tests print the cache's log.
A response that ended early for another reason than the size limit was
passed on as complete. TestRelayStopsAtLimit failed that way in CI. The
test upstream now reads the request body as a git server does, and the
test prints the relay's log when it fails.
Shutdown cancelled the eviction, retention, reclaim, backfill and stats
loops but closed storage and the database without waiting for them, and
the stats loop never stopped. A pass still running kept the database
open, which on Windows left proxy.db locked after a server test. Shutdown
now waits up to 30 seconds for them, and for mirror jobs.
The job store tests left their jobs running after the test, and on
Windows the test's directories could not be removed while a job was
still writing to them. JobStore.Wait now waits for jobs to return, and
the tests cancel their jobs and wait for them.
A detached fetch waiting on another instance's lease ran the wait under
the request context, so a client leaving cancelled every caller joined
to it. The wait now runs past the client, bounded by the holder's fetch
and the lease expiry.
When the holder's fetch failed, each waiting instance took the lease in
turn and repeated it, so against a hanging upstream the k-th instance
waited about k timeouts. The holder now marks the lease failed. The
instances waiting on it then serve their stale metadata copy if they
have one, and otherwise fetch at the same time.
A schedule's claim was kept when its run was cut off, so after a
restart every instance skipped it for most of an interval. The scheduler
now drops the claim when the run ends with its context cancelled.
Clients resolve base_url and shards_base_url against the proxy's URL. A
scheme or host sends them around the proxy, and a leading slash leaves
its mount, so neither layout is served.
Instances sharing a Postgres database each fetched a missed artifact from
upstream; only callers within one instance shared a fetch. The instance
that fetches now holds a lease on the miss in a new fetch_leases table and
renews it while it downloads. Another instance missing the same key polls
the cache and serves the stored copy, or takes the lease over when the
holder releases it without storing anything or stops renewing it.
Metadata misses use the same lease when metadata caching is on with a TTL
above zero; without a stored copy there is nothing to wait for.
proxy_fetch_lease_waits_total counts the misses that waited. Reclaim
deletes leases left by stopped instances.
Refs #543
Instances started together on an empty database created the schema at the
same time and all but one failed on duplicate types. Schema creation and
migrations now run under a Postgres advisory lock.
Eviction, reclaim, retention and the Gradle build cache sweep take a
per-job advisory lock for each pass, and an instance that finds the job
running elsewhere skips the pass. Before, every instance evicted the whole
overshoot over max_size at once and emptied the cache.
Mirror schedules record their last start in a new job_runs table, so each
schedule runs once per interval instead of once per instance.
Refs #543
With cooldown on, every npm, Composer and Chef metadata request rewrote
the whole document, because a cached rewrite could keep hiding a
version after its cooldown ended. For a 24.5 MB packument that is about
45 ms and 31 MB of allocations per request, and the result was served
uncompressed and without Last-Modified, so Composer, which revalidates
p2 files with If-Modified-Since only, never got a 304.
The rewrite cache now keeps cooldown rewrites with an expiry of one
minute, so a version leaving cooldown shows up at most that late.
Concurrent requests share one rewrite, clients get the gzip copy, and
the rewrite is dated through metadataDate like any other. Its ETag is
the hash of its output, so replicas agree on it.
The cache is keyed by the digest the metadata cache already recorded,
instead of hashing the document again, which also lets storedRewrite
serve cooldown rewrites without reading the stored document.
Without the rewrite cache, cooldown rewrites behave as before.
Every Composer archive download resolved its dist from the package
metadata before checking the artifact cache. With cache_metadata off,
the default, that meant an uncoalesced upstream GET of the p2 document
for every download of an archive the proxy already had: three downloads
of a cached archive made three metadata requests. With the cache on,
each download after the TTL revalidated and rescanned the document.
The lookup is only needed for the publish time cooldown checks
downloads against, since lock files name archive URLs directly. Like
the npm handler, record that time in the versions table when the dist
is first resolved, and on later downloads of a tagged release read it
from there and serve the cached archive straight away. Without a stored
time, as for archives cached by an older proxy, the download takes the
metadata path, which then records it. The time is stored even with
cooldown off, so enabling cooldown later keeps the fast path.
The cache still enforces the denylist and verifies the stored digest,
and an archive is only ever cached under a name that was checked
against the metadata, so those guarantees hold. Branches, whose
archive and time change with every push, routed packages, whose
versions row would not say which repository the time came from, and
requests with a query string such as ?sha= keep resolving from the
metadata.
`npm search` requests /-/v1/search?text=... . The path has no "/-/"
after the leading slash is trimmed, so it fell through to the package
metadata handler: upstream got /-/v1%2Fsearch without the query
string, and the answer was cached under "-/v1/search", so every later
search was served the first one's results. Search and /-/ping (used
by `npm ping`, which lost its query string the same way) are now
relayed to upstream uncached with their query strings, like the
signing-keys endpoint.
Packuments share one metadata cache row whatever format they were
fetched in. Without cooldown the proxy asks for the abbreviated
format, which has no "time" map. After cooldown was turned on, a
still-fresh abbreviated packument kept being served from cache, and
used as the stale copy when upstream failed, so cooldown filtered
nothing and the tarball check let recent versions through. When full
metadata is required (cooldown or npm_full_metadata on), packument
lookups now pass a validator that rejects an abbreviated document
(top-level "modified", no "time"), so it counts as a cache miss, is
refetched in full and is never used as a stale fallback. Behaviour
without cooldown is unchanged.
fetchMetadataDocument takes the validate callback that
fetchOrCacheMetadata already has; other callers pass nil.
Each request for a General registry tarball whose hash differed from
the loaded name map started a goroutine that downloaded the whole
registry from upstream again, with no limit on concurrent loads and no
deadline. Alternating between two valid snapshots made every request
cost a full download and parse, and left the map at whichever snapshot
came last, older ones included.
The refresh now only loads the hash /registries currently lists, read
through the metadata cache, runs one load at a time and skips while one
is running, and stops after five minutes. Registry.toml is read up to
64 MB.
A failed .info fetch counted as an unknown time, so a rate-limited or
flaky upstream let versions in cooldown through. It now fails the list
or download with 502; only a version upstream doesn't know, or whose
.info has no time, is let through.
Times are stored under the escaped module path, which keeps modules
that differ only in case apart. Modules with uppercase letters were not
stored before and fetched every .info again on each list request. Only
canonical versions are stored, since a query such as a branch name can
resolve to a later commit.
A .mod or .zip whose version doesn't unescape is answered with 404
instead of being passed upstream unchecked.
With cooldown on, @v/list leaves out versions whose .info time is
inside the window, and @latest falls back to the newest release out of
cooldown when upstream's answer is too new. Each listed version's
.info is read once and its time is stored in the versions table, as
npm does, except for modules whose path has uppercase letters, since
golang PURLs are lowercased.
A .info, .mod or .zip for a version in cooldown, and a query such as a
branch name that resolves to one, gets 403. The go command moves on to
the next GOPROXY entry after 404 or 410, so a 404 would let
GOPROXY=proxy,direct fetch the version directly.
Go has no publish time: .info carries the commit time of the revision,
so a version tagged on an old commit counts as old. The docs say so.
Part of #132
The shard index (repodata_shards.msgpack.zst) and the shards of CEP 16
fell through to the uncached passthrough, so clients that use them, such
as pixi, got no cooldown filtering and no offline copy.
Without cooldown, the index and shards are now served through the
metadata cache. With cooldown, shards have to be filtered, but clients
cache them by the hash in the index and never fetch a hash twice. The
proxy rewrites the index instead: each hash is XORed with a mask derived
from channel, subdir and the current hour, and shards_base_url points at
cooldown-shards/{hour}/. That route undoes the mask, fetches the
upstream shard and drops records still in cooldown. The unmasked shard
paths answer 404 while cooldown is on.
When the index puts packages or shards somewhere the proxy doesn't
route, or has an unknown version, the proxy answers 404 and clients fall
back to repodata.json, which is filtered as before.
Fixes#529
Denylist entries now also match Composer versions with Packagist's
leading v, OCI repositories written as a path, Swift registry packages
in either form (also when streaming), Go versions with escaped uppercase
letters, and gems whose names contain a dash followed by a digit. OCI
manifests are checked by tag and digest, and denied blobs and Swift
archives answer 403 instead of 500 or 502.
Package files outside Hashed/, such as All/{name}-{version}.pkg from
repositories built without pkg repo --hash and the bootstrap
Latest/pkg.pkg, were relayed without any checks. They are still passed
through uncached, since their names can be reused for rebuilds, but a
versioned name is now checked against the denylist first, and while a
scanner covers freebsd they are refused, because only cached artifacts
can be scanned.
CI, the release binaries and swagger take their Go version from go.mod,
which pinned 1.26.7 and so missed the stdlib fixes in 1.26.8 to 1.26.10.
A toolchain line now sets 1.27.3 for them, and the Docker builder uses
the same version. The go line stays at 1.26.7, so source builds with an
older local Go still work.
golangci-lint 2.13.1 can't read Go 1.27.2+ export data, so CI moves to
2.15.0. Its errcheck flags two test cleanups that dropped the server's
exit error; they now check it.
Tomcat and Jetty drop ";params" from a path segment before resolving it,
so "..;" passed the dot-segment check and still got the credentials
configured for the upstream path. The check now ignores ";params", and
package routes refuse ';' like '?' and '#', which also stops "1.0;x"
from passing the denylist as a different version. Composer archives
stay exempt, since git branch names may contain ';'.
Record the rules agreed in #514: one approval from a maintainer other
than the author plus green CI, a "request changes" review blocks the
merge, and bigger changes start with an issue.
Denylist entries were compared as plain PURL strings, but handlers build
PURLs from download URLs that spell packages differently, so some denied
versions were still served. Entries and checked PURLs now go through one
per-ecosystem comparison form: PyPI names follow PEP 503 (case and runs of
-, _ and . are equivalent), NuGet ids and versions ignore case, a gem
version also matches its platform builds, and a conda entry matches the
package in every channel unless it carries a channel qualifier. Cache keys
are unchanged.
Routers clean literal dot segments, but encoded ones such as %2e%2e
reach the handlers. Maven, conda, cran, go and gem append the decoded
path to the upstream URL, so the upstream resolved it outside the
configured base path. Package routes now answer 400 for a '.' or '..'
segment, also when encoded twice or written with a backslash.
Upstream credentials matched such a URL by its path prefix, so
/scope/../other still got the credentials configured for /scope. URLs
with dot segments now get no credentials.
Conda download caching keyed artifacts on channel, name, version and
filename only, but the same filename can hold different bytes in each
subdir (python_abi-3.11-4_cp311.conda in linux-64 and osx-arm64, for
example). A client on one platform could be served another platform's
package. The subdir is now part of the cached filename, so each one gets
its own entry. Entries cached under the old key are no longer used and
are re-fetched on first request.
Only files with a known artifact or checksum extension went through the
download path; anything else, such as the .exe binaries protoc and gRPC
plugins publish or .hpi and .so files, was relayed from upstream as is.
Those files skipped the denylist, cooldown, scanning and the cache. Any
file in a version directory named {artifact}-{version}..., the way Maven
names what it publishes (timestamped for SNAPSHOTs), now takes the
download path. Directory listings and catalogs are still proxied.
proxy mirror and mirror API jobs resolved download URLs with a bare
resolver, so npm, cargo, gem, golang, hex, pub, maven and nuget packages
came from the public registries whatever upstream.* says. The resolver
now takes the configured upstreams as base URLs.
npm packages in upstream.npm_routes are fetched from their route and
stored under the route's filename prefix, the way the npm handler does,
so a public package can't stand in for a routed one and clients are
served the mirrored tarball.
Refs #414
proxy mirror filed pkg:gem and pkg:composer packages under rubygems and
packagist, the names purl gives their registries, while the handlers
store them under gem and composer. Since the resolver accepts both
spellings, mirrored gems were downloaded and filed under rubygems, so
the packages list filtered by gem didn't show them.
The mirror now uses the handler names. The package list, count and
search match every spelling of the requested ecosystem, so rows already
written as rubygems or packagist, by the mirror or by git-pkgs, show up
too without a migration.
Fixes#415
proxy mirror and mirror API jobs listed the versions of unversioned
PURLs from the public registries, ignoring upstream.*. They now list
them from the configured upstream through the proxy's upstream client,
and npm packages from their npm_routes entry. Where the upstream has no
listing API, or the ecosystem has no listing at all, the error asks
for a version.
Refs #524
Catalogues (meta.conf, packagesite and data) carry their signature
inside the archive and go through the metadata cache byte for byte.
Packages under Hashed/ have the content hash in their name, so they go
to the artifact cache as pkg:bsd/freebsd/{name}@{version}. Other files,
including the bootstrap files under Latest/, are passed through.
pkg.freebsd.org is the built-in freebsd repository; more mirrors go
under upstream.freebsd.
Closes#536
Serving cached metadata when the upstream refuses is documented for
generic upstreams, and GitHub rate limits are 403s, so the fallback
stays as it is. #503 handles npm's definite answers. What is left here
is the stale warning for Composer, Chef and hosted npm metadata, which
went through the rewrite cache without it.
A scheduled workflow tags the next patch version when main is green,
something has merged since the last tag and no pull request labelled
breaking was merged. It dispatches release.yml and publish.yml on the
new tag, since a tag created with GITHUB_TOKEN doesn't start them.
Scheduled runs are dry runs until the MONTHLY_RELEASE variable is true.
Closes#532
* Serve Git repositories from a local cache
Named upstream.git hosts are served read-only over Git's smart HTTP
protocol at /git/{name}/{repo}, so git clone and git fetch work against
the proxy through an insteadOf rewrite. Each repository is a bare clone
at git.cache_dir/{name}-{hash of the upstream URL}/{repo}.git, served
by git upload-pack.
Every request for refs checks the upstream's branches, tags and HEAD
with a protocol v2 ls-refs limited to those prefixes, and fetches only
when they changed, atomically and with --prune. Concurrent requests
share one refresh. Only branches and tags are cached, and HEAD follows
the upstream's default branch.
A commit a client asks for by ID that the cache lacks, such as a pull
request head, is fetched and kept under a hidden ref until a branch or
tag includes it or for 7 days. One the upstream cannot provide is not
asked for again for a minute, or an hour when it is larger than
git.max_repo_size leaves. This is off for upstreams upstream.auth has
credentials for, since GitHub serves commits from any repository in a
fork network.
git never reaches an upstream itself. Its fetches go through a loopback
relay that forwards with the proxy's upstream client, so address
checks, upstream.auth and the access log apply, and git never sees a
credential. Redirects are not followed, and git runs with an
environment of its own.
The cached copy is served while the upstream fails or is slower than
git.refresh_wait, until git.stale_max_age has passed since the last
successful check. Refusals and missing repositories are never answered
from the cache.
git.max_repo_size cuts a transfer off as it streams, and every fetched
pack is kept as a pack, so a cut-off leaves nothing behind. A
repository that can no longer be refreshed within the limit is cloned
again with only its branches and tags. git.max_size evicts the least
recently used repositories, never one in use. At startup only the
directory of a repointed upstream's old URL is removed, and Close waits
for refreshes in progress.
Hosts with upstream.auth credentials need a repos allowlist, and a
repository outside it answers 404.
Refs #403
* Document setting the Git rewrite through environment variables
The Git section of the configuration docs now explains what insteadOf
does, with the URL a clone fetches from, and shows the same rewrites set
through GIT_CONFIG_COUNT, GIT_CONFIG_KEY_n and GIT_CONFIG_VALUE_n for
CI. Package managers that run git for Git dependencies inherit those
variables, so their fetches are cached too; npm's GitHub tarball
downloads do not go through git and are not.
Refs #403
* Note that upstream.auth credentials reach every route to a host
Credentials are matched by URL, so a token configured for a Git upstream
is also sent by a generic upstream at the same URL, which forwards any
path and checks no repos allowlist. With a token for a private GitHub
org, /generic/github/acme/private.git/info/refs would list that
repository's refs to anyone who can reach the proxy.
* Cap concurrent upload-pack processes and request body size
Every POST git-upload-pack started a pack-objects with git's defaults,
a delta-search thread per CPU and unbounded window memory, and nothing
limited how many ran at once, so a burst of CI clones of one large
repository could run the proxy out of memory. At most two upload-pack
processes per CPU (GOMAXPROCS, so a container's limit) now build packs
at once, and further requests wait for a slot while the client is
connected. ls-refs builds no pack and does not wait. Serving runs with
pack.threads=1 and pack.windowMemory=64m, and every git process with
core.bigFileThreshold=8m, so index-pack streams large files instead of
holding them in memory.
A request body was streamed to upload-pack unbounded, so a small gzip
body could feed it an arbitrarily long negotiation for as long as the
request may run. Bodies are now limited to 100MB, both as received and
decoded, and one past the limit fails with an error instead of being
cut short quietly.
* Apply refresh_wait to the ref check only
refresh_wait cut off the whole refresh, fetch included, so after a push
with a large pack a client got the old refs once it passed, though the
upstream was fine and the docs say a clone right after a push sees the
new commit. It now bounds only the check of the upstream's branches and
tags. Once the check finds they changed, requests wait for the fetch;
one that fails, or stalls and is abandoned by http.lowSpeedTime, still
falls back to the cached copy within stale_max_age.
* Lock cache_dir against a second process
The cache clears its temporary directory at startup, so a second
process given the same cache_dir, such as a second pod on a shared
volume, would delete the first one's clones in progress, and each
process's eviction knows only its own readers. New now locks
{cache_dir}/.gitcache-lock before touching anything and holds it until
Close: flock on Unix, a handle opened without sharing on Windows. The
OS releases it when the process exits, so a crashed process can
restart. A second process fails to start with an error naming the
directory.
* Evict again once repositories in use are released
Eviction skips repositories a request or refresh is using, and only the
release of one too large to refresh ran it again, so a cache filled past
max_size while its repositories were in use stayed over it once they
went idle, until the next fetch. Eviction now records when it is left
over max_size, and the next release runs it again.
* Cap git.max_repo_size at git.max_size
A repository larger than max_size could still be cloned, up to
max_repo_size, which defaults to 5GB whatever max_size is. Once released
it was evicted to bring the cache back within max_size, and the next
request cloned it again. max_size now caps max_repo_size, so such a
repository is refused for an hour like any other too large one, with an
error that names both limits.
* Count fetches in progress toward git.max_size
Fetches enforced only max_repo_size, and eviction ran after each one,
so up to 16 concurrent fetches could take the cache past max_size by as
much as each may download. Each fetch now reserves space within
max_size as the relay receives its bytes, a chunk at a time, evicting
least recently used idle repositories to make room. When the fetch
ends, the repository's measured size replaces the reservation under the
cache lock, so its bytes are neither counted twice nor missed.
When evicting every idle repository would not make room, none is
evicted for the fetch, and it is cut off with ErrCacheFull rather than
ErrTooLarge, so the repository is not refused for an hour: a clone
answers 507, and a refresh serves the cached copy as when the upstream
fails. A fetch that runs out of room while another has just been cut
off waits, up to a minute, for that one's space, so two fetches that do
not fit together do not both fail. A fetch that was cut off never
waits, so no two wait on each other.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
Composer names are case-insensitive, so acme/private and acme/Private
share the version record and its integrity but get different filenames.
A second push under the other case was accepted and broke the integrity
check of the first archive. For ecosystems with one file per version,
any pushed file of the version now refuses the push, both before the
upload and atomically when recording it.
- Record a fetched artifact with its package and version in one
transaction that refuses to replace a pushed artifact, so a fetch that
finishes after a push keeps both the pushed file and its version's
integrity. Pushes and cache fills share this write, including the
compare-and-swap on the storage path that queues every replaced object.
- Don't open the stored file for ecosystems without a manifest.
- Derive multi-file versions from the file being named in the path.
- Validate DELETE paths like PUT paths, answering 400 for malformed ones.
- Parse ?sha256= with integrity.ParseHex.
- Record the package, version and artifact of a push in one transaction
that only inserts the artifact if no pushed copy exists. Of two
concurrent pushes of the same file exactly one wins, and the loser's
version integrity is rolled back with it.
- Check for an existing file before reading the request body, and send
Expect: 100-continue from proxy hosted push, so a refused re-push isn't
uploaded first.
- Leave an existing version record alone for multi-file ecosystems, so a
generic push doesn't reset a version cached from upstream.generic.
- Reserve the upstream.generic name "hosted" only while hosting is
enabled, and only intercept /generic/hosted/ then.
- Accept a file argument in proxy hosted delete for generic only.
- Mark multi-file ecosystems explicitly, and give parse a min/max range.
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
With cooldown or a version denylist, /pkg/<version> reads the filtered
packument through handleFilteredNPMVersionMetadata, which dropped the
stale flag.
The npm handler logged a debug line for every version whose tarball URL
it rewrote. Packages with hundreds of versions produced hundreds of lines
per packument, and with cooldown enabled, where the rewrite cache is
bypassed, again on every request. It now logs one line per document with
the number of rewritten URLs.
* Serve additional Maven repositories at /maven/{name}/
upstream.maven_repositories maps names to further Maven repositories,
each served at /maven/{name}/ from its own upstream, like
debian_repositories does for APT. upstream.maven stays the default at
/maven/ with unchanged cache keys and its Gradle Plugin Portal fallback;
named repositories don't fall back to the portal.
Named repositories cache their files under {name}/{filename} and hash
their metadata keys from name, upstream URL and path, so the same
coordinates in two repositories, which may hold different bytes, never
share a cache entry.
A repository name is matched before the default repository and would
hide group IDs starting with it, so names of two letters or less and
common top-level domains (com, org, net, io, ...) are refused at config
load. The documentation asks for names that can't start a group ID.
Fixes#295
* Refuse every Maven Central group root as a repository name
The hand-written list missed real roots such as tools (Jackson 3),
software (AWS SDK), jakarta, javax and junit, so a repository named after
one would have hidden those groups in the default repository. The name
check now uses the first path segments of all group IDs on Maven Central,
plus Google's androidx and android, embedded from the repository's
directory listing.
Names are compared case-sensitively, since routing is, so "ORG" is no
longer refused for hiding "org".
A named repository's files are now cached under its name and a hash of
its upstream URL, so pointing it at a different URL stops serving what
the old one sent.
* Refuse Gradle Plugin Portal roots and unroutable repository names
- The default repository falls back to the Gradle Plugin Portal, so its
group roots (gradle, nebula, ...) would be hidden by a repository of
the same name too. The embedded list now holds the roots of Maven
Central and the Plugin Portal.
- Repository names are limited to letters, digits, '.', '_' and '-', so
a name can't contain characters such as '?', '#' or spaces that never
reach the handler as configured.
- Named repositories are built once in WithRepositories, with their
cache prefix, instead of on every request.
Module zips are stored under the module's PURL, which lowercases golang
names, and under the filename {last element}@{version}.zip. So
github.com/Sirupsen/logrus and github.com/sirupsen/logrus shared one
cache entry although upstream serves different zips, and the second one
requested failed go's checksum check.
A module path with capitals now gets a hash of its exact spelling in the
filename. All-lowercase paths keep the plain name, so existing cache
entries stay valid.
A capitalised module cached before this change sits under its lowercase
twin's plain filename. When a cached zip's upstream URL names a different
module path, it is discarded and fetched again. Only the module path is
compared, so a cache filled from another upstream.go host is kept.
Fixes#466
With cache_metadata on, a failed upstream metadata fetch fell back to the
cached copy for any error. A 404 (package removed) and a 401/403 (the
proxy's credentials rejected) are definite answers, though, and serving
the old version list in their place kept them from reaching the client.
They now pass through: 404 as 404, 401/403 as 502 like other upstream
failures. Timeouts, connection errors, 5xx and 429 still fall back.
The npm, Composer and Chef handlers serve metadata through the rewrite
cache, which dropped the stale flag, so a fallback looked like a fresh
200. They now set the same Warning: 110 header as other metadata.
Fixes#499
* Use shared archives browser for cached artifacts
* Pin archives to combined browser and Hex changes
* Use archives v0.9.0 release
* Return 404 when browsing archive directories as files
* Delete cached artifacts through the API and the UI
cache_delete (PROXY_CACHE_DELETE) registers
DELETE /api/cache/{ecosystem}/{name}/{version}, which removes a
version's cached files, or only ?filename=, so the next request fetches
them from the upstream again. Records are cleared and the stored objects
queued for deletion like a replaced artifact. Pushed archives are left
alone: a whole-version delete skips them and naming one answers 409.
With it enabled, the version page shows a Delete button for each cached
file and one for the whole version. The UI calls /ui/api/cache so the
same ingress rules that guard /ui guard its deletes. Neither route has
authentication of its own, so the option is off by default.
* Load the cache delete script from a static file
Move the version page's delete-button JavaScript into
static/cache-delete.js and load it with a script tag, as
packages-list.js does.
---------
Co-authored-by: Giles Westwood <201443+gilesw@users.noreply.github.com>
* Separate artifact and metadata cache metrics on /metrics and the analytics page
* Serve metadata cache stats from a shared snapshot and split Grafana cache panels by kind
* Abort the response when a cached artifact fails its integrity check
* Read the seeded artifact's storage path from the database in the integrity test
* max_size now counts cached metadata as well as artifacts, and eviction drops whichever was used least recently, so metadata no longer piles up for good (#461). Metadata rows record when they were last served, an artifact not yet served counts from its download, and Arch stores a database blob again if eviction removed it under a refreshed row.
* Shorter comments on the metadata last-access migration.
* Show evicted metadata on the analytics page next to evicted artifacts, and cover the new counter in the metric surface test.
* Evicting metadata now drops the row and queues its path in one transaction, so a crash before the blob is deleted leaves it to reclaim instead of orphaning it.
* The last-access backfill now runs in the background in batches of 1000 instead of inside the migration, resuming after a restart, and until a row is backfilled eviction orders it by its write time.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
* Pub downloads without .tar.gz fetched the HTML version page from pub.dev and cached it as the archive, so one request broke that version for everyone after; they now get a 404, and the upstream URL is built from the escaped name and version so an encoded ? or # can't become a query or fragment.
* Pub archives that older versions cached from a path without .tar.gz are now discarded and fetched again, so versions that were already broken heal on the next download.
Every Composer archive download fetched the package's p2 metadata straight
from upstream to find the archive's URL, before the archive cache was even
checked. A cached archive cost one or two uncached Packagist requests per
package and install, and failed with a 502 while Packagist was down.
With cache_metadata on, downloads now look the dist up in the metadata the
proxy already caches for Composer, under the keys handlePackageMetadata uses.
Within the TTL that needs no upstream request, and when upstream fails the
stale copy still resolves the archive. A version or file the cached
documents do not list, such as one published since they were cached, is
looked up upstream as before.
When upstream answered a revalidation with 304, the proxy read the cached
document back from storage and then stored it again: a full upload and a
SHA-256 over identical bytes, for every popular package once per TTL.
A 304 now only moves the cache row's fetched_at on, keeping its digest.
The update is conditional on the row still recording the digest of the
document that was read, so a concurrent fetch that replaced it is not
overwritten; in that case, and for rows written before digests were
recorded, the document is stored as before.
The metadata route caches a package's packument in the abbreviated
install format, unless cooldown or npm_full_metadata needs the full one.
Tarball downloads read the packument from the same cache row but asked
for application/json. Once the row was due for revalidation, a tarball
the proxy had not cached yet fetched the full packument, which did not
match the abbreviated document's ETag, and stored it in its place. Until
the next refresh every client was then served the full document,
typically several times the size.
Both now use packumentAccept. Downloads read only dist.tarball,
dist.integrity and dist.shasum, which the abbreviated format carries; the
publish times the cooldown needs come with the full format it selects.
* Add mirror.schedules, a list of image tags the server re-mirrors at a fixed interval. Digests are rejected since they never move.
* Run each schedule through the normal mirror, one run after the other. The first run starts at a random point within ten minutes and later runs get ±10 % jitter, so several instances don't hit the registry at once.
* Start the schedules with the server and let shutdown wait for a running refresh before storage and database are closed. Docs included.
* Check cached layers during a mirror without opening them, so a refresh no longer counts as a cache hit and doesn't keep scheduled images at the top of LRU eviction.
A response arriving during a hold was sent before it began, so a late
success no longer releases the host; only one after the hold has ended
does.
Artifact fetches check RateLimitGate.Held before calling the fetcher, so
held cache misses fail with the HeldError without counting as failures
in the host's circuit breaker.
Every handler that answers an upstream fetch error with 502 now answers
a held request with 503 and Retry-After instead.
OnHold and RecordUpstreamRateLimitHold drop the unused duration.
- Keep a gzip copy of each rewrite cache entry when it is smaller, made once
per rewrite and counted against metadata_rewrite_cache_size
- Serve it with Content-Encoding: gzip to clients that accept gzip, with an
ETag of its own ("<sha256>-gzip") and Vary: Accept-Encoding
- Rewrites made outside the cache (cache disabled, cooldown on) stay
uncompressed
- rewriteCache.get and rewrite return rewrittenMetadata
A 429, or a 503 with Retry-After, from an upstream host now stops the
proxy sending that host anything for as long as it asked (capped at an
hour), or for a minute doubling up to fifteen when no Retry-After came.
Held requests fail locally: cached metadata is served stale as before,
and anything uncached answers 503 with the remaining Retry-After so the
client paces itself too.
The gate is a RoundTripper under the auth transport, so metadata,
artifact and passthrough fetches all obey it. Metadata fetches had no
retry or circuit breaker, so a rate-limited upstream kept receiving one
request per cache miss for the whole block. /health lists held hosts
under upstream_holds and proxy_upstream_rate_limit_holds_total counts
holds.
Checksum and .asc files beside a versioned artifact are as immutable as the
artifact, but were routed through the metadata path: uncached until a
first successful fetch, with no retry or circuit breaker. A build pulling
hundreds of artifacts sent as many checksum requests straight to Maven
Central, which rate-limited the proxy, after which every further checksum
request still went upstream and failed.
Only maven-metadata.xml and its own checksums stay on the metadata path,
since the version listing changes with each release.
TestServerKeepsFetchAfterClientHangsUp gave the upstream fetch 5s to start
and the follow-up request 5s to be answered. On a Windows CI runner, where
the whole test took 13s, the follow-up timed out awaiting headers with
nothing wrong. The test checks the outcome, not the speed, so allow 30s
for both. A real hang still fails.
A version's integrity is a single value shared by every archive cached
for it, but a dev branch caches one archive per commit. Recording the
listed head's shasum made a cache hit for an older pinned commit fail its
check, so its archive was discarded and fetched again, which cleared the
head's recorded shasum in turn. The listed shasum only ever describes the
current head, and Composer checks dist.shasum itself.
- Skip the shasum for dev versions
- Test a pinned commit, then the head, then both again from the cache
npm versions published before dist.integrity existed carry only a SHA-1
dist.shasum, and Composer metadata lists only a SHA-1 dist.shasum, so
neither could be checked.
- Accept native integrity given as an algorithm-prefixed hex checksum,
verified with integrity's exact-digest check, so a recorded "sha1-<hex>"
is also checked on cache hits
- npm: fall back to dist.shasum when dist.integrity is missing or cannot
be parsed; a usable dist.integrity still takes precedence
- Composer: verify an archive against its dist.shasum, except another
commit of a branch fetched in place of the listed one, which the listed
shasum does not describe
- Fetch unverified as before when no shasum is listed, as Packagist does
for GitHub archives, or it is not a SHA-1 digest
Credentials configured for a token service were applied to any challenge
naming it. A host the proxy fetches from, such as one in a Composer dist
URL or a Helm index, could answer a /v2/ request with a Bearer challenge
naming that realm, and the proxy fetched a token with the configured
credentials and sent it to that host. A token cached from a legitimate
pull was handed over the same way.
- Follow a challenge to a realm with configured credentials only for a
registry that has configured credentials of its own, or is on the same
site (registrable domain) as the realm, as registry-1.docker.io is with
auth.docker.io; otherwise pass the 401 back
- Keep following anonymous challenges for any registry, as before
- Document the rule
Package routes share an origin with /ui, and serve bodies whose
Content-Type comes from upstreams or from whoever pushed a hosted package.
An artifact served as text/html or image/svg+xml ran its scripts with
access to the UI.
- Send X-Content-Type-Options: nosniff and Content-Security-Policy: sandbox
on every package-route response: cached and streamed artifacts, cached
metadata, relayed upstream responses and errors
- Archive files browsed from /ui already get the same two headers
The proxy cached the first tarball upstream sent for a version without
checking it against the packument, so a tampered tarball was served to
every client after it.
- Pass the packument's dist.integrity to the existing integrity checks, so
a fresh tarball is verified as it is stored or streamed; a mismatch is
discarded and returns 502, or aborts the streamed response
- Record it with the cached version, so cache hits are verified against it
through the same check as before
- Verify with integrity.Result.Verify, which accepts any digest of the
strongest algorithm listed
- Fetch unverified as before when dist.integrity is missing or cannot be
parsed, or the packument cannot be fetched
Docker sends one Accept header per media type. Reading only the first
forwarded just that type upstream and ruled out cached manifests of the
others, so pulls failed when the upstream was down.
- GetLocalArtifact went through checkCache, which reports a miss when StreamArtifacts is set, so npm and Composer downloads fetched the upstream archive instead of the pushed one
- Split readStored out of lookupCache and read pushed artifacts through it regardless of StreamArtifacts
- Keep the denylist check and the 503/404 answers for unreadable or missing pushed artifacts
- Test npm and Composer downloads of pushed archives with StreamArtifacts set
Refs #133
proxy list and proxy delete sat next to serve, stats and mirror and read as cache operations. The commands are now proxy hosted push, proxy hosted list and proxy hosted delete, before a release makes the names hard to change. proxy hosted without a known subcommand prints the three.
Refs #133
Pushing and downloading refuse denied versions, but npm and Composer metadata listed every pushed version. A version denied after it was pushed stayed listed, so resolvers picked it and then got 403 on the download. localVersions now leaves denied versions out, which covers both ecosystems and the hosted-only namespaces; npm latest falls back to the highest allowed version. /api/hosted still lists them, so operators can see and delete them.
Refs #134
The manifest lookup took the first composer.json at the root or inside a top directory, in zip order. Dist archives keep composer.json files for tests and examples, so tests/composer.json stored before the root file refused the push with a name mismatch. The root composer.json now always wins. Without one, the manifest inside a top directory is used only when there is exactly one; more than one refuses the archive as ambiguous.
Refs #134
checkCache answered nil both when a stored object couldn't be opened and after an integrity failure had cleared the record. Download handlers then fetched the upstream archive for the same version and filename, and recording it turned the pushed row into an upstream one without its manifest and queued the pushed object for deletion. One storage error while downloading a patched version served the unpatched upstream archive and dropped the push.
checkCache now looks at the record on a miss. For a pushed artifact it returns ErrHostedArtifactUnavailable when the object is recorded but couldn't be served, and ErrHostedArtifactMissing when no object is recorded any more. Every path that would fetch upstream goes through it, so none of them can replace the pushed artifact. Downloads answer 503 and 404 for the two.
Refs #133
The artifact source change added source = excluded.source to the packages upsert as well. source isn't in that INSERT's column list, so every package upsert set the column to NULL. The packages upsert is back to what it was, with a test that an existing source survives it.
Refs #133
A package the upstream doesn't know is served from its pushed versions, but the upstream is still asked about it. Once someone publishes the same name there, their versions are listed next to the pushed ones and a version range can resolve to them.
hosted.npm_scopes and hosted.composer_vendors (PROXY_HOSTED_NPM_SCOPES, PROXY_HOSTED_COMPOSER_VENDORS) name namespaces whose packages are served from pushed versions alone. Their metadata is built from what was pushed, npm latest included, and anything not pushed answers 404. The upstream is never asked about them. Packages in other namespaces are proxied as before.
The npm metadata handler's decisions about pushed versions move into serveLocalMetadata.
Refs #133
Refs #134
The packument lists pushed versions without cooldown, but the tarball download checked cooldown against the upstream's publish times before looking at the cache. A pushed override of a version the upstream published recently was listed and then withheld with 404. Pushed tarballs are now served right after the denylist check.
Refs #134
With hosted packages enabled, a package's p2 metadata lists its pushed versions, built from composer.json without the root-only keys, with a zip dist at the proxy and the time they were pushed. dist.reference is the archive's sha256 so a re-push isn't served from Composer's cache, and shasum is left empty. Dev versions go to the ~dev file and tagged versions to the other.
A pushed version takes the place of an upstream entry with the same version, with or without a leading v. Pushed entries follow the upstream ones and go through the same minified writer, so fields an earlier upstream entry set and the pushed one lacks are written as __unset rather than inherited.
When the upstream doesn't know the package, the document holds only the pushed versions. Like npm, documents with pushed versions bypass the rewrite cache.
Refs #134
With hosted packages enabled, the packument of a package with pushed versions lists each of them, built from its package.json with dist pointing at the proxy and carrying the archive's sha512, and the time it was pushed. A pushed version takes the place of an upstream entry with the same version. Cooldown and the denylist still filter only upstream versions, and dist-tags stay as upstream has them.
When the upstream doesn't know the package, the packument holds only the pushed versions, with latest set to the highest of them. Requests for a single pushed version are answered from its entry.
Rewrites that include pushed versions bypass the rewrite cache, which is keyed by the upstream document alone, so a push or delete shows on the next request and changes the ETag. Packages without pushed versions take the cached path as before, and with hosted packages disabled there is no extra lookup.
Refs #134
Resolvers take a version's dependencies from the registry metadata, not from the archive, so listing a pushed version needs its manifest. A push now reads package.json from the top directory of an npm tarball, or composer.json from the root or the single top directory of a Composer zip, and keeps it in the new artifacts.manifest column (migration 013_add_artifact_manifest).
The manifest has to name the pushed package and version. A Composer manifest may leave the version out, and its name is compared case-insensitively. An archive that can't be read or whose manifest names something else is refused with 400 and removed from storage.
The version's integrity is now the archive's sha512, which npm clients expect in dist.integrity.
Refs #134
proxy push, proxy list and proxy delete call the /api/hosted endpoints of a running proxy. The proxy URL comes from -url or PROXY_URL and the token from PROXY_HOSTED_TOKEN, so it never ends up in shell history or process listings.
Document the hosted config block, the endpoints, the supported ecosystems and how pushed archives interact with eviction.
Refs #133
With hosted.enabled, PUT /api/hosted/{ecosystem}/{name}/{version} stores the request body as the archive for that version, marked local. It replaces any cached upstream copy and is served from the cache without contacting the upstream. GET /api/hosted/{ecosystem}/{name} lists what was pushed and DELETE removes a version again, after which clients get the upstream archive if there is one. Pushing and deleting need the bearer token; listing is open like the rest of the API.
Download handlers look archives up by the filename clients request, so a push is stored under the name its ecosystem derives from name and version: {name}-{version}.tgz for npm, without the scope, and {package}-{version}.zip for Composer. Other ecosystems are refused for now.
The Composer download handler resolves the upstream metadata before it looks at the cache and answers 404 for versions the upstream doesn't list. It now serves a pushed archive first. Only artifacts marked local take that path, so cached upstream archives still go through the metadata and commit checks.
The version's integrity is set to the pushed bytes, so the cache's integrity check verifies them rather than an upstream digest. Names and versions with path separators, dot segments or control characters are refused before anything is stored.
Refs #133
hosted.enabled turns on pushing packages to the proxy and is off by default, like mirror_api. A pushed version is served in place of the upstream version with the same number, so anyone who can write could swap a package for every client. Enabling it therefore requires hosted.token, which supports ${VAR_NAME} expansion and can be set with PROXY_HOSTED_TOKEN. hosted.max_upload_size caps one archive and defaults to 100MB.
requireBearerToken compares SHA-256 digests of the presented and configured tokens in constant time and answers 401 with a WWW-Authenticate challenge otherwise.
Refs #133
Artifacts get a source column: upstream for cached copies of a registry, local for archives pushed to the proxy. Existing rows become upstream through the new 012_add_artifact_source migration.
Local artifacts aren't a cache, so the LRU size limit doesn't count or evict them and storage retention never selects them. They are only removed by deleting them. A local artifact has no upstream URL and stores it as empty.
Refs #133
The compare endpoint buffers both archives whole, up to 512 MB each, and
anyone who can reach the UI can call it. A handful of parallel requests for
large cached versions was enough to exhaust the proxy's memory.
- Run at most two diffs at once; refuse more with 503, code BUSY and
Retry-After instead of queueing them
- Keep the per-archive cap, which legitimately large packages need
- Move the compare test's setup into a helper shared by the new tests
Artifacts had no size limit. A fetch was bounded only by the server's write
timeout, and storage.max_size evicts only after the bytes are written, so
any client could fill storage or keep fetches busy by requesting huge
artifacts.
- Add artifact_max_size (PROXY_ARTIFACT_MAX_SIZE), unlimited by default
- Refuse an artifact whose declared size is over the limit before reading
it, and fail one of unknown size once the download passes the limit
- Apply it in the fetcher so stored, streamed and mirrored artifacts all
honour it; nothing is cached and clients get 502
Handlers set the configured upstream auth header on the request itself
before handing it to the client. When upstream redirected to another host,
Go's client copied that header to the next hop: it strips only
Authorization and Cookie, so a "header" type credential such as
X-JFrog-Art-Api reached the redirect target. The auth transport already
applies credentials per hop, scoped to each hop's URL.
- Remove Proxy.applyUpstreamAuth and Proxy.AuthForURL; auth comes only
from the transport
- Add a server test that redirects a header-authenticated upstream to
another host and asserts the header stays with the upstream
- Wire handler tests' auth through the transport, as the server does
The compare page loaded diff2html from cdn.jsdelivr.net without a version
or integrity hash, so whatever jsDelivr served as the latest release ran
with the proxy origin's privileges, and the page broke where the proxy
cannot reach the internet.
- Vendor diff2html 3.4.56 (MIT) under static/vendor like tailwind and lucide
- Load its script and stylesheet from /ui/static/vendor
- Fail a test when any template loads a script or stylesheet from another origin
The source browser built its file tree with onclick handlers such as
loadFile('${escapeHTML(file.path)}'). The browser decodes HTML entities in
an attribute before running the handler, so the ' that escapeHTML
produces turns back into a quote and ends the string. A cached package
with a crafted file name ran script on the proxy origin for anyone who
browsed it.
- Carry the path in data-path and dispatch clicks from one delegated
listener on the file tree
- Render the binary download button as a link instead of an onclick
that assigns window.location
- Fail the template tests when an inline handler interpolates data
Cooldown only filtered the metadata. A composer.lock names the proxy's
/files URL directly, so `composer install` fetched versions still inside
the cooldown window without ever seeing the filtered metadata. npm already
checks the publish time on tarball downloads.
- Return the version's publish time with its dist URL, following the
minified format's inheritance
- Answer 404 for a version in cooldown before fetching its archive
- Read the download lookup's metadata through ReadMetadata, so it has the
same size limit as other metadata reads
gocloud commits whatever a writer received when it is closed, unless its
context was cancelled first. Store closed the writer after a failed copy,
so every download that broke off midway left a partial object under its
fetch path. No database row points at it, so it was never served, but
nothing deleted it either and the orphans kept taking disk space.
- Cancel the writer's context before closing it when the copy fails
- Add a test storing from a reader that errors midway
Packages are stored as pkg:deb/{name}, without a vendor and whichever configured archive they came from. Package rules may name a vendor the way PURLs usually do (pkg:deb/debian/curl, pkg:deb/ubuntu/curl); it's dropped, so a rule covers the package from every archive. The docs row warns that Debian and Ubuntu archives only keep the current version per suite.
The mirror can't fetch deb packages, so there's no mirror test.
Refs #306
On a build network where clients cannot reach sum.golang.org at all, the
proxy should keep answering for the database during an outage, since a 404
on supported would only send the go command somewhere it cannot go. With
the switch set, a cached tree of any age is enough; without it, the tree
has to be fresh, as before.
That does not make every cached module verify. The go command checks a
lookup against tiles of the tree that came with it, and sum.golang.org
attaches a different tree to almost every lookup, so it can ask for a tile
nobody fetched through the proxy before. configuration.md says so.
While the database is known to be unreachable, offline mode also serves
cached lookups and tiles straight away. Otherwise each of them would sit
through the upstream timeout before the fetch falls back to the cache,
and a build would take minutes for as long as the outage lasts. A lookup
or tile that falls back to the cache renews what the proxy knows about
the database, unless a probe has reached it since the request started.
The go command asks supported only once per process, so a long build would
otherwise be back to waiting a minute into the outage. recordTimeout is
now recordFailure, since lookups use it too.
The option needs cache_metadata, since without it there is no cached tree
to fall back on, and the config refuses the combination.
The sentence promised a direct fallback whenever sum.golang.org is
unreachable. That's only true once the cached tree has gone stale; a fresh
one keeps the proxy answering. configuration.md has the details.
Also the cases around a silent or slow database: a fresh cached tree still
gets a yes on supported, a stale one does not, a late answer from the
database clears the remembered failure, a fallback to the cached tree is
not mistaken for an answer, and an answer still counts when the cache
write fails afterwards. The damaged tile test guards the size check on the
cached copy of a complete tile: it shortens the stored blob and expects
the tile to be fetched again, also past the TTL from a database that sends
an ETag.
A block page in place of a lookup must not replace the cached record, not
even one that starts like a record. The lookup fixtures now look like real
records, since lookups are checked too.
Two problems with the check behind supported. The fetch it waits on is
shared with other callers and only bounded by the upstream timeout, which
defaults to 30 seconds. A firewall that silently drops packets to
sum.golang.org would stall the first go command every minute for that long,
and the go command itself has no timeout on this request. Now we wait five
seconds at most; the fetch carries on in the background, records how it
went and fills the cache for the next check. A timeout only counts as a
failure if the fetch hasn't succeeded in the meantime.
The fetch reports no error when it falls back to the cached tree after the
database failed, so the metadata result now says when it did, and the
check treats that as the database not answering. A fresh cached tree
(within metadata_ttl) still lets a timed-out check say yes, since the
fetch only falls back once upstream has failed, which a silent database
delays past our timeout. An older tree does not count: once the proxy has
lost the database, clients go there directly, as they would without the
proxy.
And the check believed any 200. Egress proxies tend to answer blocked hosts
with a 200 HTML page, which we would have cached as the signed tree and taken
as proof that the database is reachable. latest now has to start with
"go.sum database tree", or it is refused like a bad tile.
Lookups had the same gap. A block page cached as a lookup would also
replace the good copy that is served during an outage, so a lookup now has
to be a record followed by a signed tree, the way the go command reads it.
Complete tiles are never fetched again once cached, so a bad response would
stick around until someone deletes it by hand. That could be an error page
from a proxy in between that comes back with 200.
A hash tile is always W*32 bytes, so anything else is refused now and not
stored, and the same check runs when a cached tile is served. Data tiles
have no fixed size and only must not be empty.
A cached tile that fails the check is fetched again without asking the
database whether it changed. Past the TTL the cache would ask, and a
database that sends an ETag would answer 304 and hand back the damaged
copy every time.
The fake database in the tests now guards its maps with the mutex, since the
tests change them between requests.
Once a proxy answers 200 on supported, the go command sends all checksum
requests there and never goes to the database directly. A proxy that cannot
reach sum.golang.org would therefore break every build that needs a lookup.
supported now fetches latest first, through the cache, and answers 404 if
that fails and nothing is cached, so clients fall back like before. A failed
check is remembered for a minute so an unreachable database doesn't make
each go command wait for an upstream timeout.
Everything under /go/sumdb/ was passed on to the module upstream. But
proxy.golang.org doesn't serve the checksum database, so those requests
always came back 404 and every client had to talk to sum.golang.org itself.
The Go handler now serves /sumdb/{name}/ for the databases in
upstream.go_sumdbs, which defaults to sum.golang.org. Lookups, latest and
tiles come from the database through the metadata cache. Complete tiles
never change, so once they're cached we don't ask again. Unknown names and
paths that aren't part of the protocol get a 404 without going upstream.
If the database answers 404 we pass on up to 4 KiB of its text, because the
go command prints it and it tells you why a version is missing. For that the
not-found error from the metadata fetch now carries the start of the body.
Other handlers still just see ErrUpstreamNotFound.
The first matching pattern names the package, so an upstream's own
patterns now take precedence over overlapping built-ins and keep the
package names, cache entries and rules already keyed on them.
Downloads are routed as /{channel}/{arch}/{filename}, so the stored name is always {channel}/{name}. A key like pkg:conda/conda-forge/label/rc/numpy passed validation and matched nothing; it's now refused.
Refs #306
parseTarballFilename split at the last hyphen before a digit, so amqp_client-4.1.5-1.tar was stored as package amqp_client-4.1.5. Hex names can't contain a hyphen, so it now splits at the first one. The docs caveat is gone.
Refs #306
The handler stores packages as {channel}/{name}, which purl writes as pkg:conda/conda-forge%2Fnumpy. Package rules name the channel as the namespace (pkg:conda/conda-forge/numpy) and get mapped to that form, with the package name lowercased the way conda treats it. A key without a channel or with an empty part could never match and is refused.
The mirror can't fetch conda packages today, so there's no mirror test.
Refs #306
Tarballs are stored under the name from the tarball filename, and purl keeps hex names as they are, so package rules use the default mapping with the name as published.
The filename parser splits versions like 4.1.5-1 at the wrong hyphen and files them under a package like amqp_client-4.1.5, so package rules don't reach them; the docs row says so until the parser is fixed.
Refs #306
The mirror passed ids through as written, so pkg:nuget/Newtonsoft.Json@13.0.3 was stored under a copy no client is served while the handler kept serving its lowercased one. GetOrFetchArtifact now lowercases nuget names, and the docs no longer ask for lowercase ids in mirror lists.
Refs #306
parseGemFilename split at the last hyphen before a digit, so sorbet-static-0.5.11002-universal-darwin-22.gem was stored as package sorbet-static-0.5.11002-universal-darwin. It now splits at the first one, which only misreads gem names with a -<digit> segment. The docs caveat is gone.
Refs #306
The history is process-wide, and tests cleared it only after running, and
not all of them. Records from earlier tests or an earlier -count repetition
reached later assertions: TestRecordUpstreamOperations left a record for
TestRecentFailuresNewestFirst, and the server tests never cleared it.
- Export ResetRecentFailures so tests outside the metrics package can use it
- Clear the history before and after every test that records or reads
failures directly, in metrics, server and handler
- Assert exactly one failure in the handler tests instead of looking only
at the newest
Source archives are stored under the registry identity, pkg:generic/swift-registry/{scope}.{name}, lowercased. Package rules write the scope as the namespace (pkg:swift/apple/swift-nio) or use the stored form, in any case. A source repository key like pkg:swift/github.com/apple/swift-nio doesn't name a registry identity and is refused.
The mirror can't fetch swift packages, so there's no mirror test.
Refs #306
Since #431 the handler stores Conan archives under pkg:conan/{name} with the version as the version and the revisions and any user/channel in the filename. purl keeps conan names as they are, so package rules use the default mapping, and pkg:conan/zlib covers every version, revision and user/channel of the recipe.
The mirror can't fetch conan packages, so there's no mirror test.
Refs #306
Archives are stored under pkg:composer/{vendor}/{package}, which purl lowercases, so case in a package rule doesn't matter. A key without a vendor could never match and is refused. Packages that git-pkgs wrote as "packagist" fall under the composer rules too.
The mirror can't fetch composer packages today (its resolver has no composer support, and #415 would file them under packagist), so there's no mirror test.
Refs #306
Crates are stored under the name they were published with, and purl keeps cargo names as they are. Package rules have to match that name exactly, so pkg:cargo/inflector doesn't cover Inflector.
Refs #306
Generic upstreams pointed at archive.apache.org, nodejs.org or
dl.google.com/go now cache their versioned downloads as artifacts without
any generic_artifacts patterns. Configured patterns add to the built-in ones.
- Start from the schema before migration 011 on SQLite and PostgreSQL, with
npm and Composer metadata cached, and check the rows and stored bytes
survive the migration unchanged
- After the upgrade, check two replicas serve the cached document with
matching ETag and Last-Modified and answer conditional requests with 304
- Change the upstream document and check neither replica answers the old
validators with 304, and both agree on the new ones
- Give the PostgreSQL handler test a database of its own, so it shares no
tables with the database package tests on the same server
The handler stores NuGet ids lowercased, but purl keeps nuget names as written, so package rules get lowercased before they're matched. Case in a rule doesn't matter.
The mirror stores ids as given, so a mixed-case PURL ends up as a copy clients never get. Until that's fixed, the docs ask for lowercase ids in mirror lists.
Refs #306
Packages are stored as pkg:rpm/{name}, without a vendor and with the name's case kept. Package rules may name a vendor the way PURLs usually do (pkg:rpm/fedora/curl); it's dropped. The docs row warns that Fedora and EPEL update repositories only keep the latest build.
The mirror can't fetch rpm packages, so there's no mirror test.
Refs #306
Packages are stored under the name from the download path, and purl keeps pub names as they are, so package rules use the default mapping with the name as published.
Refs #306
Gems are stored under the name from the gem filename, and purl keeps gem names as they are, so package rules use the default mapping and match the name as published, case included. Packages that git-pkgs wrote as "rubygems" fall under the gem rules too.
The filename parser splits platforms that end in a number (universal-darwin-22) at the wrong hyphen and files those gems under a package like sorbet-static-0.5.11002-universal-darwin, so package rules don't reach them; the docs row says so until the parser is fixed.
The mirror can't fetch gems today (it asks the resolver for "rubygems", see #415), so there's no mirror test.
Refs #306
Julia takes no package rules, since its PURLs need a uuid qualifier the cache doesn't store; configuring one already fails validation. The julia rule and the default cover package tarballs, artifacts and registry snapshots alike. Everything there is content-addressed, so an evicted tarball comes back unchanged.
Refs #306
Helm takes no package rules: charts are cached under the configured repository's name and the chart digest, not under the chart name, so a key like pkg:helm/nginx would name a repository. The helm rule and the default cover charts from every HTTP repository; charts pulled over OCI go through the container handler and fall under oci.
Refs #306
Dates were kept in each process's rewrite cache, so replicas sharing a
database and storage dated independently: one dated a document, upstream
changed in that second, and another gave the new document the same date,
so a client holding it got 304 and kept the old document.
- Date from metadata_dates instead of process memory: a package's date is
the second it started serving its current representation, recorded once
for every replica
- Switch a package to another representation only in a second later than
its recorded date, and serve it undated until then
- Date nothing in the second the process started, so dates clients hold
from before this change, or from the process it replaced, stay older
- Serve no date ahead of the local clock, recorded by a replica whose clock
is ahead
- Date rewrites without the rewrite cache too; the cache keeps only ETags
- Add two-replica, pre-fix date and clock skew regression tests
Replicas sharing a database need one place to agree on the Last-Modified
of rewritten metadata, which process memory cannot give them.
- Add metadata_dates, keyed by ecosystem, proxy URL and package, holding the
ETag being dated and its date in Unix seconds, so no zone is involved
- Insert a package's first date only if no replica recorded one first
- Swap a date only if it is still the one read, so of two replicas
switching a package at once only one succeeds
- Create it in both schemas and in migration 011
Two false 304s remained. A rewrite kept the date it was made, so when
upstream went from A to B and back to A, A's cached rewrite came back with
a date earlier than B's and a client holding B got 304. And a second
rewrite in the same second was dated a second ahead of the clock, a date
only that process remembered: after a restart a changed document could be
dated earlier than one the client held.
- Record which rewrite each package is serving and since when, and date a
rewrite by the second its package switched to it, including back to one
still cached
- Never date ahead of the clock: a rewrite switched to in a second that
already dated another for the package goes undated until the next second
- Date nothing in the second the cache starts, the last second the previous
process may have dated a rewrite in
- Drop a package's record when its current rewrite is evicted, so records
stay bounded by the cache; forgetting one dated in the current second
holds back dates for new packages until the next second
Last-Modified was upstream's date clamped to the process start, so
different documents could share a date: after a restart, a cached document
and the newer one a refresh fetched were both dated at startup, and a client
holding the first got 304 for the second.
- Date each rewrite by when the rewrite cache first made it, and keep that
date while the entry stays cached
- Give every new rewrite of a package a date later than the last one it got,
even within the same second or after its entry was evicted, so no two
representations share a date; dates are remembered only until the clock
passes them
- Send no Last-Modified when the rewrite cache is off, since nothing records
when an uncached rewrite was made; cooldown still sends none
- Drop the upstream Last-Modified from metadata lookups, which nothing uses
now
- With cooldown on every rewrite runs, and hashing each rewritten body for
its ETag made every npm metadata request about 20% more expensive
- Hash what fixes the output instead: the upstream digest, proxy URL,
package name, the versions cooldown and the denylist kept, and a
per-process nonce standing for the startup configuration
- Carry the stored document's digest and upstream Last-Modified with the
metadata lookup result, so neither is read from a second cache row lookup
that a concurrent refresh could have changed
- Fall back to hashing the body when the digest is unknown, and keep the
body hash with cooldown off, where it is computed once per cached rewrite
- Rewritten metadata went out with only Content-Type, so pnpm, npm and
Composer could never revalidate and downloaded every document in full
- ETag is the SHA-256 of the rewritten body, computed once per rewrite and
kept with the rewrite cache entry; exact under cooldown as well
- Last-Modified is the upstream Last-Modified, never earlier than the process
start, since a restart can change the rewrite; omitted with cooldown on
- Answer If-None-Match, and If-Modified-Since when there is no If-None-Match,
with 304 on the rewrite fast path and on the fetch path
* npm: version lookup returns 502
Looking up a specific version will trigger a 502 instead of return the version metadata.
* scope only not handled explicitly
* honor cooldown in version lookups
tested (set default cooldown to 7d and 8.0.7 is released four days ago or something):
```
$ curl -sS 'http://localhost:8080/npm/@babel%2Fcore/8.0.7' | jq
{
"error": "package not found"
}
```
* package not found -> version not found
* gocognit
* Enable storage retention for npm
npm stores packages under the PURL purl builds from the full name, so package rules use the default mapping. Both pkg:npm/@babel/core and pkg:npm/%40babel/core work.
Refs #306
* Tighten npm retention tests and docs
The docs now say npm package rules match the name exactly, so pkg:npm/jsonstream doesn't cover JSONStream. The refetch, package rule and mirror tests were weaker than their names and now check what they claim.
Refs #306
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
* Enable storage retention for pypi
PyPI names are stored the way purl normalizes them, lowercased and with _ turned into -, so Typing_Extensions and typing-extensions hit the same package. The mirror can't fetch PyPI files today, so there's no mirror test.
Refs #306
* Correct the pypi package rule note
Dots in PyPI names aren't normalized, so zope.interface and zope_interface files end up under two packages. The docs now say to add a rule for each.
Refs #306
* Mention repeated separators in the pypi rule note
Runs like "__" aren't collapsed either, so a__b is stored as a--b.
Refs #306
Module paths are stored lowercased, so package rules match whatever case they're written in. A major version like /v2 is its own module and needs its own rule. Keys in the !-escaped form from proxy URLs and the module cache are refused, since they could never match.
Refs #306
Blobs are stored under the repository name of the pull, which purl writes as one name (pkg:oci/library%2Fnginx). Package rules write it as a path, pkg:oci/library/nginx, and get mapped to that form; keys with an empty segment, a tag or digest, or a bare upstream/{name} prefix are refused. Named upstreams keep their prefix, also when containerd reaches them through ns. Homebrew bottles and Helm charts pulled over OCI fall under oci too, bottles under Homebrew's image name (openssl@3 is homebrew/core/openssl/3). Manifests live in the metadata cache and aren't touched.
Refs #306
Package rules use pkg:maven/{group}/{artifact} with the case as published. Keys without a group or with the group written as a path can never match and are refused. Files from the Gradle Plugin Portal fallback fall under the maven rule; the Gradle build cache isn't affected.
The mirror can't fetch Maven packages today (it passes group/artifact where the resolver wants group:artifact), so there's no mirror test.
Refs #306
- Add ui_failure_details / PROXY_UI_FAILURE_DETAILS, off by default
- Each failure row expands to its latest failures, newest first
- Show a note naming the config key when failures exist and the flag is off
- Document the flag in README, configuration reference and example config
- Failure recorders take a FailureDetail: PURL, filename, URL, storage path, error
- Keep the last 20 failures per counter series in memory, reset on restart
- Strip userinfo, query and fragment from URLs, including those inside error text
- Pass the detail from upstream, storage, integrity, health probe and scan call sites
- Package names and URLs stay out of Prometheus labels
Source, archived and binary packages are stored under pkg:cran/{name}, the name from the filename with its case kept, and purl keeps cran names as they are. Package rules use the default mapping and match the name as published, so pkg:cran/rcpp doesn't cover Rcpp.
The mirror can't fetch cran packages, so there's no mirror test.
Refs #306
Packages are stored as pkg:apk/alpine/{name} whichever configured repository they came from, so a package rule covers the package in all of them. Rules may leave out the alpine namespace (pkg:apk/curl); another namespace could never match and is refused. The docs row warns that Alpine mirrors only keep the current build, so an evicted older one can't come back.
The mirror can't fetch apk packages, so there's no mirror test.
Refs #306
* Cache configured version-pinned paths of generic upstreams
The generic handler only knew GitHub release asset paths were immutable, so
any other download, such as Maven from archive.apache.org, went through the
metadata route: streamed uncached by default, or kept as metadata and never
listed as a package.
upstream.generic_artifacts maps a generic upstream to regular expressions
for its version-pinned paths. A pattern must match the whole path after
/generic/{name}/ and has version and file named groups, plus an optional
name group. Matching downloads take the release asset route: cached in the
artifact cache, served while the upstream is down, and listed under the
generic ecosystem. Other paths are unchanged.
* Fix generic artifact cache identity and query forwarding
* Serve generic release assets cached under the unhashed key
Release assets cached before hashed keys used {repository}/{asset} and
became unreachable after an upgrade, so an offline upstream returned 502.
Serve such an entry when its recorded upstream URL matches the request.
---------
Co-authored-by: Giles Westwood <201443+gilesw@users.noreply.github.com>
* Proxy the OCI referrers API in the container handler
GET /v2/{name}/referrers/{digest} used to fall through to a plain 404,
so clients only ever saw referrers stored under the sha256-<hex> tag
schema. Registries that keep SBOMs and signatures behind the referrers
API looked empty through the proxy.
The index is now fetched from the upstream that the repository routes
to and cached like the tag list: fresh within the metadata TTL,
revalidated with If-None-Match, and served stale when the upstream is
down. artifactType is not forwarded, so one cached row covers every
filter and clients filter the full list themselves, which they do when
OCI-Filters-Applied is absent. An upstream 404 is passed through
untouched, so clients still fall back to the tag schema for registries
like GHCR. When the upstream cannot answer and nothing is cached, the
proxy keeps answering 404 as before rather than failing the request.
Closes#144
* Cover absolute upstream links in the referrers link test
The second page now answers with an absolute Link pointing back at the
upstream host, and the test checks that it comes back rewritten to the
proxy. That was missing so far, and it also gives the upstream variable
a reason to be declared before the handler closure, which staticcheck
complained about.
* Read cached referrers through a small shared loader
loadContainerReferrers was a line-for-line copy of loadContainerTags with
the types renamed, which dupl flags in CI. The part that opens the cache
row and reads the body now lives in loadContainerMetadata next to
storeContainerMetadata, and the referrers loader only maps the row onto
its struct. The tag and manifest loaders are left as they are.
* Make the manifest routing test notice which handler answered
For foo/manifests/referrers/<digest> the manifest handler and the
referrers handler send the very same upstream path, so the test passed
even with the referrers case moved in front of the manifest case. The
fake upstream now rejects requests that only accept an image index,
which is what the referrers handler sends.
* Check that a cached referrers row links back to whoever reads it
A repository can be reached as upstream/<name>/<repo> and, when the
default registry is the same host, as plain <repo>. Both end up in the
same cache row, so the pagination Link has to be rewritten when the row
is served, not when it is stored. The link test now reads the row
through both paths; with a store-time rewrite the second path would get
the first one's link.
* Make the referrers ETag test prove the revalidation
Without If-None-Match the fake upstream just sends the full index again,
so the old assertions held either way. The test now counts the 304s the
upstream sent and expects exactly one.
* Reword which tools use the referrers API in the README
The cosign mention was too broad. cosign reaches the referrers API for
its newer signature bundles, while the old .sig tags keep going through
the normal manifest path. Also rewraps the paragraph, one line had run
past the others.
* Check that a 304 restarts the referrers cache TTL
The ETag test now backdates the cached row instead of running with a
zero TTL. After the upstream answers 304, a third request has to come
from the cache. Before, the test would also have passed if the 304
branch forgot to bump fetchedAt and every request kept revalidating.
* Test the referrers Content-Type fallback
None of the fake upstreams left out Content-Type, so the default to the
image index type was never exercised. oras-go rejects anything else, so
it is worth pinning, both for the first fetch and for the cached copy.
* Test referrers digests with extra characters around them
The invalid-request table only had digests that were too short or used
an unknown algorithm, so it would not notice if the digest pattern lost
its anchors. Two cases now put junk before and after an otherwise valid
sha256 digest.
* Reuse the manifest cache key for referrers
The referrers handler had its own key function, but it did exactly what
containerManifestCacheKey already does with four strings. Referrers rows
sit in their own oci-referrers ecosystem anyway, so they can't collide
with manifests.
* Drop the separate struct for cached referrers
A cached referrers response carries the same fields as a cached tag
list, and the freshness check was a straight copy too. So the referrers
code just uses cachedContainerTags and containerTagsFresh now. Loading,
storing and writing stay on their own, since the ecosystem and the Link
handling are different.
* Route referrers requests by ns like the other OCI endpoints
Now that containerd ns support is on main, referrers requests have to
go through registryForRequest too, otherwise a _default mirror would
look every subject up on Docker Hub. ns only picks the registry, so it
is dropped from the forwarded query and the cache key, and the next-page
Link keeps it, the same way tag lists do. This also follows the new
rewriteContainerTagsLink signature, which takes the namespace.
* Without a cached index, referrers now pass upstream errors through and answer 502 when the upstream can't be reached or sends garbage, so clients no longer read an outage as a registry without the referrers API. Stale cache is still served when there is one.
* The tag list and manifest routes now only match paths that end in their endpoint, so a repository with tags/list or manifests in its name can still reach the referrers endpoint.
* Add age-based retention for cached artifacts
Cached artifacts stayed forever unless storage.max_size pushed them out.
storage.retention adds a default, per-ecosystem and per-package duration
after which an artifact nobody downloaded gets evicted.
Ecosystems opt in one at a time: a handler registers with
retention.Default, and until it does, configuring that ecosystem (or one
of its packages) fails validation, and the default doesn't touch it. This
commit registers none, so nothing changes yet. Startup logs which
ecosystems the default covers and which it doesn't.
An artifact counts as expired when both its fetch time and its last
access are older than the cutoff. Checking only the last access would
evict a refetched artifact again right away, since clearing a record
keeps its old access time and a cache miss doesn't record a hit.
The sweep scans artifacts in id windows, so no single query holds the
SQLite connection for long, and checks each row against its own rule.
Expired records are cleared and their files go through pending_deletes
instead of being deleted directly, because a buffered hit can make an
artifact that's being served right now look expired. Buffered hits are
flushed before each sweep, and one sweep clears at most 10,000
artifacts.
Since a big first sweep can queue a lot, reclaim now keeps working
through due entries for up to 30 seconds per tick instead of stopping
after 100, and pending_deletes gets an index on queued_at.
Also:
- proxy_artifacts_evicted_total{reason, ecosystem} counts LRU and
retention evictions.
- gradle.build_cache.max_age accepts "7d", as its comment always said.
Refs #306
* Document storage retention
Describes the new storage.retention block, the order rules apply in,
that the default only covers ecosystems that support retention yet, and
what to expect from the first sweep over an old cache. Also lists the
new eviction counter and notes that gradle max_age takes days.
Refs #306
* Fix retention timing on Postgres and tighten the sweep
On Postgres, fetched_at and last_accessed_at are TIMESTAMP columns
without a zone. They hold the local time the proxy wrote, but lib/pq
reads them back labelled UTC. East of UTC that makes every artifact look
hours younger than it is, so the sweep evicted late: two hours in
Berlin, nine in Tokyo. The candidates' times are now read back as local
time, and the database tests run on both SQLite and Postgres, with one
that sets the local zone to Asia/Tokyo.
A few smaller things:
- When a delete fails, reclaim stops for the rest of the minute instead
of trying every queued path. Otherwise a storage outage turns into
thirty seconds of failed deletes and warnings every minute.
- If the buffered hits can't be written, the sweep skips that round.
Running on stale access times could clear something that is being
downloaded right now.
- DefaultCanonical only accepts a package key when rebuilding it gives
the same PURL. For alpine and deb it would otherwise produce keys
that never match.
- sweep_interval has to be at least a minute.
- New tests cover the eviction counter and an artifact that gets a hit
between the scan and the clear.
- The docs no longer say nothing changes without retention, and they
point out that the example's ecosystem and package rules only pass
validation once those ecosystems support retention.
Refs #306
* Reject qualifiers and subpaths in retention package keys
A rule names a whole package. Qualifiers like ?type=pom were dropped on
the way to the stored PURL and the rule quietly covered every file of the
package. Now such a key fails validation.
Refs #306
* Say when a package key doesn't match how packages are stored
A package rule for an ecosystem that supports retention, written in a
form its packages aren't stored under (pkg:npm/babel/core without the
@), failed with "retention is not supported for npm packages". Once the
ecosystem is enabled that is no longer true. Such a key now fails with
"does not match how npm packages are stored" and points to the package
rule forms in the docs. Keys for ecosystems without retention support
keep the old message.
Refs #306
* Note that re-running the mirror keeps packages cached
A mirror run over a package that is still cached goes through the normal
cache lookup and records a download, so the package's age starts over.
The docs only said that mirrored packages age from the time they were
mirrored.
Refs #306
* Say when an ecosystem takes no package rules
Some ecosystems will support retention without package rules, because nothing in their stored records names a package the way a PURL would (conan, helm). A package key for one of them used to fail with "does not match how conan packages are stored; see the package rule forms", which sends people looking for a form that doesn't exist. It now says package rules aren't supported there and points to the ecosystem rule.
Refs #306
* Use helm as the example of an ecosystem without package rules
Conan gets package rules once its handler stores archives under the recipe name, so the config test that shows the refusal now uses helm, whose cache records name the repository, not the chart.
Refs #306
* Say that each cached file ages on its own
The retention docs said each artifact, each version of a package, ages on its own. A version can have several cached files (a jar and its pom, several wheels, Conan sources and binaries), and each of them ages separately.
Refs #306
* Clear an expired artifact and queue its file in one transaction
The sweep cleared the record and queued the file for deletion as two separate writes. If queueing failed, or the process stopped in between, the file stayed in storage with no record pointing at it and no queue entry, and no later sweep would find it again. Both writes now share a transaction, so a failed queue rolls the clear back and the next sweep retries.
Refs #306
* Show artifact evictions on the analytics page
proxy_artifacts_evicted_total had no place on /ui/analytics. The Runtime card now lists evicted artifacts by reason and ecosystem, without the failure tint, and the coverage test records an eviction so it doesn't depend on test order.
Refs #306
* Store artifact times in UTC
fetched_at and last_accessed_at are now written in UTC, and the retention cutoffs are compared in UTC, the same convention #434 uses for the metadata cache. lib/pq sends a time with its offset and a Postgres TIMESTAMP keeps only the wall clock, so UTC on write is what reads back correctly. That replaces reinterpreting the times as local on read.
Rows written before this still hold local time. East of UTC they look younger by the offset until they are rewritten, so retention evicts them a few hours late once; west of UTC that much early.
Refs #306
---------
Co-authored-by: t <t@t>
* GitLab Composer archives are now cached per commit and fetched for the commit the lock file pins, and branch names with a slash are escaped in dist URLs so the download route matches them.
* Composer branch archives that an older proxy cached from another commit's URL are now discarded and fetched again, and a refetched artifact records the URL it actually came from.
* When a GitLab commit is fetched under its own name, the archive older proxies cached under the shared archive.zip name is dropped. A test now moves the branch from A to B and checks that A keeps coming from the cache.
* Reject package requests whose decoded path contains '?' or '#', in one middleware in front of every ecosystem mount. Handlers build the upstream URL from the decoded path, so busybox.apk%3F/foo-9.9-r0.apk fetched busybox but cached it as foo; PyPI could be poisoned the same way, and the other ecosystems stored files under mangled names past the denylist and cooldown.
* Let Composer archive downloads through the '?'/'#' guard: they resolve the upstream URL from metadata, and with #433 a branch like dev-fix/#42 is requested as dev-fix%2F%2342. Adversarial review finding F1.
* Add Composer package routes for private repositories
- Add upstream.composer_routes mapping package name patterns to repositories
- Serve matching packages only from their route, with no fallback to the default repository
- Qualify metadata and artifact cache keys with the route's upstream
- Give routed versions an empty notification-url so installs are not reported to notify-batch
- Document routes in configuration docs, example config and README
* Add npm package routes for private registries
- Add upstream.npm_routes mapping package name patterns to registries
- Serve matching packages only from their route, with no fallback to upstream.npm
- Resolve and validate tarball URLs against the route's registry
- Qualify metadata and artifact cache keys with the route's upstream
- Document npm routes and the audit endpoint limitation
* Share routed artifact filename qualification between handlers
* Match Composer routes without the ~dev suffix
Packagist serves a package's dev versions from {name}~dev.json, and the
handler matched routes against the name with ~dev still attached. An exact
route for example/library therefore missed /p2/example/library~dev.json, and
the private package's dev metadata came from the default repository.
- Match routes against the package name without ~dev, keeping it in the
upstream URL and the cache key so the two files stay apart
- Add a regression test with an exact route
* Keep routed npm publish times out of the shared versions row
The download cooldown check stores publish times under the plain version
PURL, which does not say which registry a version came from. A time read
from the default registry was trusted for a private version of the same
name, so a private release published an hour ago passed a seven-day
cooldown when the public one was a month old.
- Skip the stored row for routed packages, both reading and writing, and
take the publish time from the route's metadata, which is cached under
the route
- This also stops a private time from holding back the public version, and
a route's previous upstream from lending its times to the new one
- Cover adding a route after a public download, the reverse order, and
moving a route to another upstream
* Fetch the pinned commit for Composer branch downloads
The download handler resolved a version's dist URL from the current metadata and cached the archive under whatever file name the client asked for. A lock file pinning an older commit of a branch like dev-master got the branch head, stored and served as that older commit.
Downloads now fetch the commit the file name stands for when the dist URL ends in a commit hash (GitHub zipballs, Bitbucket archives), and return 404 for other names the metadata no longer lists.
* Accept bare commit names from older Composer lock files
Proxy versions before the .zip suffix handed out GitHub zipball URLs ending in the bare commit hash. Lock files written back then still ask for those names, and the new file name check turned them into 404s.
* Rebuild Composer commit URLs only for branch versions
A tagged release lists one commit and lock files ask for that one. Rebuilding the URL for any requested commit let a client store another commit, even one from a fork, as the release, and the browse view could then show it.
* Note that the Composer cache key leaves out the query
* Conan 2 downloads recipe and package files from /v2/conans/.../revisions/{rrev}/files/{file} and .../packages/{id}/revisions/{prev}/files/{file}, but the handler only cached the invalid /v1/files and /v2/files paths, so archives were never stored. This adds the real routes and drops the old ones; manifests and conaninfo.txt stay in the metadata cache.
* Conan 2.25 can also compress archives with xz or zstd (conan_package.txz, conan_package.tzst and so on), so cache those too.
* Store Conan archives under plain name and version, like pkg:conan/zlib@1.3.1, with the revisions and any user/channel in the filename. The old name/version@user/channel with one version per revision never matched a denylist entry or what scanners look up.
* Route containerd ns mirror requests to configured registries
containerd's hosts.toml mirrors append ?ns=<registry-host> to every
request. The container handler ignored it, so a single _default mirror
entry could not serve more than one registry.
ns is a closed-world lookup key and is never dialed. Docker Hub aliases
and the host of upstream.oci_default select the default route, hosts of
upstream.oci entries select their named upstream. Unknown hosts return
NAME_UNKNOWN so containerd falls back to its next host. Registry URLs
with a path are not indexed, because ns names the registry at the root
of a host. Host collisions log a warning and resolve deterministically.
With ns the path is the verbatim upstream repository, the reserved
upstream/ prefix is rejected, and repository prefix routes such as
Homebrew's do not apply. Cache names match the unprefixed and
upstream/{name}/ routes, so all routes share cached blobs, manifests and
tag lists. ns is no longer forwarded on tag-list requests, and the
pagination Link is rewritten per request instead of at store time so
clients on different routes get links for their own route.
Refs #303
* Document containerd ns mirroring
Refs #303
* Version the tag-list cache identity for the raw-link format
Tag-list rows now store the upstream Link verbatim and rewrite it per
request. Rows written by earlier versions hold a Link already rewritten
for one route. Under the unchanged key an older binary running next to
this one (rolling update on shared Postgres, or a rollback) would serve
the raw Link unrewritten, sending clients to the wrong route for the
next page. A format marker in the key keeps the two apart at the cost of
one cache miss per tag list after the upgrade.
Adversarial review finding F1: tag-list cache rows hold raw upstream
Links under unchanged keys, so an older binary serves them unrewritten.
* Qualify the README claim about ns mirror coverage
Registries whose URL has a path and hosts shared by two entries are not
reachable through ns; point to the configuration guide for both rules.
Adversarial review finding F3: README says ns covers every upstream.oci
registry.
* Redact registry URLs in ns index warnings
Configured registry URLs may carry userinfo credentials, which the new
startup warnings wrote to the log verbatim. Log them with the password
masked, and say precisely what a path-prefixed default registry means:
its host is not indexed, while the Docker Hub aliases still select it.
Adversarial review finding F2: startup warnings print registry URLs
unredacted and the default-registry message is misleading.
* Keep non-default ports apart in the ns index
Dropping 80 and 443 regardless of scheme turned https://host:80 and
https://host into the same key, so a request for one of them could land
on the other registry. Only the default port of the URL's scheme is
optional now. A configured host is indexed with and without that port,
because image references spell it either way; any other port has to
match exactly, and the ns value is used as containerd sends it, apart
from case and IPv6 brackets.
Since a URL now yields two keys that can belong to different routes,
the collision warning lists the owner per key.
The handler test runs two registries on one host that differ only in
the port. Tests can't bind 80 or 443, so a dialer maps those addresses
to the fake servers.
* Allow upstream/{name}/ together with ns when they agree
containerd sends ns on every request to a mirror host, override_path or
not. Per-registry mirrors pointing at /v2/upstream/{name} therefore
arrive as upstream/{name}/...?ns=<registry>, and refusing every prefixed
name under ns broke them, including the only way to mirror an upstream
whose URL has a path.
Such requests are now handled like the prefix route without ns, as long
as ns names that upstream's own host under the same port rules as the
index. Any other ns on a prefixed name is still NAME_UNKNOWN, so ns
can't be used to create cache entries under another registry's name.
* README: containerd config_path and how to check it
A _default/hosts.toml does nothing while CRI has no hosts directory
configured, so add the config.toml snippet for containerd 2.x and 1.x.
To check the setup, look at containerd config dump and pull with
crictl, then find the request in the proxy log; ctr --hosts-dir reads
the directory on its own and would succeed even without config_path.
Also mention that existing per-registry override_path entries keep
working and that k3s generates the directory itself.
* Let the prefix route take ns for aliases and mirrored registries
With upstream.oci.hub pointing at registry-1.docker.io and a docker.io
hosts.toml mirror on /v2/upstream/hub, containerd sends ns=docker.io,
and the prefix check only accepted the upstream's own host. The same
happens whenever the upstream is a mirror of the registry the nodes
pull from, say an Artifactory remote for ghcr.io: ns carries ghcr.io,
the upstream host is something else, and the pull got a 404.
The prefix already decides where the content comes from and which
cache entries are used, so ns can't redirect anything there. The check
now only refuses an ns that contradicts the prefix, meaning a host the
proxy knows that belongs to another route. The Docker Hub aliases count
as one host, and a host the proxy doesn't know at all is accepted,
since a client can only arrive at the prefix through its own mirror
entry.
* Take docker.io on every prefix route
A Docker Hub mirror configured as a named upstream (mirror.gcr.io, an
Artifactory remote) with a docker.io hosts.toml pointing at
/v2/upstream/hub got a 404 again: docker.io is always in the index for
the default route, so the check saw a known host of another route. That
setup worked before this branch. Docker Hub repository names have
exactly two path components, so docker.io/upstream/... can never be a
real image, and accepting the aliases on every prefix route shadows
nothing.
While here: the collision warning claimed ns requests go to the other
route, which is only true without the prefix, and the README said pulls
for unknown registries always get a 404, which isn't the case for paths
under upstream/{name}/.
* Only take a foreign ns on a prefix route when it is configured
A containerd _default mirror sends the same request for a pull of
unconfigured.example/upstream/mirror/owner/app as a ghcr.io hosts.toml
with override_path does for a mirror upstream. Accepting any host the
proxy didn't know therefore let such a pull be answered with owner/app
from the mirror upstream, a different image, instead of a 404 that lets
containerd go to the registry the image actually names.
The prefix route now accepts ns only for the upstream's own host, for
Docker Hub, and for hosts listed in the new upstream.oci_mirrors setting.
That setting is how an Artifactory remote of ghcr.io keeps working:
upstream:
oci_mirrors:
ghcr: ["ghcr.io"]
The listed hosts also select the upstream for unprefixed ns requests,
which makes a mirror with a path in its URL reachable through the
_default entry as well. Hosts from configured URLs still win over
mirror entries; the proxy warns about such overlaps at startup.
* Fix the docs sentence on which ns a prefix route takes
configuration.md still said a prefix request with ns is only refused
when ns names another route's host. Since the last commit it is the
other way round: only the upstream's own host, its oci_mirrors entries
and Docker Hub get through.
* Note what a host accepted on a prefix route gives up
Listing ghcr.io for an upstream (or the upstream's own host) still has
the ambiguity from the wildcard case: ghcr.io/upstream/ghcr/owner/app
lands on that upstream. That is the price of an explicit entry and
fine in practice, but it should be written down.
* Test that oci_mirrors entries stay with their own upstream
Two gaps in the coverage of the last change: nothing failed if a host
listed for one upstream was also accepted on another upstream's prefix,
and nothing checked that the server actually hands upstream.oci_mirrors
to the container handler. The handler test now has a second upstream
that must refuse ghcr.io, and a small server test pulls through a
mirror entry on both the prefix and the unprefixed route.
* Log when a prefix route refuses an ns
Setups with a ghcr.io hosts.toml pointing at a mirror upstream worked
before this branch and now need an oci_mirrors entry. Without one,
containerd gets a 404, quietly pulls from ghcr.io directly, and nobody
notices the proxy is being skipped. The refusal is now logged as a
warning that names the upstream and the ns and points at
upstream.oci_mirrors.
* Drop a check in validateOCIMirrors that could never fire
user@ghcr.io parses with ghcr.io as its host, so the parsed.Host != host
comparison already rejects it and the extra parsed.User test was dead.
A test case for a host with credentials keeps that path covered.
* Reject an oci_mirrors host that is listed for two upstreams
With ghcr.io under both art and nexus, both prefix routes took ns=ghcr.io
while unprefixed requests went to whichever name sorted first, with
nothing but a startup warning. Config validation now refuses the second
entry, comparing hosts without case and without the default :443.
* Log each refused ns once per upstream and refuse overlong values
The refusal warning fired on every request, and a node with a missing
oci_mirrors entry asks for the manifest and each layer, so the same line
piled up. Each upstream/ns pair is now logged once, for up to 1024
pairs; after that every refusal is logged again.
ns comes straight from the client and can be as long as the request
line allows, so remembering it as is would let anyone pin a lot of
memory with a handful of requests. No registry host is longer than 259
characters (a 253-character name plus a port), so anything longer is
refused right away, before it is logged or remembered.
* Compare oci_mirrors hosts the way the handler does
The duplicate check lowercased the host and trimmed :443, while the
handler splits host and port, brackets IPv6 literals and drops an empty
port. So ghcr.io and ghcr.io:, or [::1] and [::1]:, passed as different
hosts in two upstreams and then collided in the handler. The check now
builds the same key the handler uses.
* Rewrap the oci_mirrors paragraphs in the configuration docs
Two sentences had been dropped into the middle of wrapped lines, and the
prefix-route paragraph named Docker Hub twice in a row. Same content,
shorter lines, one mention.
* Stop logging ns refusals once the set is full
Past 1024 different upstream/ns pairs every refusal was logged again,
so anyone could fill the set with made-up hosts and then flood the log
at warning level. When the set is full the proxy now says so once and
keeps quiet after that. A real misconfiguration shows up long before
that, with the first pull.
* Test the limit on remembered ns refusals
Nothing covered what happens once 1024 pairs are remembered. The new
test fills the set, then sends a known pair and two new ones: the proxy
must log the limit notice exactly once, log none of the refusals and
keep the set at its size.
- Walk version lists in place instead of decoding the document into
map[string]any and expanding the minified format
- Keep minified metadata minified: each written version carries what differs
from the client's expanded state, so changes made by versions dropped by
cooldown fold into the next kept one; every version gets its own dist
- Copy everything else byte for byte; only dist values are written fresh
- Resolve download URLs by walking the list in place, tracking inherited
fields, instead of decoding and expanding the whole document
- Add forEachJSONElement, rewriteJSONMembers and jsonStringIs to the scanner
The Maven handler cached only .jar, .war, .ear, .pom, .aar, .klib and
.module files, so distributions published as archives, such as
apache-maven-{version}-bin.tar.gz, were proxied uncached on every request.
.zip, .tar.gz, .tgz, .tar.bz2 and .tar.xz files now go through the artifact
cache like other artifacts.
Co-authored-by: Giles Westwood <201443+gilesw@users.noreply.github.com>
Rewriting a packument decoded the whole document into map[string]any and
encoded it again. For a full packument of tens of megabytes that costs many
times the document size per request, and with cooldown enabled (which skips
the rewrite cache and forces full packuments) it happens on every request.
- Rewrite npm metadata in place: walk the raw JSON for member offsets, decode
only the version names, time and dist-tags for the existing filters, and
assemble the response from slices of the original bytes with only tarball
URLs (and filtered time/dist-tags) written fresh. Untouched values now reach
clients byte for byte instead of round-tripping through encoding/json.
- Look up versions[v].dist.tarball and time[v] directly on the tarball
download path instead of decoding every version.
- Record the stored metadata's SHA-256 in metadata_cache.content_digest and
key the rewrite cache by it, so a repeat npm or Composer request inside the
TTL is served from the rewrite cache without reading or hashing the stored
document.
On a synthetic 25 MB packument with 5000 versions, rewriteMetadata drops from
~150-174 MB / 706k allocations per call to ~31 MB / 90k, and runs in about
two thirds of the time. npmVersionTarball drops from ~1 MB / 10k allocations
to one allocation.
* experimental filtering pills
* align data in the dashboard table more nicely (#318)
* Bump zizmorcore/zizmor-action from 0.6.2 to 0.6.3 (#333)
Bumps [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) from 0.6.2 to 0.6.3.
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](3dc1ecc9bc...70fb788f84)
---
updated-dependencies:
- dependency-name: zizmorcore/zizmor-action
dependency-version: 0.6.3
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump docker/setup-qemu-action from 4.2.0 to 4.3.0 (#331)
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 4.2.0 to 4.3.0.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](96fe6ef7f3...1f40c72289)
---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
dependency-version: 4.3.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump azure/setup-helm from 4.3.1 to 5.0.1 (#332)
Bumps [azure/setup-helm](https://github.com/azure/setup-helm) from 4.3.1 to 5.0.1.
- [Release notes](https://github.com/azure/setup-helm/releases)
- [Changelog](https://github.com/Azure/setup-helm/blob/main/CHANGELOG.md)
- [Commits](1a275c3b69...9bc31f4ebc)
---
updated-dependencies:
- dependency-name: azure/setup-helm
dependency-version: 5.0.1
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump modernc.org/sqlite from 1.57.0 to 1.58.0 (#337)
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.57.0 to 1.58.0.
- [Changelog](https://gitlab.com/cznic/sqlite/blob/master/CHANGELOG.md)
- [Commits](https://gitlab.com/cznic/sqlite/compare/v1.57.0...v1.58.0)
---
updated-dependencies:
- dependency-name: modernc.org/sqlite
dependency-version: 1.58.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump github.com/aws/aws-sdk-go-v2/config from 1.32.40 to 1.33.2 (#336)
Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.32.40 to 1.33.2.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.32.40...config/v1.33.2)
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
dependency-version: 1.33.2
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump github.com/prometheus/client_model from 0.6.2 to 0.6.3 (#330)
Bumps [github.com/prometheus/client_model](https://github.com/prometheus/client_model) from 0.6.2 to 0.6.3.
- [Release notes](https://github.com/prometheus/client_model/releases)
- [Commits](https://github.com/prometheus/client_model/compare/v0.6.2...v0.6.3)
---
updated-dependencies:
- dependency-name: github.com/prometheus/client_model
dependency-version: 0.6.3
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump golang.org/x/sync from 0.22.0 to 0.23.0 (#335)
Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.22.0 to 0.23.0.
- [Commits](https://github.com/golang/sync/compare/v0.22.0...v0.23.0)
---
updated-dependencies:
- dependency-name: golang.org/x/sync
dependency-version: 0.23.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump github.com/aws/aws-sdk-go-v2/service/ecr from 1.61.0 to 1.64.0 (#334)
Bumps [github.com/aws/aws-sdk-go-v2/service/ecr](https://github.com/aws/aws-sdk-go-v2) from 1.61.0 to 1.64.0.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.61.0...service/s3/v1.64.0)
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/ecr
dependency-version: 1.63.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(handler): let the ProxyCached path request a per-call Accept-Encoding (#324)
* fix(handler): fetch conda repodata gzip-compressed on both hops
#304 made the ProxyCached path request Accept-Encoding: identity so the
metadata cache stores upstream bytes verbatim. That is required for the
signed / hash-pinned index ecosystems, but conda's repodata.json is large
plain JSON: linux-64 repodata.json is ~441 MB uncompressed (over the
metadata_max_size cap, so it 502s today) versus ~34 MB gzip.
Replace the ProxyCached path's verbatim bool with an explicit
acceptEncoding string ('' = leave unset / transparent, 'identity', or
'gzip'), reusing #304's existing store-and-replay of Content-Encoding
unchanged. ProxyCached keeps its exported signature and continues to send
identity, so the nine other ecosystems and helm/maven are untouched; only
conda's repodata.json / current_repodata.json now request gzip. Setting
Accept-Encoding explicitly disables Go's transparent decompression, so the
compressed bytes and the Content-Encoding: gzip header are cached and
replayed exactly as identity bytes are. conda, mamba and pixi solicit and
decode gzip on .json URLs; repodata.json.bz2 stays identity.
Fixes#305
* fix(handler): pass metadata content-encoding with the body it describes
The adversarial review of the conda gzip route found a reachable
regression: writeMetadataCachedResponse took Content-Encoding from a
fresh cache-row read while cacheMetadataBlob skips the row write when
Storage.Store fails. Under identity that was benign (the body was plain
anyway), but on the new gzip route a disk-full or object-store outage
served raw gzip bytes as Content-Type: application/json with no
Content-Encoding and HTTP 200 -- conda, mamba and pixi fail to parse
them, with no HTTP signal and only a Warn log, on every request until a
cache write succeeds.
fetchOrCacheMetadata now returns the encoding of the body it hands back
(the upstream value on a fetch, the stored row's value on a TTL hit or
stale fallback) and proxyCachedWithEncoding passes it to
writeMetadataCachedResponse, so the header always describes the bytes
actually written. cachedMeta drops its now-unused content_encoding
field. helm and maven pass "" -- both fetch transparently, so their
stored encoding was always empty and behaviour is unchanged.
Also fixes a vacuous assertion in the new conda test: the upstream
request counter incremented behind the availability gate, so the
cached-replay block could never observe a refetch.
* fix(handler): pin the stale-fallback content-encoding and drop a dead guard
Follow-ups from the adversarial review of the #305 branch, limited to
code this branch introduced:
- proxyMetadataStream is only ever reached with an explicit
Accept-Encoding (ProxyCached passes identity, conda passes gzip or
identity), so the guard around the header set was unreachable; replace
it with the plain one-token substitution of the former literal, which
is the smallest change from main.
- The stale-fallback return of fetchOrCacheMetadata (encoding taken from
the cache row) was the one #305 return site no test pinned: replacing
it with an empty encoding survived the whole suite. Add a conda test
that expires the entry, fails the upstream, and asserts the stored
gzip blob is served with Content-Encoding: gzip.
Not changed, by scope: cacheMetadataBlob still discards the
UpsertMetadataCache error (pre-existing on main). If Storage.Store
succeeds and the row write fails, a later stale fallback or TTL hit can
serve the gzip blob with the row's stale encoding; that needs a DB write
failure plus a second event and is tracked separately.
* fix(handler): restore the pre-existing cachedMeta content-encoding field
The third adversarial review classified deleting cachedMeta.contentEncoding
and its lookupCachedMeta populate as elective: neither line was created by
this branch nor forced by the fix (writeMetadataCachedResponse now reads
the encoding from its parameter and ignores the row value). Under the rule
that pre-existing code this branch did not have to touch stays untouched,
restore both as they are on main. No behaviour change.
Residuals the review documented, unchanged by scope (both share one root
cause: the encoding lives in the cache row and the bytes in the blob, and
neither is written or read atomically):
- cacheMetadataBlob discards the UpsertMetadataCache error, so after a
successful gzip Store and a failed row write a later stale fallback or
TTL hit can serve the gzip blob with the row's stale encoding.
- During the one-time identity->gzip rollout, a request that read a
pre-branch identity row, lost the upstream race to a request that stored
the gzip blob, and then failed upstream serves the gzip bytes with no
Content-Encoding for that one response; later requests self-heal.
- helm and maven now pass an empty encoding; on main a spec-violating
upstream that answered a transparent gzip request with an encoding Go
does not decode (e.g. br) would have had that header replayed from the
row. Degenerate; documented rather than changed.
* fix(handler): keep conda's proxyCached and .bz2 route as on main
Threading acceptEncoding through CondaHandler.proxyCached changed the
form of two pieces of original code the fix did not need to touch: the
repodata.json.bz2 route (method value rewritten as a closure) and
proxyCached itself (new parameter, new call). Restore both exactly as on
main; ProxyCached still sends identity, so the .bz2 route is unchanged in
behaviour. handleRepodata's non-cooldown branch now derives the cache key
inline and calls proxyCachedWithEncoding with gzip directly, so the only
original conda.go line that changes is that one call.
* fix(handler): keep writeMetadataCachedResponse and its callers as on main
Adding a contentEncoding parameter to writeMetadataCachedResponse changed
a signature that predates #304 and dragged its two pre-#304 callers
(helm.go, maven.go) into the diff, even though #304 only ever added the
cm.contentEncoding block inside the function body.
Restore writeMetadataCachedResponse's doc and signature exactly as on
main and make it a delegate that passes an empty encoding to a new
unexported writeMetadataCachedResponseWithEncoding, which carries the
original body with #304's block reading the parameter instead of the
cache row. proxyCachedWithEncoding calls the sibling with the encoding
returned alongside the body. helm.go and maven.go drop out of the diff;
their behaviour is unchanged (both fetch transparently, so their stored
encoding was always empty). Same split pattern as ProxyCached ->
proxyCachedWithEncoding.
* fix(handler): move the conda gzip change to its own branch
The conda call site in handleRepodata predates #304 and #304 never
touched it, so under the rule that this PR only corrects code and
behaviour #304 introduced it does not belong here. Restore conda.go and
conda_test.go as on main; the conda change continues on a stacked branch
against its own issue.
Replace the conda-route tests with tests that exercise
proxyCachedWithEncoding directly, so this PR still pins its own plumbing:
gzip is requested and the compressed bytes plus Content-Encoding are
cached and replayed (cached and streaming paths), the header survives a
metadata cache write failure, and the stale fallback keeps the stored
encoding.
* fix(homebrew): fetch the JSON API gzip-compressed on both hops
Homebrew (#254) routes every API path through ProxyCached and so, since
#304, fetches formula.jws.json (~33 MB plain, ~5 MB gzip) uncompressed on
every refresh -- the case that motivated #305.
Request gzip for the JSON API via proxyCachedWithEncoding: brew fetches
every API download with curl --compressed and decodes Content-Encoding
itself, so the compressed bytes and header are cached and served as-is
and both hops stay compressed. The analytics endpoints are the one brew
consumer fetched without --compressed; they stay on identity.
* fix(handler): leave Accept-Encoding unset in proxyMetadataStream for an empty value
fetchUpstreamMetadata treats an empty acceptEncoding as 'do not set the
header'; proxyMetadataStream set it unconditionally, which would send an
empty Accept-Encoding line if a caller ever passed . Guard it the same
way so both paths agree. No caller passes today.
* fix(handler): keep the metadata row and blob from describing different bytes
Two ways the cache row could stop describing the stored blob once a
caller requests gzip, both raised by the review of #324:
- cacheMetadataBlob stored the blob and then discarded the
UpsertMetadataCache error. After a successful gzip store and a failed
row write, a later TTL hit or stale fallback served the gzip blob with
the previous row's encoding. On a row-write failure, log it and delete
the blob just written, so the next request refetches instead.
- fetchOrCacheMetadata read the row once up front and reused it for the
stale fallback. A request that read an identity row, lost the upstream
race to a request that stored the gzip blob, and then failed upstream
labelled the new blob with the old row. Re-read the row before falling
back so the encoding matches the blob as it is now.
Both only become harmful with an encoding change, which this branch
introduces; the pre-existing validator-from-row read is tracked
separately.
* Drop unused cachedMeta.contentEncoding and fix stale doc reference
The field was added by #304 and its only reader is replaced in this
branch by the encoding parameter passed alongside the body. The
proxyCachedWithEncoding comment named conda repodata, which was moved
out of this branch in 5991d95; Homebrew is the caller that ships here.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
* Fix duplicate fetches and 502s on concurrent cache misses (#329)
* Return the stored artifact from storeArtifact, not a reader
storeArtifact returned a CacheResult holding an open file handle. A
handle has one read position, so it can only ever serve a single caller,
which is what blocks sharing one fetch between concurrent requests.
Return the artifact and its storage path instead, and let each caller
open its own reader through openStoredArtifact. Threading that type
through fetchAndCache, fetchAndCacheFromURL and their error paths is
mechanical; behaviour is unchanged.
* Coalesce concurrent cache misses
A cache miss went from checkCache straight to an upstream fetch with
nothing tracking in-flight work, so N concurrent requests for one
uncached artifact produced N upstream fetches and N stores to the same
key. That is the CI shape: parallel jobs installing overlapping
dependencies against a cold cache. The duplicate stores also fail
requests, racing fileblob's per-key ".attrs" sidecar into a partial read
served as a 502. Over 12 runs of 8 simultaneous requests for one
uncached tarball, against bb2205a: before, 8 fetches per run and 12 of
96 responses were 502; after, 1 fetch per run and none failed.
Route both miss paths through a shared in-flight map keyed on the
artifact, including the download URL and upstream-declared hash so
callers expecting different bytes never share a fetch.
singleflight does not fit: Do gives waiters no way to leave, while
DoChan lets the caller running the fetch abandon it, breaking
storeArtifact's scan-on-disconnect contract. Deciding roles under a
mutex gives both behaviours. The fetch runs on the first caller's
context and is seen through; waiters leave when their own clients do.
This removes the sidecar trigger on this path. The race is in fileblob
and three writers bypass this path entirely, so it is fixed separately.
Fewer failures now reach the circuit breaker, so it trips later.
Sixteen concurrent callers against real file:// storage fail 10 of 10
runs on main and pass 10 of 10 here. Other tests pin key discrimination,
failure propagation, resolver-path coalescing, per-caller readers,
waiter cancellation, key release and panic safety. allocs/op is
unchanged. mockStorage gains a mutex so concurrent tests can use it.
* Normalize digest case in the coalescing key
artifactHashMatches compares digests with strings.EqualFold, but the
coalescing key used the hash verbatim. The same digest in two casings
produced two keys, so two callers for one artifact each ran their own
upstream fetch and store, which is what the coalescing is meant to
prevent.
* Make the panic coalescing test deterministic
The test timed the second caller's arrival with a sleep, so which caller
became the leader was left to the scheduler. When it lost that race the
second caller ran the fetch itself, and its panic was not recovered, so
the test binary died instead of the test failing.
Whether a caller has reached the wait is not observable from outside:
it runs a cache lookup against the database first, so releasing the
leader on a timer races that query. Drive coalesceFetch directly and
hold the shared entry instead, which removes the timing entirely. The
panicking fetcher is no longer needed.
* Recheck the cache before running a shared fetch
A caller checks the cache before it reaches coalesceFetch, so a fetch
that commits in that gap is invisible to it. Arriving after the sharing
entry is gone, it became a new leader and fetched, stored and scanned an
artifact the cache already held.
The leader now rechecks the committed record first. It serves that
record only if its bytes still open, because a record can outlive them,
and refetching is the recovery the cache lookup already makes for that
case. Waiters are unaffected: the record fills the same shared value a
fetch would, and each caller opens its own reader from it.
The recheck is the leader's alone. A waiter has a fetch in flight to
wait on, and rechecking would race it for no gain.
* Lock the mock fetcher's bookkeeping
Coalescing tests call the handler from many goroutines. The key keeps
the fetch itself serialized, but the mock should not lean on that: it
now locks the fields it records, so any concurrency the handler applies
is safe under the race detector.
* Wait for the leader's fetch instead of sleeping
The canceled-waiter test slept 200ms and assumed the leader had taken
the key by then. On a slow scheduler the canceled call could become the
leader and the test would no longer cover waiter cancellation. The
fetcher now signals when its first fetch begins, which happens only once
the key is held.
* Bump google.golang.org/grpc from 1.83.1 to 1.83.2 (#346)
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.83.1 to 1.83.2.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.83.1...v1.83.2)
---
updated-dependencies:
- dependency-name: google.golang.org/grpc
dependency-version: 1.83.2
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Stop writing fileblob's .attrs sidecar (#328)
* Stop writing fileblob's .attrs sidecar
fileblob stores blob metadata in an ".attrs" file per object and rewrites
it with os.Create, truncating in place outside the atomic rename that
protects the blob. A read overlapping a write decodes a partial file and
fails with "opening reader: EOF", served as a 502. One writer against
four readers on a single key failed 408 of 2000 reads. cacheMetadataBlob
is most exposed to it, rewriting a key on every refresh while readers
are served from it.
Nothing in the proxy reads what the sidecar holds. gocloud.dev/blob is
imported only by internal/storage, Store sets no ContentType, and
Attributes is used only for Size, which comes from os.Stat. A missing
sidecar already defaults cleanly, so "metadata=skip" removes the hazard
rather than locking around it, and saves a write per store.
* Clear .attrs sidecars left by earlier versions
metadata=skip stops fileblob rewriting sidecars but does not delete ones
already on disk, so a sidecar left partial by an interrupted write now
fails every read of its key for good. Before, a later store repaired it
by rewriting.
Store therefore removes the sidecar for the key it writes. Removal is
atomic where the rewrite was not, so a concurrent reader gets the whole
old file or nothing. Delete already removes sidecars, so the two paths
drain a cache between them.
Deriving that path is necessary because fileblob's key escaping is
unexported. It is the identity for a plain key and parts from one only
for keys that are not valid local paths, which is what filepath.Localize
rejects. That also keeps the removal inside the cache directory: without
it a key holding ".." resolves outside.
The clearing test runs one key per storage path the proxy builds, seeded
through a bucket that still writes sidecars so the path under test is
fileblob's own.
* Explain why failed sidecar cleanup does not fail the store
Move the removal into clearLegacySidecar and say why its error is
dropped rather than returned. A failed removal leaves exactly the state
this change inherited, while failing the write would turn a cleanup miss
into a failed request.
Windows makes that concrete: Go opens files with FILE_SHARE_READ and
FILE_SHARE_WRITE but not FILE_SHARE_DELETE, so a reader holding the
sidecar open blocks deletion, and that reader is the workload this
change exists to protect. Propagating would fail stores during exactly
the overlap being fixed. The next store of the key retries.
A test pins it, using a non-empty directory at the sidecar path to make
os.Remove fail with something other than not-exist on any platform.
* Fail the concurrency test if its writer stops
The writer returned silently when Store failed, so the test could pass
with no concurrent writes at all. Its error is now reported, and the
test also checks that at least one write completed.
Reporting it showed the writer had been dying on Windows at its first
collision: Go opens files without FILE_SHARE_DELETE, so a reader holding
the file open makes the writer's rename fail with access denied. The
test now skips there, since it cannot contend a writer with readers on
that platform.
* Clear legacy sidecars for keys fileblob escapes
legacySidecarPath declined any key filepath.Localize rejects, which on
Windows is every key with a colon: OCI digests and Debian epochs. Their
sidecars were never cleared there, and a truncated one kept failing
reads, since fileblob still reads a sidecar it finds under metadata=skip.
fileblob hex-escapes such characters on the way to disk. The path is now
derived the same way, so the sidecar is looked for where fileblob wrote
it. Localize still validates the escaped form, which keeps the removal
inside the cache directory.
* Drop stale comment about declining colon keys on Windows
623ff3e made legacySidecarPath escape keys the way fileblob does, so
colon-bearing keys are now cleared on Windows and the OCI and Debian
rows in TestStoreClearsLegacyAttrsSidecar prove it. The comment
described the behaviour before that commit.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
* feat(debian): serve additional APT archives at /debian/{name}/
A Debian release is served by more than one archive. Security updates live on
a different host than the main archive, so a single upstream.debian URL cannot
serve a complete suite set and -security suites are unreachable.
Add upstream.debian_repositories, a name-to-URL map served at /debian/{name}/,
modelled on upstream.apk. The field is additive: upstream.debian keeps serving
/debian/pool/... and /debian/dists/... with unchanged cache identities, so
existing deployments and their warm caches are unaffected, and the scalar
field and PROXY_UPSTREAM_DEBIAN are untouched.
Named repositories scope both caches by name, since the same filename can hold
different bytes in different archives. Metadata keys are hashed over name,
upstream URL, and path, as APKHandler does. Names are validated through
validateNamedUpstreams, and "pool" and "dists" are refused because they would
shadow the main archive's own paths.
An unconfigured first path segment stays a main-archive path rather than
returning 404 as the APK handler does: the main archive is unnamed and serves
paths of its own at the root.
* fix(server): tune the shared upstream transport defaults (#351)
* fix(server): tune the shared upstream transport defaults
server.serve builds the shared client with safehttp.New, which clones
Go's default transport: MaxIdleConnsPerHost stays 0 (an effective limit
of two idle connections per host) and ResponseHeaderTimeout stays 0.
Handing that client to fetch.NewFetcher via fetch.WithHTTPClient
replaces the fetcher's own defaults of 10 idle connections per host and
a 60-second response-header timeout.
Set both on the shared transport, matching the fetcher defaults: a
second burst of concurrent cache misses to one registry now reuses its
connections instead of re-dialling most of them, and an upstream that
accepts a request but stalls before sending headers is cut off after 60
seconds rather than only by the client's overall timeout.
Tests measure connection reuse across two concurrent bursts against a
TLS upstream that counts accepted connections (Go default: at most two
reused; tuned: all eight) and assert that a stall before headers fails
with the response-header timeout.
Fixes#327
* fix(server): reconcile http_timeout docs and tighten the transport tests
The http_timeout documentation and the config comment said "0" disables
the upstream timeout entirely. With a fixed 60-second
ResponseHeaderTimeout on the shared transport that is no longer the
whole story, so both now say that waiting for response headers stays
bounded independently of the setting.
Test cleanup from review: the 50ms settle between bursts was dead time
(the transport returns a connection to the idle pool before the body's
final Read returns, so burst returning already means the pool is
settled); the maxNewInBurst sentinel became explicit min/max bounds per
case; the stall test dropped the client.Timeout override and the
elapsed-time assertion, which was redundant with the error-text check
in any realistic run and whose failure message misattributed the cause,
and its comment now says plainly that the field assertions pin
production while the behavioural half runs at a lowered timeout.
* test(server): hold each burst at the upstream instead of sleeping
The reuse test kept a burst in flight with a 100ms handler sleep, so a
process stall longer than that between spawning the goroutines and
their dials let a request finish early and hand its connection to a
sibling. Review reproduced this with forced stalls: the default-transport
case then dialled 5 instead of 6 connections.
The handler now answers only once burstSize requests are waiting at the
same time. With HTTP/1.1 pinned that forces every burst onto burstSize
distinct connections regardless of scheduling, and the measured counts
stay exactly 6 new for Go's default and 0 for the tuned transport, also
under the same forced stalls.
* fix(database): set connection pool limits for Postgres (#350)
* fix(database): set connection pool limits for Postgres
OpenPostgres returned sqlx.Open's handle with database/sql's defaults:
no cap on open connections and two idle ones. Under load nearly every
request opened a new Postgres session, ran its few statements and closed
it again, paying a backend fork and SCRAM authentication each time.
Set the pool limits the issue suggests: 32 open and 32 idle connections,
idle connections closed after 5 minutes and every connection recycled
after 30 minutes. The values live in named constants because the mnd
linter rejects the literals inline.
The test (skipped without PROXY_DATABASE_URL, like the other Postgres
tests) takes 16 connections from the pool, releases them and checks that
all 16 stay idle; with the default pool only two survive.
Fixes#323
* test(database): pin the pool properties instead of the wiring
Review pointed out that the pool test compared MaxOpenConnections to the
constant it was set from, so the assertion followed the constant and
would have accepted postgresMaxOpenConns = 0 (unlimited), and that its
burst of 16 only proved MaxIdleConns >= 16.
The burst now takes postgresMaxIdleConns connections, so every
configured idle slot has to survive the release, and the open cap is
checked to be finite and large enough for that burst before any
connection is taken, so a cap below the idle count fails fast instead
of blocking in db.Conn. The doc comment now says which settings the
test covers; the idle-time and lifetime settings only show up in
DBStats.MaxIdleTimeClosed and MaxLifetimeClosed after minutes of
wall-clock time and stay unexercised.
* test(database): guard the pool test against a vacuous burst
Review showed that with the burst tied to postgresMaxIdleConns the test
also passed for a constant of 2, database/sql's default, or of 0, where
it took no connections at all. It now fails outright unless the
configured idle count exceeds the default, and every connection it
takes is released in a cleanup, so an assertion failure mid-burst no
longer leaves sessions open for the rest of the test binary.
* test(database): drop every schema table in the Postgres fixture (#349)
createTestPostgresDB dropped artifacts, versions, packages and
schema_info before calling CreateSchema, but not the migrations table.
On a database that has seen one test the migration records survive, so
the next CreateSchema fails while recording 001_add_packages_enrichment_
columns with a duplicate key on migrations_pkey. Running the package
against one Postgres therefore failed from the second Postgres-backed
test on.
Drop vulnerabilities, metadata_cache and migrations as well, so the
fixture clears every table CreateSchema creates. The package now passes
repeatedly against the same database.
* Discard a stale cache entry under the coalescing key (#348)
The digest-aware cache check discarded a stale entry before its caller
took the key. A slow caller could delete an entry another caller's
fetch had just committed, and everyone sharing that fetch then failed
to open it.
The check now only reports the miss. The caller running the shared
fetch discards the entry under the key, after the recheck, so one a
previous fetch refreshed is served, not deleted. The mismatch warning
fires once per refresh instead of once per request.
Swift HEAD no longer discards either, having no fetch to do it under.
It probes upstream as before, and the next GET replaces the entry.
Different digests or URLs, or no digest, use different keys and can
still collide on the storage path. That is the storage layout follow-up.
* Separate OCI request timeout from readiness probes in loopback test (#347)
The 250ms client is meant for the readiness poll, where a timeout is
retried. Reusing it for the OCI manifest request makes the test flake
under -race on Windows CI when fetch and cache I/O take longer, as
seen on #328. The request checks upstream routing, not latency.
* Fix NuGet cooldown enforcement for listings and downloads (#340)
* fix(nuget): enforce cooldown across metadata and downloads
* fix(nuget): support legacy registration and preserve valid metadata cache
* refactor(nuget): address maintainer review cleanup
* Fall back to embedded build info for Version (#352)
go install does not apply goreleaser's -ldflags -X, so binaries
installed that way reported 'dev'. Read debug.BuildInfo.Main.Version
when the ldflag is unset.
* docs(debian): trim comments on the named-repositories change
The motivation for separate archives was repeated across six files and the
reserved-name rationale across five. Keep each in one place -- the reference
docs -- and leave the code comments to what an informed reader cannot get
from the code: why an unknown first segment is not a 404 as it is for APK,
and why the main archive keeps its legacy cache identities.
Comments and prose only; no behaviour change.
* docs(debian): drop a stray space in a test comment
* Bump github.com/aws/aws-sdk-go-v2/service/ecr from 1.64.0 to 1.65.0 (#357)
Bumps [github.com/aws/aws-sdk-go-v2/service/ecr](https://github.com/aws/aws-sdk-go-v2) from 1.64.0 to 1.65.0.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.64.0...service/s3/v1.65.0)
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/ecr
dependency-version: 1.65.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump github.com/git-pkgs/spdx from 0.3.1 to 0.3.2 (#355)
Bumps [github.com/git-pkgs/spdx](https://github.com/git-pkgs/spdx) from 0.3.1 to 0.3.2.
- [Commits](https://github.com/git-pkgs/spdx/compare/v0.3.1...v0.3.2)
---
updated-dependencies:
- dependency-name: github.com/git-pkgs/spdx
dependency-version: 0.3.2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump zizmorcore/zizmor-action from 0.6.3 to 0.6.4 (#354)
Bumps [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) from 0.6.3 to 0.6.4.
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](70fb788f84...cc914d7f37)
---
updated-dependencies:
- dependency-name: zizmorcore/zizmor-action
dependency-version: 0.6.4
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump github.com/aws/aws-sdk-go-v2/config from 1.33.2 to 1.33.4 (#356)
Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.33.2 to 1.33.4.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.33.2...config/v1.33.4)
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
dependency-version: 1.33.4
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* feat(debian): report a misspelled repository name
A request to /debian/{name}/ for a name that is not configured fell through
to the main archive, so a typo surfaced as the upstream's own HTML 404 with
no mention of the repository. The reply now names the unknown repository and
lists the configured ones.
Scoped to {name}/dists/ and {name}/pool/. The main archive serves README,
indices/ and project/ at its root, so an unknown first segment stays a
main-archive path everywhere else.
apt prints only the status line, so this is visible via curl or the proxy
log rather than in apt output.
* fix(helm): support OCI references in HTTP repository indexes
* feat: add exact-version package denylist
* Exit non-zero when mirror denylist is invalid
* Omit malformed Helm chart releases instead of failing the index
A single bad release in a third-party index made the whole
repository unusable through the proxy.
* End an eviction pass that cannot evict anything
A failed delete leaves the record eligible, so the same batch came back
on the next iteration while the recorded size never dropped. Against a
backend refusing deletes the pass retried forever, burning CPU and
flooding the log. A canceled context made it faster: every delete fails
at once, so the loop spun as quickly as the database could answer.
The pass now ends when a batch clears no record, and returns as soon as
the context is canceled. Progress counts records rather than bytes,
because a cached record can have no size recorded. A pass can therefore
end with the cache still above its limit, leaving the records that
failed for the next pass to retry.
* Fix the handler test build
The denylist tests call NewDebianHandler with three arguments, but the
additional APT archives change gave it a fourth. Each passed CI on its
own branch, and together they leave internal/handler without a test
binary that compiles, so go vet and go test fail on main.
* docs: explain APT denylist limitations and version pinning
* Assert omitted Helm releases are not downloadable
* build(deps): bump go.opentelemetry.io/otel/sdk from 1.44.0 to 1.45.0
Bumps [go.opentelemetry.io/otel/sdk](https://github.com/open-telemetry/opentelemetry-go) from 1.44.0 to 1.45.0.
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.44.0...v1.45.0)
---
updated-dependencies:
- dependency-name: go.opentelemetry.io/otel/sdk
dependency-version: 1.45.0
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
* Widen wall-clock bound in flaky probe timeout test
Loaded CI runners exceeded the 500ms bound; 5s still proves the
probe timeout fired instead of blocking on Store.
* Add and align CI security and dependency automation
* fix(mirror): match cached artifacts by resolved filename
* Standardize the README license footer
* Test that GCS Delete of a missing object returns nil
* Give each fetch its own storage path
Every fetch of an artifact wrote one path, so fetches with different URLs
or digests, which do not share a coalescing key, could overwrite each
other's bytes, serve the wrong ones, or delete them on a digest mismatch
while the other was about to open them.
storeArtifact now writes {ecosystem}/{name}/{version}/{fetch id}/{filename},
with a random id per fetch. Most ecosystems learn the digest only after
the fetch, and storage has no rename, so the id is the one rule that fits
all of them. Existing records keep their paths and stay readable. On a
file:// bucket, Delete now removes an emptied fetch directory, which
fileblob leaves behind, and no other directory, since another fetch may
be creating its own inside it.
A path a record stops pointing at is queued in pending_deletes rather
than deleted, since a request that read the record may still open it. A
loop deletes queued paths after max(1h, direct_serve_ttl), whether or not
max_size is set. UpsertArtifact and clears apply only if the record still
holds the path the caller read, so racing commits each queue the path
they replaced and a clear never orphans a newer commit. Eviction still
deletes inline, and counts space as freed only when the record still
pointed at what it deleted.
* build(deps): bump docker/setup-buildx-action from 4.3.0 to 4.4.1
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 4.3.0 to 4.4.1.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](37fe631027...f87e5991a6)
---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
dependency-version: 4.4.1
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump docker/setup-qemu-action from 4.3.0 to 4.4.0
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 4.3.0 to 4.4.0.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](1f40c72289...9901266195)
---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
dependency-version: 4.4.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump docker/build-push-action from 7.3.0 to 7.4.0
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 7.3.0 to 7.4.0.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](53b7df96c9...c3c9e263c2)
---
updated-dependencies:
- dependency-name: docker/build-push-action
dependency-version: 7.4.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/aws/aws-sdk-go-v2/service/ecr
Bumps [github.com/aws/aws-sdk-go-v2/service/ecr](https://github.com/aws/aws-sdk-go-v2) from 1.65.0 to 1.66.0.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.65.0...service/s3/v1.66.0)
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/ecr
dependency-version: 1.66.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump golang.org/x/net from 0.58.0 to 0.59.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.58.0 to 0.59.0.
- [Commits](https://github.com/golang/net/compare/v0.58.0...v0.59.0)
---
updated-dependencies:
- dependency-name: golang.org/x/net
dependency-version: 0.59.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/git-pkgs/registries from 0.9.1 to 0.9.2
Bumps [github.com/git-pkgs/registries](https://github.com/git-pkgs/registries) from 0.9.1 to 0.9.2.
- [Release notes](https://github.com/git-pkgs/registries/releases)
- [Commits](https://github.com/git-pkgs/registries/compare/v0.9.1...v0.9.2)
---
updated-dependencies:
- dependency-name: github.com/git-pkgs/registries
dependency-version: 0.9.2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/git-pkgs/magic from 0.3.1 to 0.4.0
Bumps [github.com/git-pkgs/magic](https://github.com/git-pkgs/magic) from 0.3.1 to 0.4.0.
- [Release notes](https://github.com/git-pkgs/magic/releases)
- [Commits](https://github.com/git-pkgs/magic/compare/v0.3.1...v0.4.0)
---
updated-dependencies:
- dependency-name: github.com/git-pkgs/magic
dependency-version: 0.4.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump modernc.org/sqlite from 1.58.0 to 1.59.0
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.58.0 to 1.59.0.
- [Changelog](https://gitlab.com/cznic/sqlite/blob/master/CHANGELOG.md)
- [Commits](https://gitlab.com/cznic/sqlite/compare/v1.58.0...v1.59.0)
---
updated-dependencies:
- dependency-name: modernc.org/sqlite
dependency-version: 1.59.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/git-pkgs/vers from 0.7.0 to 0.7.1
Bumps [github.com/git-pkgs/vers](https://github.com/git-pkgs/vers) from 0.7.0 to 0.7.1.
- [Commits](https://github.com/git-pkgs/vers/compare/v0.7.0...v0.7.1)
---
updated-dependencies:
- dependency-name: github.com/git-pkgs/vers
dependency-version: 0.7.1
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/aws/aws-sdk-go-v2/config
Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.33.4 to 1.33.5.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.33.4...config/v1.33.5)
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
dependency-version: 1.33.5
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump github.com/git-pkgs/archives from 0.7.0 to 0.7.1
Bumps [github.com/git-pkgs/archives](https://github.com/git-pkgs/archives) from 0.7.0 to 0.7.1.
- [Commits](https://github.com/git-pkgs/archives/compare/v0.7.0...v0.7.1)
---
updated-dependencies:
- dependency-name: github.com/git-pkgs/archives
dependency-version: 0.7.1
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix(conda): preserve gzip for large repodata indexes
* fix: centralize pass-through response relaying
* Clear before deleting on eviction, and requeue failed reclaims
Eviction deleted an object before checking its record still pointed at
it. If a refetch had moved the record, that object was queued and a
request that read the record earlier could still be opening it. Eviction
now clears first, skips the delete if the record moved, and queues the
object when the delete fails after a clear.
Reclaim queues a failed delete again, which moves it behind the rest, so
objects the backend keeps refusing cannot fill every batch.
Also drop the clearArtifactCache passthrough and ClearCachedArtifact's
unused context, note that only tests call ArtifactPath, and document that
max_size does not count objects waiting to be reclaimed.
* Log and count requests aborted mid-relay
A truncated relay panics with http.ErrAbortHandler, which skipped the
request log line, the request metrics and the access log entry.
* Batch cache hit writes
Every cached download wrote its hit in its own UPDATE. SQLite runs on
one connection, so each download was a write transaction the rest
queued behind, holding a warm npm cache near 1,000 requests a second
with the CPU two thirds idle.
Hits are now counted in memory and written in one transaction every
database.hit_flush_interval, "1s" by default, "0" for a write per hit.
A failed write keeps the hits for the next flush, and Close writes
what is pending. The same test then served about 2,200 requests a
second.
Closes#322.
* fix pnpm audit
* apply review changes
* Name every mounted route in requestEcosystem
- Map /apk to alpine, matching what the handler writes to the database
- Report /helm, /homebrew, /generic and /swift under their own names
- Test every one of the 23 mounts, so the next route added cannot
quietly fall through to "other"
Part 1 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(handler): stop asserting on an upstream connection the proxy abandons
TestRelayRoutes fails intermittently on CI with "write: connection reset
by peer" at the upstream handler's final Flush, on /hex/packages/demo and
/gem/info/demo.
Those two routes are the only ones in the table that fan out a second
upstream request: with cooldown enabled, the hex and gem handlers fetch
version timestamps concurrently with the artifact. The test server answers
that sidecar request with the same 64 KiB chunked 502, but the handler
returns as soon as it sees a non-200 and closes the body unread, so the
proxy drops the connection while the test server is still writing to it.
Whether the reset lands before or after the Flush is a scheduling race,
which is why it only surfaces on loaded runners.
The flush error says nothing about the response under test, which is
asserted on the downstream side, so stop reporting it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Per-ecosystem cache and download statistics
- Add GetEcosystemStats, aggregating packages, versions, artifacts,
cache size, downloads and downloaded bytes per ecosystem
- Publish six proxy_ecosystem_* gauges from it on the existing
one-minute cache-stats tick
- Set and selectively delete rather than Reset, so no scrape lands on
a half-populated vector
- Report artifacts with no package row under "unattributed" rather
than dropping them, so the figures still add up
Part 2 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Add the /ui/analytics page
- Ring of download size by ecosystem, cache figures, per-ecosystem
table, vulnerability overview and a Runtime card
- Add metrics.Gather so the page can render counters that were never
in the database
- Add proxy_response_bytes_total and response-writer byte counting
- Serve a retained snapshot behind a staleness banner when the
aggregation fails, rather than rendering old figures as current
- Extract the security overview into a shared component
Part 3 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Report the per-ecosystem breakdown from GET /stats
- Add downloaded_bytes, downloads and an ecosystems array, served from
the snapshot the gauges and the page already share
- Add stats_unavailable so a failed aggregation is distinguishable
from an idle proxy
- Regenerate the OpenAPI spec
Part 4 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Add a Grafana dashboard
- 33 panels, no hardcoded data source UID
- Three ecosystem variables, because the label means three different
things across /metrics: the package record, the request path, and
the handler's own name
Part 5 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Attribute requests to a caller and a client tool
- Track per-caller request and byte counts in a bounded table
- Export proxy_client_requests_total and
proxy_client_response_bytes_total over a closed label set; addresses
are never labels
- Gate the sources card behind ui_request_sources, off by default,
since /ui carries no authentication of its own
- Move access_log.trust_forwarded_for to a top-level key
- Keep remote as the TCP peer and add remote_ip alongside it
Part 6 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Update enrichment to v0.7.2 (#394)
* Update ecosystems-go to v0.5.0
* Update enrichment to v0.7.2
* Update git-pkgs dependencies (#396)
* Keep hit timestamps from moving backwards
With proxies sharing a database, a batch can flush after a newer hit is
already written, and overwriting last_accessed_at and updated_at with
its older time could reorder LRU eviction. Both now keep the later of
the stored and incoming times. The count still adds up.
* build(deps): bump alpine from 3.24.1 to 3.24.2 (#397)
Bumps alpine from 3.24.1 to 3.24.2.
---
updated-dependencies:
- dependency-name: alpine
dependency-version: 3.24.2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Install development tools directly in CI (#398)
* Add storage.cache_artifacts to serve artifacts without storing them (#387)
* Add storage.passthrough to serve artifacts without storing them
With storage.passthrough enabled the proxy streams every artifact from
upstream to the client and never writes it to storage or the cache
database. Metadata filtering, cooldown and the denylist work as before,
so the proxy can sit behind another cache (e.g. an Artifactory remote)
purely as a policy layer without holding a second copy of every package.
Artifacts whose digest is known up front (OCI, Swift, Helm) are verified
while streaming. Their responses are sent chunked and the connection is
aborted on a mismatch, so a client never receives a tampered artifact as
a complete response.
Passthrough is rejected together with scanning, direct_serve and
mirror_api, and the mirror command refuses to run with it, since all of
them depend on stored artifacts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address review: abort truncated streams, guard every cache read
- serveArtifact aborts the response on any body read error, and when fewer
bytes than the declared size were written, not only on a digest mismatch.
A failed or short upstream body in passthrough mode no longer reaches the
client as a complete 200.
- Streamed upstream read failures are logged and counted as
stream_failed upstream errors before the response is aborted.
- checkCache reports a miss when artifact caching is off, so every cache
read path, including Swift archive HEAD requests, ignores entries stored
before the mode was enabled. Denylisted versions are still rejected.
- Rename storage.passthrough to storage.cache_artifacts (default true,
PROXY_STORAGE_CACHE_ARTIFACTS), which names what actually changes.
Tests cover an OCI blob shorter than its Content-Length and an npm tarball
whose chunked response ends without the final chunk, both served through
the HTTP handlers, plus a Swift HEAD request against a cached archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Rename Proxy.Passthrough to StreamArtifacts
Match the storage.cache_artifacts option: the field says what changes
(artifacts are streamed instead of stored) and its zero value keeps the
usual caching behaviour for every Proxy built without the server config,
such as the mirror command and tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Show the by-client table regardless of ui_request_sources
The flag exists to withhold caller addresses. The by-client table carries
none, and the same figures are already public at /metrics, so gating it
left two metrics with no tile on the page in the default config.
* Drop unused registry from TestUpdateEcosystemStatsSumsAliasedRows
* Cache rewritten npm and Composer metadata (#402)
The npm and Composer handlers rewrite every metadata document they serve
so that download URLs point at the proxy: decode the whole document into
generic maps, change the URLs, encode it again, and for Composer expand
the minified format first. That ran on every request, cached metadata
included. With metadata caching on and upstream out of the picture, a
cached request still cost 7 ms for @babel/core, 17 ms for
symfony/console and 155 ms and 134 MB of allocations for typescript, and
an 8 vCPU VM serving cached npm packuments to 20 clients ran the proxy
at 450% CPU for about 1,100 requests a second.
Rewritten documents are now kept in memory, keyed by the ecosystem,
proxy URL, package and a SHA-256 of the raw document, so new bytes from
upstream are rewritten again and nothing is served stale. Requests that
arrive while a document is being rewritten wait for that rewrite rather
than running their own; a waiter leaves when its client does, and the
rewrite still completes and is cached. The denylist is fixed at startup,
so it needs no place in the key. Cooldown filtering depends on the
current time, so with cooldown on the cache is bypassed.
metadata_rewrite_cache_size bounds the cache (default "256MB", least
recently used out first, "0" to rewrite on every request). NewProxy
callers keep the old behaviour unless they set it.
Whole cached requests, measured locally:
@babel/core 7.1 ms -> 0.38 ms, 54,489 -> 119 allocations
typescript 155 ms -> 10.5 ms, 1.37M -> 133 allocations
symfony/console 17.4 ms -> 0.48 ms, 165,909 -> 123 allocations
What remains is reading the raw document from storage and hashing it.
* Copy inherited Composer fields shallowly when expanding (#401)
expandMinifiedVersions deep-copied every inherited field into every
version (b68184c) so that rewriting one version's dist URL in place
could not change the versions that inherited it. For a package with a
long history that means recursively copying require, autoload and the
rest hundreds of times per document, on every metadata request.
dist is the only field the proxy changes after expansion, so copy just
that: rewriteDistURL now gives the version its own dist map before
setting the URL, and expansion shares the other inherited values.
TestComposerExpandMinifiedSharedDistReferences still guards the original
bug, and fails if the copy in rewriteDistURL is removed.
On the symfony/console metadata from Packagist the rewritten output is
byte-identical, and the rewrite drops from about 20.7 ms to 13.8 ms,
with allocations down from 13.6 MB to 9 MB.
* Support HTTP byte-range requests for local file cached artifact downloads (#404)
* feat(storage): expose efficient seek for local files
Open file-backed artifacts with os.Open so local readers support
efficient seeking. Leave cloud storage readers unchanged.
Keep the legacy sidecar mapping test focused on fileblob's reader:
Blob.Open now reads local files directly and no longer consults the
sidecar. Continue checking sidecar cleanup and post-store reads.
* feat(handler): preserve efficient seek through integrity checks
Keep the cache result's single Reader field and preserve io.Seeker
when the underlying storage reader supports efficient seeking.
Sequential reads continue through whole-object integrity verification.
A successful seek switches subsequent reads to the source for range
responses, where full-object verification is not possible.
Test both modes and verify the source is closed once.
* feat(handler): add request-aware byte range responses
Add a request-aware artifact serving helper for bounded, open-ended,
and suffix byte ranges, including If-Range handling, 206 responses,
and 416 responses for valid unsatisfiable ranges.
Ignore malformed and multi-range requests by serving the full artifact.
Advertise byte-range support only for seekable readers, preserving HEAD,
redirect, and non-seekable behavior. Clarify that net/http recovers
ErrAbortHandler per request and keeps the server running.
* feat(handler): enable byte ranges across artifact downloads
Forward each artifact request to the shared range-aware response helper,
including cached generic release assets and OCI blobs. This lets the
handler honor ranges only when its reader supports efficient seeking,
without changing direct-storage redirect behavior.
Add warm-cache endpoint tests that assert partial response headers and
bytes without an upstream refetch. Document supported range behavior,
unsupported reader fallback, and the integrity limitation of partial reads.
* Coalesce concurrent metadata fetches (#400)
* Coalesce concurrent metadata fetches
Concurrent cache misses for one artifact already share a single upstream
fetch (#329), but metadata misses did not: every request for a package's
metadata went to the registry, even when the same document was already
being fetched. In CI that is the common case. Composer resolves versions
on every job, since MediaWiki core and its extensions commit no
composer.lock, and each package is one /p2/ document, so a burst of jobs
fetched each document once per job.
Metadata misses now follow the same pattern as artifacts: the first caller
fetches, the rest wait for its result, and the caller that takes the key
rechecks the cache first so a fetch that just finished is not repeated.
The key includes the Accept and Accept-Encoding headers, which change the
bytes upstream returns (npm's abbreviated and full documents), and whether
the caller validates the response.
One difference from artifacts: the shared fetch is detached from the first
caller's cancellation. Artifact fetches keep it for scanning and mirroring,
neither of which applies to metadata, and otherwise one job disconnecting
would fail every job waiting on the same document. runScan detaches the
same way, and the HTTP client's timeout still bounds the fetch.
This covers every ecosystem that goes through fetchOrCacheMetadata
(Composer, npm, PyPI, Maven, Cargo, Swift, Pub, Helm, NuGet), with
metadata caching on or off. Streamed metadata with caching off is not
coalesced.
On an 8 vCPU test instance with caching off, 10 concurrent clients
requesting the npm metadata of 1,263 packages went from 12,630 upstream
requests to 1,898. With cache_metadata on, 20 clients requesting 1,538
packages made exactly 1,538.
* Run each caller's validate on a shared metadata fetch
A caller that joined another's metadata fetch got the shared bytes
without running its own validate. NuGet's cooldown path uses validate to
decode the document for the request, so a joined download found no
publication date and was allowed, bypassing the cooldown.
coalesceMetadata now reports whether the caller joined, and
coalescedMetadataMiss runs that caller's validate on the shared bytes.
Validation before caching is unchanged: the first caller still runs it
inside the fetch.
TestNuGetCooldownConcurrentDownloads reproduces the bypass through
NuGetHandler.Routes() with two overlapping downloads of a version inside
its cooldown.
* Support npm content-addressed tarball URLs
* Add cooldown package pattern overrides (#407)
* Add cooldown package pattern overrides
* Expose full cooldown decisions through pattern policy
* Reject unsafe cooldown pattern syntax
* Fix cached package ecosystem filters
* Update supportedEcosystems comment for filter pills
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: pinguinfuss <pinguinfuss@users.noreply.github.com>
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
Co-authored-by: montehurd <montehurd@users.noreply.github.com>
Co-authored-by: Andrew Rich <andrew.rich@beacon.bio>
Co-authored-by: Abhinav Gautam <abgautam1017@gmail.com>
Co-authored-by: montehurd <mhurd@wikimedia.org>
Co-authored-by: wickedOne <wicliff.wolda@gmail.com>
Co-authored-by: Wicliff Wolda <wicliff.wolda@quentic.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Daniil Skripchenko <90247855+DANIILSKRIPCHENKO@users.noreply.github.com>
Co-authored-by: PhantomPhoton <120879831+PhantomPhoton@users.noreply.github.com>
* Coalesce concurrent metadata fetches
Concurrent cache misses for one artifact already share a single upstream
fetch (#329), but metadata misses did not: every request for a package's
metadata went to the registry, even when the same document was already
being fetched. In CI that is the common case. Composer resolves versions
on every job, since MediaWiki core and its extensions commit no
composer.lock, and each package is one /p2/ document, so a burst of jobs
fetched each document once per job.
Metadata misses now follow the same pattern as artifacts: the first caller
fetches, the rest wait for its result, and the caller that takes the key
rechecks the cache first so a fetch that just finished is not repeated.
The key includes the Accept and Accept-Encoding headers, which change the
bytes upstream returns (npm's abbreviated and full documents), and whether
the caller validates the response.
One difference from artifacts: the shared fetch is detached from the first
caller's cancellation. Artifact fetches keep it for scanning and mirroring,
neither of which applies to metadata, and otherwise one job disconnecting
would fail every job waiting on the same document. runScan detaches the
same way, and the HTTP client's timeout still bounds the fetch.
This covers every ecosystem that goes through fetchOrCacheMetadata
(Composer, npm, PyPI, Maven, Cargo, Swift, Pub, Helm, NuGet), with
metadata caching on or off. Streamed metadata with caching off is not
coalesced.
On an 8 vCPU test instance with caching off, 10 concurrent clients
requesting the npm metadata of 1,263 packages went from 12,630 upstream
requests to 1,898. With cache_metadata on, 20 clients requesting 1,538
packages made exactly 1,538.
* Run each caller's validate on a shared metadata fetch
A caller that joined another's metadata fetch got the shared bytes
without running its own validate. NuGet's cooldown path uses validate to
decode the document for the request, so a joined download found no
publication date and was allowed, bypassing the cooldown.
coalesceMetadata now reports whether the caller joined, and
coalescedMetadataMiss runs that caller's validate on the shared bytes.
Validation before caching is unchanged: the first caller still runs it
inside the fetch.
TestNuGetCooldownConcurrentDownloads reproduces the bypass through
NuGetHandler.Routes() with two overlapping downloads of a version inside
its cooldown.
* feat(storage): expose efficient seek for local files
Open file-backed artifacts with os.Open so local readers support
efficient seeking. Leave cloud storage readers unchanged.
Keep the legacy sidecar mapping test focused on fileblob's reader:
Blob.Open now reads local files directly and no longer consults the
sidecar. Continue checking sidecar cleanup and post-store reads.
* feat(handler): preserve efficient seek through integrity checks
Keep the cache result's single Reader field and preserve io.Seeker
when the underlying storage reader supports efficient seeking.
Sequential reads continue through whole-object integrity verification.
A successful seek switches subsequent reads to the source for range
responses, where full-object verification is not possible.
Test both modes and verify the source is closed once.
* feat(handler): add request-aware byte range responses
Add a request-aware artifact serving helper for bounded, open-ended,
and suffix byte ranges, including If-Range handling, 206 responses,
and 416 responses for valid unsatisfiable ranges.
Ignore malformed and multi-range requests by serving the full artifact.
Advertise byte-range support only for seekable readers, preserving HEAD,
redirect, and non-seekable behavior. Clarify that net/http recovers
ErrAbortHandler per request and keeps the server running.
* feat(handler): enable byte ranges across artifact downloads
Forward each artifact request to the shared range-aware response helper,
including cached generic release assets and OCI blobs. This lets the
handler honor ranges only when its reader supports efficient seeking,
without changing direct-storage redirect behavior.
Add warm-cache endpoint tests that assert partial response headers and
bytes without an upstream refetch. Document supported range behavior,
unsupported reader fallback, and the integrity limitation of partial reads.
expandMinifiedVersions deep-copied every inherited field into every
version (b68184c) so that rewriting one version's dist URL in place
could not change the versions that inherited it. For a package with a
long history that means recursively copying require, autoload and the
rest hundreds of times per document, on every metadata request.
dist is the only field the proxy changes after expansion, so copy just
that: rewriteDistURL now gives the version its own dist map before
setting the URL, and expansion shares the other inherited values.
TestComposerExpandMinifiedSharedDistReferences still guards the original
bug, and fails if the copy in rewriteDistURL is removed.
On the symfony/console metadata from Packagist the rewritten output is
byte-identical, and the rewrite drops from about 20.7 ms to 13.8 ms,
with allocations down from 13.6 MB to 9 MB.
The npm and Composer handlers rewrite every metadata document they serve
so that download URLs point at the proxy: decode the whole document into
generic maps, change the URLs, encode it again, and for Composer expand
the minified format first. That ran on every request, cached metadata
included. With metadata caching on and upstream out of the picture, a
cached request still cost 7 ms for @babel/core, 17 ms for
symfony/console and 155 ms and 134 MB of allocations for typescript, and
an 8 vCPU VM serving cached npm packuments to 20 clients ran the proxy
at 450% CPU for about 1,100 requests a second.
Rewritten documents are now kept in memory, keyed by the ecosystem,
proxy URL, package and a SHA-256 of the raw document, so new bytes from
upstream are rewritten again and nothing is served stale. Requests that
arrive while a document is being rewritten wait for that rewrite rather
than running their own; a waiter leaves when its client does, and the
rewrite still completes and is cached. The denylist is fixed at startup,
so it needs no place in the key. Cooldown filtering depends on the
current time, so with cooldown on the cache is bypassed.
metadata_rewrite_cache_size bounds the cache (default "256MB", least
recently used out first, "0" to rewrite on every request). NewProxy
callers keep the old behaviour unless they set it.
Whole cached requests, measured locally:
@babel/core 7.1 ms -> 0.38 ms, 54,489 -> 119 allocations
typescript 155 ms -> 10.5 ms, 1.37M -> 133 allocations
symfony/console 17.4 ms -> 0.48 ms, 165,909 -> 123 allocations
What remains is reading the raw document from storage and hashing it.
The flag exists to withhold caller addresses. The by-client table carries
none, and the same figures are already public at /metrics, so gating it
left two metrics with no tile on the page in the default config.
* Add storage.passthrough to serve artifacts without storing them
With storage.passthrough enabled the proxy streams every artifact from
upstream to the client and never writes it to storage or the cache
database. Metadata filtering, cooldown and the denylist work as before,
so the proxy can sit behind another cache (e.g. an Artifactory remote)
purely as a policy layer without holding a second copy of every package.
Artifacts whose digest is known up front (OCI, Swift, Helm) are verified
while streaming. Their responses are sent chunked and the connection is
aborted on a mismatch, so a client never receives a tampered artifact as
a complete response.
Passthrough is rejected together with scanning, direct_serve and
mirror_api, and the mirror command refuses to run with it, since all of
them depend on stored artifacts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address review: abort truncated streams, guard every cache read
- serveArtifact aborts the response on any body read error, and when fewer
bytes than the declared size were written, not only on a digest mismatch.
A failed or short upstream body in passthrough mode no longer reaches the
client as a complete 200.
- Streamed upstream read failures are logged and counted as
stream_failed upstream errors before the response is aborted.
- checkCache reports a miss when artifact caching is off, so every cache
read path, including Swift archive HEAD requests, ignores entries stored
before the mode was enabled. Denylisted versions are still rejected.
- Rename storage.passthrough to storage.cache_artifacts (default true,
PROXY_STORAGE_CACHE_ARTIFACTS), which names what actually changes.
Tests cover an OCI blob shorter than its Content-Length and an npm tarball
whose chunked response ends without the final chunk, both served through
the HTTP handlers, plus a Swift HEAD request against a cached archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Rename Proxy.Passthrough to StreamArtifacts
Match the storage.cache_artifacts option: the field says what changes
(artifacts are streamed instead of stored) and its zero value keeps the
usual caching behaviour for every Proxy built without the server config,
such as the mirror command and tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
With proxies sharing a database, a batch can flush after a newer hit is
already written, and overwriting last_accessed_at and updated_at with
its older time could reorder LRU eviction. Both now keep the later of
the stored and incoming times. The count still adds up.
- Track per-caller request and byte counts in a bounded table
- Export proxy_client_requests_total and
proxy_client_response_bytes_total over a closed label set; addresses
are never labels
- Gate the sources card behind ui_request_sources, off by default,
since /ui carries no authentication of its own
- Move access_log.trust_forwarded_for to a top-level key
- Keep remote as the TCP peer and add remote_ip alongside it
Part 6 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- 33 panels, no hardcoded data source UID
- Three ecosystem variables, because the label means three different
things across /metrics: the package record, the request path, and
the handler's own name
Part 5 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Add downloaded_bytes, downloads and an ecosystems array, served from
the snapshot the gauges and the page already share
- Add stats_unavailable so a failed aggregation is distinguishable
from an idle proxy
- Regenerate the OpenAPI spec
Part 4 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Ring of download size by ecosystem, cache figures, per-ecosystem
table, vulnerability overview and a Runtime card
- Add metrics.Gather so the page can render counters that were never
in the database
- Add proxy_response_bytes_total and response-writer byte counting
- Serve a retained snapshot behind a staleness banner when the
aggregation fails, rather than rendering old figures as current
- Extract the security overview into a shared component
Part 3 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Add GetEcosystemStats, aggregating packages, versions, artifacts,
cache size, downloads and downloaded bytes per ecosystem
- Publish six proxy_ecosystem_* gauges from it on the existing
one-minute cache-stats tick
- Set and selectively delete rather than Reset, so no scrape lands on
a half-populated vector
- Report artifacts with no package row under "unattributed" rather
than dropping them, so the figures still add up
Part 2 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Map /apk to alpine, matching what the handler writes to the database
- Report /helm, /homebrew, /generic and /swift under their own names
- Test every one of the 23 mounts, so the next route added cannot
quietly fall through to "other"
Part 1 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every cached download wrote its hit in its own UPDATE. SQLite runs on
one connection, so each download was a write transaction the rest
queued behind, holding a warm npm cache near 1,000 requests a second
with the CPU two thirds idle.
Hits are now counted in memory and written in one transaction every
database.hit_flush_interval, "1s" by default, "0" for a write per hit.
A failed write keeps the hits for the next flush, and Close writes
what is pending. The same test then served about 2,200 requests a
second.
Closes#322.
2026-10-01 00:13:58 -07:00
506 changed files with 99259 additions and 4313 deletions
- A new Prometheus metric needs a tile on `/ui/analytics` and an entry in `metricSurface` (`internal/server/analytics_coverage_test.go`). The page is meant to be a complete view of `/metrics`, so `TestEveryMetricIsSurfaced` fails until both exist.
## Testing
@ -211,6 +212,24 @@ Add PyPI registry support
- Add tests for wheel and sdist downloads
```
## Reviewing and Merging
Maintainers are the members of the proxy team in the git-pkgs org. Any maintainer can review and merge. A pull request needs green CI and an approval from a maintainer other than its author. A "request changes" review from any maintainer blocks it until that reviewer approves or withdraws the review.
Using AI to write or review is fine, but whoever approves a pull request should understand it and stand behind it. If it touches an ecosystem you don't use, you can ask someone who does to look before you approve.
Open an issue before a bigger change: anything that changes behaviour users rely on, new configuration that would be hard to remove, or a change that fans out into other git-pkgs modules. If no maintainer objects within a few days, open the pull request; it's reviewed like any other. Objections are worked out in the issue, with @andrew breaking ties for now. Where possible, deprecate before removing: an old configuration key keeps working, with a warning, in at least one release before the one that removes it.
## Releases
A release is cut on the 1st of each month by `.github/workflows/monthly-release.yml` when `main` is green and something has merged since the last tag. It is a patch release, or a new minor version if a pull request labelled `breaking` was merged since then (a new major version from 1.0 on), so label pull requests that change behaviour users rely on.
A patch release is published on its own. A breaking release is created as a draft, without images or a chart: write the upgrade notes in it and publish it, which starts `publish.yml` for the images and the chart. Until then the monthly run releases nothing.
Scheduled runs only report what they would release until the repository variable `MONTHLY_RELEASE` is set to `true`. The workflow can also be started by hand from the Actions tab, as a dry run by default.
To release by hand, push a `v*` tag.
## Questions?
Open an issue on GitHub if you have questions or need help getting started.
@ -20,6 +20,8 @@ A 3-day cooldown means that when `lodash` publishes version `4.18.0`, your build
Resolution order: package override, then ecosystem override, then global default. This lets you set a conservative default and carve out exceptions for packages where you need faster updates. See [docs/configuration.md](docs/configuration.md) for the full config reference.
One instance can offer several cooldown policies through [named profiles](docs/configuration.md#cooldown-profiles). Configure `cooldown.profiles` and point clients at paths such as `/cooldown/3d/npm/` or `/cooldown/7d/cargo/`. Profiles share the package cache; existing URLs keep using the global policy.
## Artifact Scanning
Cooldown only looks at a version's publish timestamp — it never inspects the actual bytes. Artifact scanning closes that gap: when enabled, every artifact is staged into storage and scanned by one or more external services (trivy, ClamAV, Wiz, or anything else that speaks a small HTTP/JSON contract) before it's committed to the cache and served to clients.
@ -47,11 +49,11 @@ The proxy never uploads artifact bytes to a scanner. Each scanner is notified wi
| npm | JavaScript | Yes | ✓ |
| Cargo | Rust | Yes | ✓ |
| RubyGems | Ruby | Yes | ✓ |
| Go proxy | Go | | ✓ |
| Go proxy | Go | Yes† | ✓ |
| Hex | Elixir | Yes* | ✓ |
| pub.dev | Dart | Yes | ✓ |
| PyPI | Python | Yes | ✓ |
| Maven | Java | | ✓ |
| Maven | Java | Yes‡ | ✓ |
| Gradle Build Cache | Java/Kotlin | | ✓ |
| NuGet | .NET | Yes | ✓ |
| Composer | PHP | Yes | ✓ |
@ -65,15 +67,21 @@ The proxy never uploads artifact bytes to a scanner. Each scanner is notified wi
| Debian | Debian/Ubuntu | | ✓ |
| RPM | RHEL/Fedora | | ✓ |
| Alpine | Alpine Linux | | ✓ |
| Arch | Arch Linux | | ✗ |
| Chef | Chef | | ✗ |
| Arch | Arch Linux | | ✓ |
| FreeBSD | FreeBSD (ports and pkgbase) | | ✓ |
| Chef Supermarket | Chef | | ✓ |
| Generic | Any | | ✓ |
| Helm | Kubernetes | | ✓ |
| Git | Any | | ✓ |
| Helm | Kubernetes | Yes | ✓ |
| Vagrant | Vagrant | | ✗ |
Cooldown requires publish timestamps in metadata. Registries without a "Yes" in the cooldown column either don't expose timestamps or haven't been wired up yet.
\* Hex cooldown requires disabling registry signature verification (`HEX_NO_VERIFY_REPO_ORIGIN=1`) since the proxy re-encodes the protobuf payload.
\* Hex cooldown requires disabling registry signature verification (`HEX_UNSAFE_REGISTRY=1`) since the proxy re-encodes the protobuf payload. Origin verification can stay enabled.
† Go cooldown uses the time in each version's `.info`, which is the commit time of the tagged revision, not when the tag was pushed.
‡ Maven cooldown uses the `Last-Modified` header of each version's `.pom`. On Maven Central that is the deploy time; a Nexus or Artifactory upstream may report when it cached the file instead. SNAPSHOT versions are not held back.
## Install
@ -81,7 +89,11 @@ Cooldown requires publish timestamps in metadata. Registries without a "Yes" in
brew install git-pkgs/git-pkgs/proxy
```
Or download a binary from the [releases page](https://github.com/git-pkgs/proxy/releases).
Or download a binary from the [releases page](https://github.com/git-pkgs/proxy/releases), or run the container image:
```bash
docker run -p 8080:8080 -v proxy-data:/data ghcr.io/git-pkgs/proxy:latest
```
### Helm
@ -98,6 +110,33 @@ using SQLite and filesystem artifact storage under `/data`. See
[`deploy/charts/proxy/values.yaml`](deploy/charts/proxy/values.yaml) for ingress,
external database and object-storage configuration options.
The SQLite database needs a local disk or a block-backed volume, even with one
replica; `ReadWriteOnce` alone doesn't guarantee that, since the storage class
can still be NFS. If the cluster only offers network filesystem storage, use
Postgres. Artifact storage can stay on NFS, separately from the database. See
[SQLite](docs/configuration.md#sqlite).
To trust a private CA, for example when upstreams sit behind a TLS-intercepting
corporate proxy or are internal registries signed by an in-house CA, mount the
certificate with `extraVolumes` and `extraVolumeMounts` and add its directory
to `SSL_CERT_DIR`. The proxy process makes all upstream TLS connections itself,
Git fetches included, so no other configuration is needed. Public CAs stay
The proxy is now running. Configure your package managers to use it.
The proxy is now running. Configure your package managers to use it. For image
builds, see [Using proxy in Docker builds](docs/docker-builds.md).
## OpenAPI (Swagger)
@ -159,7 +199,12 @@ through the proxy: the audit and signing-key endpoints are passed through to the
configured upstream registry, with upstream authentication applied. Advisories
therefore come from upstream's database, not from the proxy's own vulnerability
data, and versions withheld by [cooldown](#version-cooldown) are not excluded
from the report.
from the report. Packuments point each version's attestations (provenance) at
the proxy, which fetches them from the package's registry and caches them;
attestations of versions held back by cooldown or blocked by the denylist are
not served.
To serve private packages through the same registry URL, see [Private package routes](docs/configuration.md#private-package-routes). To see new releases before `metadata_ttl` runs out, see [Change feeds](docs/configuration.md#change-feeds).
The artifact domain proxies manifests and bottle blobs under `/v2/homebrew/core/`. GHCR routing is limited to that repository. Source archives, cask application downloads, custom tap artifacts, and legacy flat-file bottle mirrors use Homebrew's normal fallback URLs. Keep fallback enabled by leaving `HOMEBREW_ARTIFACT_DOMAIN_NO_FALLBACK` unset.
Enable `cache_metadata` or set `PROXY_CACHE_METADATA=true` to retain Homebrew JSON API responses for offline fallback. Bottle blobs and their OCI manifests are cached without this setting.
Enable `cache_metadata` or set `PROXY_CACHE_METADATA=true` to retain Homebrew JSON API responses for offline fallback. Bottle blobs and their OCI manifests are cached without this setting.`proxy mirror pkg:brew/<formula>` pre-fetches a formula's bottles (see [Homebrew bottles](docs/configuration.md#homebrew-bottles)).
The upstreams default to `https://formulae.brew.sh/api` for the JSON API and `https://ghcr.io` for artifacts. To chain this proxy to another proxy, configure its Homebrew endpoints as the upstreams:
@ -225,10 +288,10 @@ The equivalent environment variables are `PROXY_UPSTREAM_HOMEBREW_API` and `PROX
@ -237,6 +300,14 @@ Or set the environment variable:
export HEX_MIRROR=http://localhost:8080/hex
```
When the proxy applies a cooldown, also set:
```bash
export HEX_UNSAFE_REGISTRY=1
```
Filtering releases removes the registry signature. The proxy preserves the package name and repository fields, so origin verification can stay enabled. Without a cooldown, leave signature verification enabled too.
### pub.dev (Dart/Flutter)
Set the PUB_HOSTED_URL environment variable:
@ -278,6 +349,8 @@ Add to your `~/.m2/settings.xml`:
The `/maven/` endpoint uses Maven Central as primary upstream and falls back to the Gradle Plugin Portal for Gradle plugin marker metadata and related artifacts when the primary upstream returns not found.
Further Maven repositories can be configured under `upstream.maven_repositories` and are served at `/maven/{name}/`, each from its own upstream (see [Maven repositories](docs/configuration.md#maven-repositories)). Point a second `<repository>` or Gradle `maven(url = ...)` entry at `http://localhost:8080/maven/{name}/`.
For Gradle plugin resolution via the same proxy endpoint:
```kotlin
@ -290,20 +363,20 @@ pluginManagement {
### Gradle HTTP Build Cache
Configure in `settings.gradle(.kts)`:
Configure in `settings.gradle.kts`:
```kotlin
buildCache {
local {
enabled = false
}
remote<HttpBuildCache> {
url = uri("http://localhost:8080/gradle/")
push = true
}
}
```
Enable build caching with `org.gradle.caching=true` in `gradle.properties`, or pass `--build-cache` when running Gradle. The local cache stays enabled and remote uploads are disabled by default.
To upload from CI, set `gradle.build_cache.read_only` to `false` on the proxy and add `isPush = System.getenv("CI") != null` inside the client's `remote<HttpBuildCache>` block. See [docs/configuration.md](docs/configuration.md#gradle-build-cache).
`composer audit`, `audit.block-insecure` and Packagist's malware filter list
work through the proxy when `upstream.composer_repository` offers them in its
`packages.json`, as repo.packagist.org does. The proxy advertises them in its
own `packages.json` and passes the advisories and filter list requests through
to the upstream uncached. Names of packages in `upstream.composer_routes` or
`hosted.composer_vendors` are removed from advisories requests before they are
sent. Versions withheld by [cooldown](#version-cooldown) are not excluded from
the report.
To serve private packages through the same URL, see [Private package routes](docs/configuration.md#private-package-routes). To see new releases before `metadata_ttl` runs out, see [Change feeds](docs/configuration.md#change-feeds).
The proxy serves the `/universe` dependency index and the cookbook API, rewriting every download location to itself while leaving dependency constraints unchanged. Cookbook archives are fetched through the Supermarket download redirect and cached, so they stay available when the Supermarket is down. With `cache_metadata` on, the universe and cookbook metadata follow the usual TTL and stale fallback.
To proxy a private Supermarket instead of `supermarket.chef.io`, set `upstream.chef` (or `PROXY_UPSTREAM_CHEF`) and add credentials under `upstream.auth` if it needs them. See [docs/configuration.md](docs/configuration.md#upstream-registries).
### Conda
Configure in `~/.condarc`:
@ -367,12 +469,17 @@ default_channels:
- http://localhost:8080/conda/main
```
`default_channels` also routes requests for `defaults` through the proxy, including `-c defaults` and `environment.yml` files that name it.
pixi and other rattler-based clients use sharded repodata ([CEP 16](https://github.com/conda/ceps/blob/main/cep-0016.md)) where a channel offers it, as conda-forge does. Point them at the same channel URLs, for example `pixi init -c http://localhost:8080/conda/conda-forge`. The shard index and shards are cached as metadata when `cache_metadata` is on.
### CRAN (R)
Set the repository in R:
@ -391,6 +498,8 @@ local({
})
```
Source packages, Windows and macOS binaries (including the `bin/macosx/big-sur-arm64/` layout used by R 4.3 and later) are cached as artifacts. `PACKAGES` indexes and the `src/contrib/Meta/*.rds` files that `remotes::install_version` and pak read are cached as metadata.
### Julia
Set the Pkg server before starting Julia:
@ -447,6 +556,79 @@ Or pull images directly:
docker pull localhost:8080/library/nginx:latest
```
The registry is pull-only. BuildKit cache exports (`--cache-to type=registry`)
and other pushes need a separate registry.
SBOMs, signatures and attestations attached to an image are found through the
OCI 1.1 referrers API (`GET /v2/{name}/referrers/{digest}`). `oras discover`
and `notation verify` use it, and so does cosign for its newer signature
bundles. The proxy forwards these requests and caches the returned index with
the same TTL as other metadata, so they keep working offline. It does not
filter by `artifactType`; it returns the full list and the client filters it,
as the spec allows. Registries without the referrers API (GHCR, for example)
answer 404, and clients then look up the `sha256-<digest>` tag instead, which
the proxy serves like any other manifest. The proxy is pull-only, so attaching
new artifacts through it is not supported.
#### containerd (Kubernetes, k3s, nerdctl)
containerd mirrors send the original registry host in an `ns` query
parameter, so one mirror entry can serve Docker Hub and the registries
configured in `upstream.oci` whose URL has no path. Point containerd's CRI
plugin at a hosts directory in `/etc/containerd/config.toml` and restart
containerd. For containerd 2.x:
```toml
version = 3
[plugins."io.containerd.cri.v1.images".registry]
config_path = "/etc/containerd/certs.d"
```
For containerd 1.x:
```toml
version = 2
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"
```
Then create `/etc/containerd/certs.d/_default/hosts.toml`:
```toml
[host."http://proxy.example.com:8080"]
capabilities = ["pull", "resolve"]
```
To check that CRI picked up the directory, confirm the setting containerd
runs with and pull through CRI (on k3s use `k3s crictl`):
```bash
containerd config dump | grep config_path
crictl pull docker.io/library/nginx:latest
```
The proxy logs a `container manifest request` line for the pull and, when
`access_log.path` is set, an access-log entry. `ctr images pull --hosts-dir`
is no substitute for this check: it reads the directory itself and succeeds
even while CRI still has no `config_path`.
`docker.io` and the host of `upstream.oci_default` use the default registry;
the host of each `upstream.oci` URL uses that named registry, and so do the
hosts listed for it in `upstream.oci_mirrors`. Pulls for any other registry get
`404 NAME_UNKNOWN`, and containerd falls back to the registry itself. Existing
per-registry `hosts.toml` files that point at `/v2/upstream/{name}` with
`override_path = true` keep working for that upstream's own host and for
Docker Hub. If the upstream is a mirror of the registry the nodes pull from,
say an Artifactory remote of `ghcr.io`, list that registry in
`upstream.oci_mirrors` (`ghcr: ["ghcr.io"]`); otherwise those pulls get the
404 as well. See [docs/configuration.md](docs/configuration.md) for how
hosts are matched and which entry wins when two share a host. k3s generates
the hosts directory itself from `/etc/rancher/k3s/registries.yaml`; `mirrors:
{"*": {endpoint: ["http://proxy.example.com:8080"]}}` produces the same
`_default` entry.
### Helm
Configure each HTTP chart repository with a name, then add the matching proxy
@ -488,12 +670,31 @@ Configure APT to use the proxy in `/etc/apt/sources.list.d/proxy.list`:
deb http://localhost:8080/debian stable main contrib
```
Replace your existing sources.list entries, then:
Or in deb822 format, in `/etc/apt/sources.list.d/proxy.sources`:
Replace your existing sources.list entries (on newer releases also
`debian.sources` or `ubuntu.sources` in `/etc/apt/sources.list.d/`), then:
```bash
sudo apt update
```
There is no proxy key to install. The proxy serves the upstream's signed
`InRelease` and `Release` files unchanged, so APT verifies them with the
upstream archive's own key from the distribution's keyring package:
`debian-archive-keyring` on Debian, `ubuntu-keyring` on Ubuntu
(`/usr/share/keyrings/ubuntu-archive-keyring.gpg`). `Signed-By:` in a
`.sources` file, or `[signed-by=...]` in a `.list` line, restricts the source
to that key. For a third-party upstream, use the key its vendor publishes.
The upstream defaults to `http://deb.debian.org/debian`. To proxy a different APT repository (e.g. Ubuntu), set `upstream.debian` in the config file or `PROXY_UPSTREAM_DEBIAN` in the environment:
```yaml
@ -526,16 +727,20 @@ indexes unchanged. To keep APT from selecting a denied version, configure
### RPM / Yum / DNF
Configure yum/dnf to use the proxy in `/etc/yum.repos.d/proxy.repo`:
In the `[fedora]` section of your existing repository file under `/etc/yum.repos.d/`, replace the `metalink` or `mirrorlist` entry with:
SQLite is the default and works well for single-node deployments. For multi-node setups or if you prefer a managed database, switch to Postgres:
SQLite is the default and works well for single-node deployments, with the database file on a local disk or block-backed volume. For multi-node setups, when only network filesystem storage such as NFS is available, or if you prefer a managed database, switch to Postgres:
The mirror command accepts the same storage and database flags as `serve`. Already-cached artifacts are skipped.
The mirror command accepts the same storage and database flags as `serve`. Already-cached artifacts are skipped. If any package version fails to mirror, the command lists the errors and exits with status 1, so a CI step that warms the cache fails on a partial mirror.
With `scanning.enabled`, the mirror command refuses every package and container image so nothing is cached unscanned. Each one is reported as failed, so the command exits with status 1. Scanners fetch the staged bytes from the running server, which this command does not start. Use the mirror API below instead, since its jobs run inside the server and go through the pre-cache scan. `--dry-run` still lists what would be mirrored.
A mirror API is also available when the server is running:
Push package archives to a running proxy with `hosted.enabled`, list them and delete them. See [Hosted Packages](docs/configuration.md#hosted-packages).
| `POST /npm/-/npm/v1/security/*` | npm/pnpm/Yarn audit endpoints, passed through to upstream |
| `GET /npm/-/npm/v1/keys` | npm registry signing keys, passed through to upstream |
| `GET /npm/-/package/{name}/dist-tags` | `npm dist-tag ls`, answered from the packument the proxy serves (cooldown and pushed versions applied); changing tags is not supported |
| `GET /npm/-/npm/v1/attestations/{name}@{version}` | npm provenance attestations of a version, fetched from the package's registry and cached |
| `GET /npm/-/v1/search`, `GET /npm/-/ping` | `npm search` and `npm ping`, passed through to upstream uncached |
| `GET /api/mirror/{id}` | Get job status and progress |
| `DELETE /api/mirror/{id}` | Cancel a running job |
### Hosted API
Registered when `hosted.enabled` is set. Pushing and deleting require `Authorization: Bearer <hosted.token>`.
| Endpoint | Description |
|----------|-------------|
| `PUT /api/hosted/{ecosystem}/{name}/{version}` | Push the request body as the archive for a version (npm and composer) |
| `GET /api/hosted/{ecosystem}/{name}` | List the archives pushed for a package |
| `DELETE /api/hosted/{ecosystem}/{name}/{version}` | Delete a pushed version |
### Projects API
Registered when `projects.enabled` is set. Uploading and deleting require `Authorization: Bearer <projects.token>`. See [Project inventory](docs/configuration.md#project-inventory).
| Endpoint | Description |
|----------|-------------|
| `PUT /api/projects/{project}/sbom` | Replace a project's recorded package versions with the ones the SBOM in the body lists |
| `GET /api/projects` | List projects, or with `?purl=pkg:npm/lodash@4.17.20` the projects using a package or version |
| `GET /api/projects/{project}` | A project's recorded package versions |
| `DELETE /api/projects/{project}` | Delete a project |
### Cache API
Registered when `cache_delete` is set. It has no authentication of its own.
| Endpoint | Description |
|----------|-------------|
| `DELETE /api/cache/{ecosystem}/{name}/{version}` | Delete a version's cached files, or only `?filename=` |
### Enrichment API
The proxy provides REST endpoints for package metadata enrichment, vulnerability scanning, and outdated detection.
@ -1077,9 +1535,7 @@ Response:
{
"cached_artifacts": 142,
"total_size_bytes": 523456789,
"total_size": "499.2 MB",
"storage_url": "file:///path/to/cache/artifacts",
"database_path": "./cache/proxy.db"
"total_size": "499.2 MB"
}
```
@ -1111,6 +1567,7 @@ Response:
The proxy serves a web UI under `/ui`. No separate frontend build is needed -- templates and assets are embedded in the binary. `GET /` redirects to `/ui/`. The UI is mounted under its own prefix so a reverse proxy can apply different access rules to it than to the package endpoints (for example, requiring auth for `PathPrefix(/ui)` while leaving `/npm`, `/pypi` etc. open to build machines).
- **Dashboard** (`/ui/`) -- cache stats, popular packages, recently cached artifacts, and vulnerability overview.
- **Analytics** (`/ui/analytics`) -- accumulated download size as a ring broken down by ecosystem with the total in the middle, the cache size, artifact, package and version counts, a per-ecosystem table, the vulnerability overview, and a Runtime card mirroring every counter `/metrics` exposes. See [Analytics](#analytics).
- **Install guide** (`/ui/install`) -- per-ecosystem configuration instructions, so you don't have to look them up here.
- **Package browser** (`/ui/packages`) -- browse all cached packages with filtering by ecosystem and sorting by hits, size, name, or vulnerability count.
- **Search** (`/ui/search?q=...`) -- search cached packages by name.
@ -1118,6 +1575,7 @@ The proxy serves a web UI under `/ui`. No separate frontend build is needed -- t
- **Version detail** (`/ui/package/{ecosystem}/{name}/{version}`) -- per-version metadata, integrity hash, artifact cache status, and hit counts.
- **Source browser** (`/ui/package/{ecosystem}/{name}/{version}/browse`) -- browse files inside cached archives with syntax highlighting for text files and image previews.
- **Version diff** (`/ui/package/{ecosystem}/{name}/compare/{v1}...{v2}`) -- side-by-side diff of two cached versions showing added, removed, and changed files.
- **Projects** (`/ui/projects`) -- with the [project inventory](docs/configuration.md#project-inventory) enabled, the package versions each repository's uploaded SBOM lists, marked when outdated. Package pages then show which projects use each version.
## Monitoring
@ -1127,10 +1585,17 @@ The proxy exposes Prometheus metrics at `GET /metrics`. All metric names are pre
|--------|------|--------|-------------|
| `proxy_requests_total` | counter | `ecosystem`, `status` | Proxy responses by package ecosystem and HTTP status |
| `proxy_cache_hits_total` | counter | `ecosystem`, `kind` | Cache hits. `kind` is `artifact` or `metadata`. |
| `proxy_cache_misses_total` | counter | `ecosystem`, `kind` | Cache misses. `kind` is `artifact` or `metadata`; metadata misses are counted only while `cache_metadata` is on. |
| `proxy_cache_size_bytes` | gauge | | Total size of cached artifacts |
| `proxy_cached_artifacts_total` | gauge | | Number of cached artifacts |
| `proxy_metadata_cache_served_bytes_total` | counter | `ecosystem` | Metadata bytes served from the metadata cache. |
| `proxy_metadata_cache_size_bytes` | gauge | `ecosystem` | Size of cached metadata documents per ecosystem. |
| `proxy_metadata_cache_entries` | gauge | `ecosystem` | Number of cached metadata documents per ecosystem. |
| `proxy_artifacts_evicted_total` | counter | `reason`, `ecosystem` | Artifacts evicted by the size limit (`lru`) or by age (`retention`) |
| `proxy_metadata_evicted_total` | counter | `ecosystem` | Cached metadata documents evicted by the size limit |
| `proxy_metadata_expired_by_change_feed_total` | counter | `ecosystem` | Cached metadata documents set to revalidate because an upstream [change feed](docs/configuration.md#change-feeds) listed them |
| `proxy_fetch_lease_waits_total` | counter | `kind` | Cache misses (`artifact` or `metadata`) that waited for another instance sharing the database to fetch them |
| `proxy_upstream_fetch_duration_seconds` | histogram | `ecosystem` | Time spent fetching from upstream |
@ -1139,8 +1604,20 @@ The proxy exposes Prometheus metrics at `GET /metrics`. All metric names are pre
| `proxy_health_probe_failures_total` | counter | `step` | Storage health probe failures by failing step (`write`, `size`, `read`, `verify`, `delete`). |
| `proxy_circuit_breaker_state` | gauge | `registry` | Artifact-fetch circuit breaker state per upstream registry (0 closed, 2 open). Published once that registry's breaker has tripped. |
| `proxy_upstream_rate_limit_holds_total` | counter | `host` | Holds placed on an upstream host after it rate limited the proxy with HTTP 429, or 503 with a `Retry-After`. See `upstream_holds` under [Health Check](#health-check). |
| `proxy_ecosystem_downloaded_bytes` | gauge | `ecosystem` | Accumulated bytes served from cache: cache hits multiplied by the artifact size they served. |
| `proxy_ecosystem_artifact_downloads` | gauge | `ecosystem` | Accumulated artifact downloads served from cache. |
| `proxy_ecosystem_cache_size_bytes` | gauge | `ecosystem` | Size of cached artifacts per ecosystem. |
| `proxy_ecosystem_cached_artifacts` | gauge | `ecosystem` | Number of cached artifacts per ecosystem. |
| `proxy_ecosystem_packages` | gauge | `ecosystem` | Known packages per ecosystem. |
| `proxy_ecosystem_versions` | gauge | `ecosystem` | Known package versions per ecosystem. |
| `proxy_response_bytes_total` | counter | `ecosystem` | Response body bytes written to clients. Route-labelled, see the label caveat below. |
| `proxy_client_requests_total` | counter | `client` | Requests by client tool, from the User-Agent. |
Cache size and artifact count are refreshed every 60 seconds. Circuit breaker state is read from the fetcher on each scrape of `/metrics` and each `/health` request, so `proxy_circuit_breaker_trips_total` counts the trips visible between those reads — a breaker that opens and recovers entirely between two scrapes is not counted. The remaining metrics update on each request.
The `ecosystem` label on `proxy_requests_total` and `proxy_request_duration_seconds` is the mounted route a request arrived on, not the ecosystem recorded against the package it served: `/gem` reports as `rubygems`, `/go` as `golang`, `/composer` as `packagist`, `/apk` as `alpine` and `/v2` as `oci`. Anything outside a package route -- the UI, `/health`, `/metrics`, `/stats` -- reports as `other`, and so did `/apk`, `/helm`, `/homebrew`, `/generic` and `/swift` before they were listed; traffic on those five routes now appears under its own name instead.
Cache size, artifact count and the per-ecosystem gauges are refreshed every 60 seconds, from a single pass over the database. Circuit breaker state is read from the fetcher on each scrape of `/metrics` and each `/health` request, so `proxy_circuit_breaker_trips_total` counts the trips visible between those reads — a breaker that opens and recovers entirely between two scrapes is not counted. The remaining metrics update on each request.
The breaker metrics carry one series per upstream host, but only for hosts whose breaker has tripped at least once since startup. A breaker is created per host the proxy fetches artifacts from, and for some ecosystems that host comes from upstream metadata rather than from configuration (composer takes it from a package's `dist.url`, helm from the chart URLs in `index.yaml`), so publishing every host would let upstream content grow the series count for the lifetime of the process. Once a host has tripped it keeps reporting, so a recovery still shows up as a transition to 0 rather than as a series that vanishes. `/health` is not a persistent time series and lists every breaker, tripped or not.
@ -1148,6 +1625,82 @@ The `registry` label is the host of the URL the artifact was fetched from. Becau
Alert on `proxy_circuit_breaker_state == 2` sustained for more than a few minutes: while a breaker is open, artifact downloads for that upstream fail with HTTP 502 on every cache miss, and only a single probe request per backoff interval reaches the upstream. Cached artifacts keep serving, and so does metadata for the same ecosystem (metadata does not go through the circuit breaker), so installs fail in a way that looks like a partial upstream outage.
#### Accumulated download size
`proxy_ecosystem_downloaded_bytes` is, for every cached artifact, the number of times it was served multiplied by its size. It answers "how much traffic has this proxy actually carried", which is the number that matters when sizing egress or justifying the cache. Two properties are worth knowing before alerting on it.
**It counts cache hits, not upstream fetches.** The request that first pulls an artifact through the proxy is a miss and is not counted; only later hits are. So the accumulated total is also the upstream bandwidth the cache has saved, not the total bytes the proxy has ever sent.
**Eviction removes history.** Evicting an artifact clears its size, so its past hits drop out of the total. That is why these are gauges rather than counters, and why the figure can step downwards. Chart them with `max_over_time` rather than `increase`, and read a drop after an eviction sweep as expected rather than as data loss.
Nothing at the schema level ties `artifacts.version_purl` to a version row, so a cached artifact can end up with no ecosystem to attribute it to. Those are reported under the ecosystem `unattributed` rather than dropped, which keeps the per-ecosystem figures adding up to `proxy_cache_size_bytes` and `proxy_cached_artifacts_total`. A non-zero `unattributed` means the database holds artifact rows whose version or package rows have gone missing.
The `ecosystem` label on these six is taken from the package record and normalized, so aliases collapse: a database carrying both `gem` and `rubygems` rows -- the proxy writes the former, git-pkgs the latter -- reports one `rubygems` series with the two summed.
### Analytics
`/ui/analytics` reports the accumulated download size as a ring broken down by ecosystem, the cache figures from the dashboard, a per-ecosystem table, the vulnerability overview, and a **Runtime** card covering every remaining metric `/metrics` exposes.
The ring shows at most six slices, because part-to-whole stops being readable past that. When more ecosystems are active the smallest are folded into a single "Other" slice; the table below lists every one of them, so nothing is hidden, only summarised.
#### No history is kept
The proxy stores no time series. The page reads the database and the in-process metric registry at request time and reports current state; there is nowhere for it to read yesterday's figures from, and nothing is written for tomorrow. That splits the figures in two, and the page says which is which.
**Database-derived figures survive a restart.** Download volume, cache size and the package, version and artifact counts come from the `artifacts`, `packages` and `versions` tables, so they are as durable as the database.
**Registry-derived figures do not.** Everything in the Runtime card -- request counts and latencies, cache hit rate, upstream and storage errors, circuit breaker state, scan results -- lives only in this process's Prometheus registry and starts from zero on restart. A small number there next to a large one above just means the proxy started recently.
For history, trends and alerting, scrape `/metrics` with Prometheus. That is the intended split: the UI answers "what is true now", Prometheus answers "what happened".
The same figures are available as JSON from `GET /stats`, which reports `downloaded_bytes`, `downloads` and an `ecosystems` array carrying the per-ecosystem breakdown, served from the same 60-second snapshot the page and the gauges read. When the aggregation fails with no snapshot to fall back on, the response carries `stats_unavailable: true` rather than passing zeros off as a count -- the endpoint keeps answering with the artifact count and cache size either way.
#### Artifact and metadata caches
The proxy keeps two caches. Artifacts (tarballs, wheels, crates and so on) live in storage and are what `proxy_cache_size_bytes`, `proxy_ecosystem_downloaded_bytes` and the other artifact figures describe. Registry metadata is cached separately when `cache_metadata` is on. It is stored persistently in the `metadata_cache` table rather than held in memory.
The `kind` label on `proxy_cache_hits_total` and `proxy_cache_misses_total` keeps the two apart, so an artifact hit rate is `sum(rate(proxy_cache_hits_total{kind="artifact"}[5m]))` over the same expression with misses added. Metadata has its own size and entry gauges plus `proxy_metadata_cache_served_bytes_total`. That one is a process counter rather than a database total, because metadata hits are not recorded per row. On `/ui/analytics` metadata appears in its own card and in the Runtime card's metadata lookups, never mixed into the artifact figures. The card and the two gauges read one 60-second snapshot, like the artifact figures. The bundled Grafana dashboard's hit ratio covers artifacts only, while its hits and misses panel shows each kind as its own series.
#### Three ecosystem label sets
`ecosystem` means three slightly different things across `/metrics`, and queries that join across them need to know which.
**From the package record, normalized.** The six `proxy_ecosystem_*` gauges, `proxy_cache_hits_total`, `proxy_cache_misses_total`, the three `proxy_metadata_cache_*` metrics, `proxy_artifacts_evicted_total`, `proxy_metadata_evicted_total`, `proxy_metadata_expired_by_change_feed_total`, `proxy_integrity_failures_total` and the scan metrics. Aliases collapse here: `gem` reads as `rubygems`, `composer` as `packagist`, `go` as `golang`.
**From the request path.** `proxy_requests_total`, `proxy_request_duration_seconds` and `proxy_response_bytes_total`. The names mostly coincide with the normalized ones -- these also report `rubygems`, `packagist` and `golang` -- but the Debian route reports `debian` where the package record says `deb`, and any path that is not a package endpoint reports `other`, which corresponds to no ecosystem at all.
**From the handler's own name.** `proxy_upstream_fetch_duration_seconds` and `proxy_upstream_errors_total`, which report `composer`, `gem` and `go` where the other two sets report `packagist`, `rubygems` and `golang`. These are published series and are deliberately left as they are; renaming them would break existing queries and alerts.
### Request sources
Package managers do not say who invoked them. A request from `pip` or `go` carries a `Host`, an `Accept` and a `User-Agent` -- no `Referer`, no originating URL, nothing naming a repository, pipeline or job. Whatever identity you want has to come from something on the wire, so the proxy attributes requests by the two things always present.
**Address** -- the TCP peer, or the leftmost `X-Forwarded-For` entry when `trust_forwarded_for` is enabled. Enable that only behind a load balancer or ingress that sets the header; any client can send it, so in front of one it lets a caller forge its own attribution and, by cycling synthetic addresses, fill the table and push every genuine caller into the overflow row. The totals stay correct; the attribution is what is lost.
**Client** -- the tool, taken from the leading User-Agent token: `pip`, `npm`, `go`, `docker`, `apt`, `curl` and so on. Anything unrecognised reports as `other`.
Set `ui_request_sources: true` and both appear on `/ui/analytics` under **Runtime -> Request sources**, as a table of the busiest callers by bytes downloaded plus a per-tool breakdown. The table is in-memory and process-lifetime, like the rest of that card. It tracks 200 callers, evicting the least recently seen once full, and summarises everything it is not showing individually -- both evicted callers and those ranked below the display limit -- in a single "other callers" row, so the rows always add up to the totals above them.
**The flag defaults to off because the page is not authenticated.** `/ui` carries no auth of its own -- it is mounted under its own prefix so a reverse proxy *can* gate it separately, as [Behind a Reverse Proxy](#behind-a-reverse-proxy) describes, but nothing makes you -- and until now it exposed only package data. The sources table changes what is on offer: anyone who can reach the proxy can read the addresses of your build fleet, which tool each runs, and how much each pulled. Turn it on once `/ui` is gated, or leave it off and read the same detail from the access log.
**What this can and cannot tell you.** How much an address gives you depends entirely on your network. A fleet of build machines with stable addresses attributes cleanly. Containerised CI usually does not: with Docker or Kubernetes executors every job gets an ephemeral address, and egress is commonly NAT'd behind one gateway, so you get runner-node or gateway granularity, not per-project. If you need per-project attribution the caller has to send something naming itself -- a basic-auth username, or a per-project base URL -- which the proxy does not currently read. Say so and it can be added.
**Why addresses are not Prometheus labels.** Client tool names are exported as `proxy_client_requests_total{client}` because they come from a closed set. Addresses are not exported at all: the caller set is unbounded and outside the proxy's control, and every new address would create a time series that lives forever in your TSDB. The same goes for anything job-scoped -- a pipeline ID must never become a label. Per-address and per-request detail belongs in the access log, which records `remote_ip`, `user_agent`, `client`, `ecosystem` and `bytes` on every line as JSONL, ready for `jq`, Loki or whatever you ship logs to.
### Failure details
The failure counts under **Runtime** on `/ui/analytics` -- upstream errors, storage errors, integrity failures, health probe failures and scan errors -- say how often something failed, not what. Set `ui_failure_details: true` and each count gets a **Latest** list underneath it, newest first: the package URL and filename, the upstream URL or storage path, the error, and how long ago it happened.
The list keeps the last 20 failures per row, in memory only, so it starts empty when the proxy starts, like the counts above it. Credentials, query strings and fragments are stripped from every URL before it is kept, including URLs inside error messages, since that is where registry tokens and signed-URL signatures live.
The flag defaults to off for the same reason as `ui_request_sources`: package names and upstream hosts say what your users depend on, and `/ui` is not authenticated. Package names and URLs are never Prometheus labels; the same detail is in the proxy's log.
### Grafana dashboard
A ready-made dashboard lives at [`deploy/grafana/git-pkgs-proxy.json`](deploy/grafana/git-pkgs-proxy.json). Import it via **Dashboards -> New -> Import** and pick your Prometheus data source when prompted; it has no hardcoded data source UID.
It carries three ecosystem filters rather than one, because `ecosystem` means three different things across `/metrics` -- see the label sets above. **Ecosystem** filters the database-derived gauges, **Route** the request-path counters, and **Upstream** the two upstream fetch metrics. All three are query variables, so they populate from whatever labels your proxy is actually reporting; a panel is on the one its metric belongs to, and the panel descriptions say which.
### Health Check
`/health` returns a structured JSON report of subsystem health. HTTP 200 if all checks pass; 503 if any fail.
@ -1162,6 +1715,9 @@ Alert on `proxy_circuit_breaker_state == 2` sustained for more than a few minute
"circuit_breakers": {
"registry.npmjs.org": "closed",
"static.crates.io": "open"
},
"upstream_holds": {
"repo1.maven.org": "2026-10-14T11:15:29Z"
}
}
```
@ -1170,6 +1726,8 @@ Failing checks include an `"error"` field. Storage failures also include a `"ste
`circuit_breakers` reports the state of each upstream's artifact-fetch circuit breaker (`"open"` or `"closed"`), keyed by upstream host — or by the `hostless-url-<digest>` placeholder described under [Monitoring](#monitoring) where the fetch URL has no host to read. The key is omitted until the proxy has fetched an artifact from at least one upstream, and a host appears only once a breaker has been created for it. Breakers trip after repeated upstream failures and retry the upstream after an exponential backoff. While one is open, artifact downloads for that host return HTTP 502 on a cache miss without contacting the upstream; already-cached artifacts are still served from storage, since the cache is checked before the fetcher. A breaker is reported as `"open"` throughout its backoff, including the half-open window in which it admits one probe request to test recovery. Breaker state is per process and in memory, so a restart clears it, but a restart is not needed for recovery: the backoff keeps retrying for as long as the breaker is open, so it closes on its own once the upstream serves again.
`upstream_holds` lists upstream hosts that answered the proxy with HTTP 429, or 503 with a `Retry-After`, and the time until which the proxy sends them nothing. Every request to a held host, metadata and artifacts alike, fails locally for the duration the upstream asked for (capped at one hour), or for one minute doubling up to fifteen when the upstream gave no `Retry-After`. Cached metadata is served stale during a hold, and anything uncached returns HTTP 503 with the remaining `Retry-After` so a client can pace itself too. Metadata requests do not go through the circuit breaker, so without the hold a rate-limited upstream would keep receiving one request per cache miss, which is exactly the traffic it is asking the proxy to stop sending. The first non-limited response after a hold ends clears it, and `proxy_upstream_rate_limit_holds_total` counts the holds placed. Holds are recorded in the database, so instances sharing one honour each other's holds and list them here too. The key is omitted while no host is held.
An open breaker does **not** set `status` to `"error"` or change the HTTP status code: it reports a specific upstream refusing to serve, not this proxy being unfit to receive traffic, and failing the readiness probe over one unhealthy upstream would pull the pod out of rotation for every other ecosystem too. Use `proxy_circuit_breaker_state` for alerting on it.
Storage probe results are cached for `health.storage_probe_interval` (default 30s) to bound the cost of probing remote backends. A probe holds an internal mutex for up to 10 seconds (the hardcoded per-probe timeout), so `/health` is intended as a Kubernetes **readiness** probe rather than a liveness probe — a slow S3 round-trip should pull the pod from rotation, not restart it.
@ -1214,13 +1772,23 @@ sudo systemctl start proxy
### Docker
A Dockerfile is included in the repo. Build and run:
Images for linux/amd64 and linux/arm64 are published to `ghcr.io/git-pkgs/proxy` for each release, tagged with the version and `latest`:
```bash
docker run -p 8080:8080 -v proxy-data:/data ghcr.io/git-pkgs/proxy:latest
```
To build the image from the repo instead:
```bash
docker build -t proxy .
docker run -p 8080:8080 -v proxy-data:/data proxy
```
The image has a health check that requests `/health` on port 8080. It checks the database and storage, so a container reports unhealthy when either fails. If you change the port with `PROXY_LISTEN`, override it with `--health-cmd`.
With `--read-only`, also pass `--tmpfs /tmp`: the proxy writes some uploads to a temporary file before reading them.
With Postgres and S3:
```bash
@ -1310,6 +1878,12 @@ cache/artifacts/
└── nginx-1.24.0-1.fc39.x86_64.rpm
```
To delete one version's cached files, so the next request fetches them again, enable `cache_delete` and use the Delete buttons on the version page in the web UI, or the [cache API](docs/configuration.md#cache-delete-api):
