`npm search` requests /-/v1/search?text=... . The path has no "/-/"
after the leading slash is trimmed, so it fell through to the package
metadata handler: upstream got /-/v1%2Fsearch without the query
string, and the answer was cached under "-/v1/search", so every later
search was served the first one's results. Search and /-/ping (used
by `npm ping`, which lost its query string the same way) are now
relayed to upstream uncached with their query strings, like the
signing-keys endpoint.
Packuments share one metadata cache row whatever format they were
fetched in. Without cooldown the proxy asks for the abbreviated
format, which has no "time" map. After cooldown was turned on, a
still-fresh abbreviated packument kept being served from cache, and
used as the stale copy when upstream failed, so cooldown filtered
nothing and the tarball check let recent versions through. When full
metadata is required (cooldown or npm_full_metadata on), packument
lookups now pass a validator that rejects an abbreviated document
(top-level "modified", no "time"), so it counts as a cache miss, is
refetched in full and is never used as a stale fallback. Behaviour
without cooldown is unchanged.
fetchMetadataDocument takes the validate callback that
fetchOrCacheMetadata already has; other callers pass nil.
Each request for a General registry tarball whose hash differed from
the loaded name map started a goroutine that downloaded the whole
registry from upstream again, with no limit on concurrent loads and no
deadline. Alternating between two valid snapshots made every request
cost a full download and parse, and left the map at whichever snapshot
came last, older ones included.
The refresh now only loads the hash /registries currently lists, read
through the metadata cache, runs one load at a time and skips while one
is running, and stops after five minutes. Registry.toml is read up to
64 MB.
A failed .info fetch counted as an unknown time, so a rate-limited or
flaky upstream let versions in cooldown through. It now fails the list
or download with 502; only a version upstream doesn't know, or whose
.info has no time, is let through.
Times are stored under the escaped module path, which keeps modules
that differ only in case apart. Modules with uppercase letters were not
stored before and fetched every .info again on each list request. Only
canonical versions are stored, since a query such as a branch name can
resolve to a later commit.
A .mod or .zip whose version doesn't unescape is answered with 404
instead of being passed upstream unchecked.
With cooldown on, @v/list leaves out versions whose .info time is
inside the window, and @latest falls back to the newest release out of
cooldown when upstream's answer is too new. Each listed version's
.info is read once and its time is stored in the versions table, as
npm does, except for modules whose path has uppercase letters, since
golang PURLs are lowercased.
A .info, .mod or .zip for a version in cooldown, and a query such as a
branch name that resolves to one, gets 403. The go command moves on to
the next GOPROXY entry after 404 or 410, so a 404 would let
GOPROXY=proxy,direct fetch the version directly.
Go has no publish time: .info carries the commit time of the revision,
so a version tagged on an old commit counts as old. The docs say so.
Part of #132
The shard index (repodata_shards.msgpack.zst) and the shards of CEP 16
fell through to the uncached passthrough, so clients that use them, such
as pixi, got no cooldown filtering and no offline copy.
Without cooldown, the index and shards are now served through the
metadata cache. With cooldown, shards have to be filtered, but clients
cache them by the hash in the index and never fetch a hash twice. The
proxy rewrites the index instead: each hash is XORed with a mask derived
from channel, subdir and the current hour, and shards_base_url points at
cooldown-shards/{hour}/. That route undoes the mask, fetches the
upstream shard and drops records still in cooldown. The unmasked shard
paths answer 404 while cooldown is on.
When the index puts packages or shards somewhere the proxy doesn't
route, or has an unknown version, the proxy answers 404 and clients fall
back to repodata.json, which is filtered as before.
Fixes#529
Denylist entries now also match Composer versions with Packagist's
leading v, OCI repositories written as a path, Swift registry packages
in either form (also when streaming), Go versions with escaped uppercase
letters, and gems whose names contain a dash followed by a digit. OCI
manifests are checked by tag and digest, and denied blobs and Swift
archives answer 403 instead of 500 or 502.
Package files outside Hashed/, such as All/{name}-{version}.pkg from
repositories built without pkg repo --hash and the bootstrap
Latest/pkg.pkg, were relayed without any checks. They are still passed
through uncached, since their names can be reused for rebuilds, but a
versioned name is now checked against the denylist first, and while a
scanner covers freebsd they are refused, because only cached artifacts
can be scanned.
Denylist entries were compared as plain PURL strings, but handlers build
PURLs from download URLs that spell packages differently, so some denied
versions were still served. Entries and checked PURLs now go through one
per-ecosystem comparison form: PyPI names follow PEP 503 (case and runs of
-, _ and . are equivalent), NuGet ids and versions ignore case, a gem
version also matches its platform builds, and a conda entry matches the
package in every channel unless it carries a channel qualifier. Cache keys
are unchanged.
Conda download caching keyed artifacts on channel, name, version and
filename only, but the same filename can hold different bytes in each
subdir (python_abi-3.11-4_cp311.conda in linux-64 and osx-arm64, for
example). A client on one platform could be served another platform's
package. The subdir is now part of the cached filename, so each one gets
its own entry. Entries cached under the old key are no longer used and
are re-fetched on first request.
Only files with a known artifact or checksum extension went through the
download path; anything else, such as the .exe binaries protoc and gRPC
plugins publish or .hpi and .so files, was relayed from upstream as is.
Those files skipped the denylist, cooldown, scanning and the cache. Any
file in a version directory named {artifact}-{version}..., the way Maven
names what it publishes (timestamped for SNAPSHOTs), now takes the
download path. Directory listings and catalogs are still proxied.
proxy mirror and mirror API jobs resolved download URLs with a bare
resolver, so npm, cargo, gem, golang, hex, pub, maven and nuget packages
came from the public registries whatever upstream.* says. The resolver
now takes the configured upstreams as base URLs.
npm packages in upstream.npm_routes are fetched from their route and
stored under the route's filename prefix, the way the npm handler does,
so a public package can't stand in for a routed one and clients are
served the mirrored tarball.
Refs #414
Catalogues (meta.conf, packagesite and data) carry their signature
inside the archive and go through the metadata cache byte for byte.
Packages under Hashed/ have the content hash in their name, so they go
to the artifact cache as pkg:bsd/freebsd/{name}@{version}. Other files,
including the bootstrap files under Latest/, are passed through.
pkg.freebsd.org is the built-in freebsd repository; more mirrors go
under upstream.freebsd.
Closes#536
Serving cached metadata when the upstream refuses is documented for
generic upstreams, and GitHub rate limits are 403s, so the fallback
stays as it is. #503 handles npm's definite answers. What is left here
is the stale warning for Composer, Chef and hosted npm metadata, which
went through the rewrite cache without it.
* Serve Git repositories from a local cache
Named upstream.git hosts are served read-only over Git's smart HTTP
protocol at /git/{name}/{repo}, so git clone and git fetch work against
the proxy through an insteadOf rewrite. Each repository is a bare clone
at git.cache_dir/{name}-{hash of the upstream URL}/{repo}.git, served
by git upload-pack.
Every request for refs checks the upstream's branches, tags and HEAD
with a protocol v2 ls-refs limited to those prefixes, and fetches only
when they changed, atomically and with --prune. Concurrent requests
share one refresh. Only branches and tags are cached, and HEAD follows
the upstream's default branch.
A commit a client asks for by ID that the cache lacks, such as a pull
request head, is fetched and kept under a hidden ref until a branch or
tag includes it or for 7 days. One the upstream cannot provide is not
asked for again for a minute, or an hour when it is larger than
git.max_repo_size leaves. This is off for upstreams upstream.auth has
credentials for, since GitHub serves commits from any repository in a
fork network.
git never reaches an upstream itself. Its fetches go through a loopback
relay that forwards with the proxy's upstream client, so address
checks, upstream.auth and the access log apply, and git never sees a
credential. Redirects are not followed, and git runs with an
environment of its own.
The cached copy is served while the upstream fails or is slower than
git.refresh_wait, until git.stale_max_age has passed since the last
successful check. Refusals and missing repositories are never answered
from the cache.
git.max_repo_size cuts a transfer off as it streams, and every fetched
pack is kept as a pack, so a cut-off leaves nothing behind. A
repository that can no longer be refreshed within the limit is cloned
again with only its branches and tags. git.max_size evicts the least
recently used repositories, never one in use. At startup only the
directory of a repointed upstream's old URL is removed, and Close waits
for refreshes in progress.
Hosts with upstream.auth credentials need a repos allowlist, and a
repository outside it answers 404.
Refs #403
* Document setting the Git rewrite through environment variables
The Git section of the configuration docs now explains what insteadOf
does, with the URL a clone fetches from, and shows the same rewrites set
through GIT_CONFIG_COUNT, GIT_CONFIG_KEY_n and GIT_CONFIG_VALUE_n for
CI. Package managers that run git for Git dependencies inherit those
variables, so their fetches are cached too; npm's GitHub tarball
downloads do not go through git and are not.
Refs #403
* Note that upstream.auth credentials reach every route to a host
Credentials are matched by URL, so a token configured for a Git upstream
is also sent by a generic upstream at the same URL, which forwards any
path and checks no repos allowlist. With a token for a private GitHub
org, /generic/github/acme/private.git/info/refs would list that
repository's refs to anyone who can reach the proxy.
* Cap concurrent upload-pack processes and request body size
Every POST git-upload-pack started a pack-objects with git's defaults,
a delta-search thread per CPU and unbounded window memory, and nothing
limited how many ran at once, so a burst of CI clones of one large
repository could run the proxy out of memory. At most two upload-pack
processes per CPU (GOMAXPROCS, so a container's limit) now build packs
at once, and further requests wait for a slot while the client is
connected. ls-refs builds no pack and does not wait. Serving runs with
pack.threads=1 and pack.windowMemory=64m, and every git process with
core.bigFileThreshold=8m, so index-pack streams large files instead of
holding them in memory.
A request body was streamed to upload-pack unbounded, so a small gzip
body could feed it an arbitrarily long negotiation for as long as the
request may run. Bodies are now limited to 100MB, both as received and
decoded, and one past the limit fails with an error instead of being
cut short quietly.
* Apply refresh_wait to the ref check only
refresh_wait cut off the whole refresh, fetch included, so after a push
with a large pack a client got the old refs once it passed, though the
upstream was fine and the docs say a clone right after a push sees the
new commit. It now bounds only the check of the upstream's branches and
tags. Once the check finds they changed, requests wait for the fetch;
one that fails, or stalls and is abandoned by http.lowSpeedTime, still
falls back to the cached copy within stale_max_age.
* Lock cache_dir against a second process
The cache clears its temporary directory at startup, so a second
process given the same cache_dir, such as a second pod on a shared
volume, would delete the first one's clones in progress, and each
process's eviction knows only its own readers. New now locks
{cache_dir}/.gitcache-lock before touching anything and holds it until
Close: flock on Unix, a handle opened without sharing on Windows. The
OS releases it when the process exits, so a crashed process can
restart. A second process fails to start with an error naming the
directory.
* Evict again once repositories in use are released
Eviction skips repositories a request or refresh is using, and only the
release of one too large to refresh ran it again, so a cache filled past
max_size while its repositories were in use stayed over it once they
went idle, until the next fetch. Eviction now records when it is left
over max_size, and the next release runs it again.
* Cap git.max_repo_size at git.max_size
A repository larger than max_size could still be cloned, up to
max_repo_size, which defaults to 5GB whatever max_size is. Once released
it was evicted to bring the cache back within max_size, and the next
request cloned it again. max_size now caps max_repo_size, so such a
repository is refused for an hour like any other too large one, with an
error that names both limits.
* Count fetches in progress toward git.max_size
Fetches enforced only max_repo_size, and eviction ran after each one,
so up to 16 concurrent fetches could take the cache past max_size by as
much as each may download. Each fetch now reserves space within
max_size as the relay receives its bytes, a chunk at a time, evicting
least recently used idle repositories to make room. When the fetch
ends, the repository's measured size replaces the reservation under the
cache lock, so its bytes are neither counted twice nor missed.
When evicting every idle repository would not make room, none is
evicted for the fetch, and it is cut off with ErrCacheFull rather than
ErrTooLarge, so the repository is not refused for an hour: a clone
answers 507, and a refresh serves the cached copy as when the upstream
fails. A fetch that runs out of room while another has just been cut
off waits, up to a minute, for that one's space, so two fetches that do
not fit together do not both fail. A fetch that was cut off never
waits, so no two wait on each other.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
Composer names are case-insensitive, so acme/private and acme/Private
share the version record and its integrity but get different filenames.
A second push under the other case was accepted and broke the integrity
check of the first archive. For ecosystems with one file per version,
any pushed file of the version now refuses the push, both before the
upload and atomically when recording it.
- Record a fetched artifact with its package and version in one
transaction that refuses to replace a pushed artifact, so a fetch that
finishes after a push keeps both the pushed file and its version's
integrity. Pushes and cache fills share this write, including the
compare-and-swap on the storage path that queues every replaced object.
- Don't open the stored file for ecosystems without a manifest.
- Derive multi-file versions from the file being named in the path.
- Validate DELETE paths like PUT paths, answering 400 for malformed ones.
- Parse ?sha256= with integrity.ParseHex.
- Record the package, version and artifact of a push in one transaction
that only inserts the artifact if no pushed copy exists. Of two
concurrent pushes of the same file exactly one wins, and the loser's
version integrity is rolled back with it.
- Check for an existing file before reading the request body, and send
Expect: 100-continue from proxy hosted push, so a refused re-push isn't
uploaded first.
- Leave an existing version record alone for multi-file ecosystems, so a
generic push doesn't reset a version cached from upstream.generic.
- Reserve the upstream.generic name "hosted" only while hosting is
enabled, and only intercept /generic/hosted/ then.
- Accept a file argument in proxy hosted delete for generic only.
- Mark multi-file ecosystems explicitly, and give parse a min/max range.
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
With cooldown or a version denylist, /pkg/<version> reads the filtered
packument through handleFilteredNPMVersionMetadata, which dropped the
stale flag.
The npm handler logged a debug line for every version whose tarball URL
it rewrote. Packages with hundreds of versions produced hundreds of lines
per packument, and with cooldown enabled, where the rewrite cache is
bypassed, again on every request. It now logs one line per document with
the number of rewritten URLs.
* Serve additional Maven repositories at /maven/{name}/
upstream.maven_repositories maps names to further Maven repositories,
each served at /maven/{name}/ from its own upstream, like
debian_repositories does for APT. upstream.maven stays the default at
/maven/ with unchanged cache keys and its Gradle Plugin Portal fallback;
named repositories don't fall back to the portal.
Named repositories cache their files under {name}/{filename} and hash
their metadata keys from name, upstream URL and path, so the same
coordinates in two repositories, which may hold different bytes, never
share a cache entry.
A repository name is matched before the default repository and would
hide group IDs starting with it, so names of two letters or less and
common top-level domains (com, org, net, io, ...) are refused at config
load. The documentation asks for names that can't start a group ID.
Fixes#295
* Refuse every Maven Central group root as a repository name
The hand-written list missed real roots such as tools (Jackson 3),
software (AWS SDK), jakarta, javax and junit, so a repository named after
one would have hidden those groups in the default repository. The name
check now uses the first path segments of all group IDs on Maven Central,
plus Google's androidx and android, embedded from the repository's
directory listing.
Names are compared case-sensitively, since routing is, so "ORG" is no
longer refused for hiding "org".
A named repository's files are now cached under its name and a hash of
its upstream URL, so pointing it at a different URL stops serving what
the old one sent.
* Refuse Gradle Plugin Portal roots and unroutable repository names
- The default repository falls back to the Gradle Plugin Portal, so its
group roots (gradle, nebula, ...) would be hidden by a repository of
the same name too. The embedded list now holds the roots of Maven
Central and the Plugin Portal.
- Repository names are limited to letters, digits, '.', '_' and '-', so
a name can't contain characters such as '?', '#' or spaces that never
reach the handler as configured.
- Named repositories are built once in WithRepositories, with their
cache prefix, instead of on every request.
Module zips are stored under the module's PURL, which lowercases golang
names, and under the filename {last element}@{version}.zip. So
github.com/Sirupsen/logrus and github.com/sirupsen/logrus shared one
cache entry although upstream serves different zips, and the second one
requested failed go's checksum check.
A module path with capitals now gets a hash of its exact spelling in the
filename. All-lowercase paths keep the plain name, so existing cache
entries stay valid.
A capitalised module cached before this change sits under its lowercase
twin's plain filename. When a cached zip's upstream URL names a different
module path, it is discarded and fetched again. Only the module path is
compared, so a cache filled from another upstream.go host is kept.
Fixes#466
With cache_metadata on, a failed upstream metadata fetch fell back to the
cached copy for any error. A 404 (package removed) and a 401/403 (the
proxy's credentials rejected) are definite answers, though, and serving
the old version list in their place kept them from reaching the client.
They now pass through: 404 as 404, 401/403 as 502 like other upstream
failures. Timeouts, connection errors, 5xx and 429 still fall back.
The npm, Composer and Chef handlers serve metadata through the rewrite
cache, which dropped the stale flag, so a fallback looked like a fresh
200. They now set the same Warning: 110 header as other metadata.
Fixes#499
* Separate artifact and metadata cache metrics on /metrics and the analytics page
* Serve metadata cache stats from a shared snapshot and split Grafana cache panels by kind
* Abort the response when a cached artifact fails its integrity check
* Read the seeded artifact's storage path from the database in the integrity test
* max_size now counts cached metadata as well as artifacts, and eviction drops whichever was used least recently, so metadata no longer piles up for good (#461). Metadata rows record when they were last served, an artifact not yet served counts from its download, and Arch stores a database blob again if eviction removed it under a refreshed row.
* Shorter comments on the metadata last-access migration.
* Show evicted metadata on the analytics page next to evicted artifacts, and cover the new counter in the metric surface test.
* Evicting metadata now drops the row and queues its path in one transaction, so a crash before the blob is deleted leaves it to reclaim instead of orphaning it.
* The last-access backfill now runs in the background in batches of 1000 instead of inside the migration, resuming after a restart, and until a row is backfilled eviction orders it by its write time.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
* Pub downloads without .tar.gz fetched the HTML version page from pub.dev and cached it as the archive, so one request broke that version for everyone after; they now get a 404, and the upstream URL is built from the escaped name and version so an encoded ? or # can't become a query or fragment.
* Pub archives that older versions cached from a path without .tar.gz are now discarded and fetched again, so versions that were already broken heal on the next download.
Every Composer archive download fetched the package's p2 metadata straight
from upstream to find the archive's URL, before the archive cache was even
checked. A cached archive cost one or two uncached Packagist requests per
package and install, and failed with a 502 while Packagist was down.
With cache_metadata on, downloads now look the dist up in the metadata the
proxy already caches for Composer, under the keys handlePackageMetadata uses.
Within the TTL that needs no upstream request, and when upstream fails the
stale copy still resolves the archive. A version or file the cached
documents do not list, such as one published since they were cached, is
looked up upstream as before.
When upstream answered a revalidation with 304, the proxy read the cached
document back from storage and then stored it again: a full upload and a
SHA-256 over identical bytes, for every popular package once per TTL.
A 304 now only moves the cache row's fetched_at on, keeping its digest.
The update is conditional on the row still recording the digest of the
document that was read, so a concurrent fetch that replaced it is not
overwritten; in that case, and for rows written before digests were
recorded, the document is stored as before.
The metadata route caches a package's packument in the abbreviated
install format, unless cooldown or npm_full_metadata needs the full one.
Tarball downloads read the packument from the same cache row but asked
for application/json. Once the row was due for revalidation, a tarball
the proxy had not cached yet fetched the full packument, which did not
match the abbreviated document's ETag, and stored it in its place. Until
the next refresh every client was then served the full document,
typically several times the size.
Both now use packumentAccept. Downloads read only dist.tarball,
dist.integrity and dist.shasum, which the abbreviated format carries; the
publish times the cooldown needs come with the full format it selects.
* Add mirror.schedules, a list of image tags the server re-mirrors at a fixed interval. Digests are rejected since they never move.
* Run each schedule through the normal mirror, one run after the other. The first run starts at a random point within ten minutes and later runs get ±10 % jitter, so several instances don't hit the registry at once.
* Start the schedules with the server and let shutdown wait for a running refresh before storage and database are closed. Docs included.
* Check cached layers during a mirror without opening them, so a refresh no longer counts as a cache hit and doesn't keep scheduled images at the top of LRU eviction.
A response arriving during a hold was sent before it began, so a late
success no longer releases the host; only one after the hold has ended
does.
Artifact fetches check RateLimitGate.Held before calling the fetcher, so
held cache misses fail with the HeldError without counting as failures
in the host's circuit breaker.
Every handler that answers an upstream fetch error with 502 now answers
a held request with 503 and Retry-After instead.
OnHold and RecordUpstreamRateLimitHold drop the unused duration.
- Keep a gzip copy of each rewrite cache entry when it is smaller, made once
per rewrite and counted against metadata_rewrite_cache_size
- Serve it with Content-Encoding: gzip to clients that accept gzip, with an
ETag of its own ("<sha256>-gzip") and Vary: Accept-Encoding
- Rewrites made outside the cache (cache disabled, cooldown on) stay
uncompressed
- rewriteCache.get and rewrite return rewrittenMetadata
A 429, or a 503 with Retry-After, from an upstream host now stops the
proxy sending that host anything for as long as it asked (capped at an
hour), or for a minute doubling up to fifteen when no Retry-After came.
Held requests fail locally: cached metadata is served stale as before,
and anything uncached answers 503 with the remaining Retry-After so the
client paces itself too.
The gate is a RoundTripper under the auth transport, so metadata,
artifact and passthrough fetches all obey it. Metadata fetches had no
retry or circuit breaker, so a rate-limited upstream kept receiving one
request per cache miss for the whole block. /health lists held hosts
under upstream_holds and proxy_upstream_rate_limit_holds_total counts
holds.
