Closes#525. With metadata caching and the rewrite cache on, a packument
is streamed from upstream into storage and rewritten from there in two
passes: the first collects the time map, dist-tags and version names, the
second copies each kept version and rewrites its tarball URL. Neither the
packument nor its rewrite is held in memory uncompressed, so
metadata_max_size no longer limits packuments.
Both passes check the stored bytes against the digest in the cache row, so
a document replaced in between is not rewritten from mixed bytes. Documents
the stream can't handle (abbreviated, no versions object) and rewrite
errors fall back to the in-memory rewrite.
A cached rewrite made with cooldown on stays right until a version it held
back leaves cooldown, so it now expires then instead of after a fixed
time. Handlers ask cooldown through Proxy.cooldown, which reports each
withheld version to the rewrites running at that moment. A rewrite that
held nothing back is kept for the shortest cooldown, at most a day.
An hour is long next to a cooldown of a few hours. The TTL is now a 24th
of the shortest cooldown configured, between a minute and an hour, and
the RubyGems and Hex section no longer says a minute.
pnpm 12 checks every lockfile entry against the registry on each install
unless trustLockfile is set, so each install reads the packument of every
package in the lockfile. The rewrite cache kept each document twice, plain
and gzipped, and with cooldown on for a minute only, so these packuments
were rewritten and compressed again on every install.
Keep only the gzip copy of a cached rewrite, about an eighth of the
document, and decompress it for clients and handlers that need the plain
bytes. Keep rewrites made with cooldown on for an hour. Answer HEAD for
packuments, which pnpm 12 sends before each GET.
The Helm index.yaml and PyPI simple pages and JSON API documents were
parsed and rewritten on every request, cache hits and conditional
requests included. They now go through the rewrite cache like npm and
Composer, so a repeat request is served from memory with an ETag of
its own, a dated Last-Modified, a gzip copy and 304 when it matches.
PyPI simple pages are keyed by their content type and the versions
filtered out as well as the upstream page. Download hrefs are now
rewritten with a literal prefix search that makes the same changes as
the old regular expression, and the link filter's expressions are
compiled once.
A chart cache miss no longer parses the whole Helm index again while
the stored index is unchanged: the chart downloads it lists are kept
per repository, keyed by the index digest.
- Name the proxy in the User-Agent of publish-time HEAD requests, as its
artifact downloads do, rather than Go's default, which Maven Central
asks tools not to send.
- Take a version's time from its Gradle .module as well as its .pom, so
either one fetched first carries it.
- Serve filtered metadata through the shared rewrite path, with gzip and
the stale warning like the other ecosystems.
- Say in the docs that a refused .pom or .module is still cached.
With cooldown on, every npm, Composer and Chef metadata request rewrote
the whole document, because a cached rewrite could keep hiding a
version after its cooldown ended. For a 24.5 MB packument that is about
45 ms and 31 MB of allocations per request, and the result was served
uncompressed and without Last-Modified, so Composer, which revalidates
p2 files with If-Modified-Since only, never got a 304.
The rewrite cache now keeps cooldown rewrites with an expiry of one
minute, so a version leaving cooldown shows up at most that late.
Concurrent requests share one rewrite, clients get the gzip copy, and
the rewrite is dated through metadataDate like any other. Its ETag is
the hash of its output, so replicas agree on it.
The cache is keyed by the digest the metadata cache already recorded,
instead of hashing the document again, which also lets storedRewrite
serve cooldown rewrites without reading the stored document.
Without the rewrite cache, cooldown rewrites behave as before.
* Separate artifact and metadata cache metrics on /metrics and the analytics page
* Serve metadata cache stats from a shared snapshot and split Grafana cache panels by kind
* max_size now counts cached metadata as well as artifacts, and eviction drops whichever was used least recently, so metadata no longer piles up for good (#461). Metadata rows record when they were last served, an artifact not yet served counts from its download, and Arch stores a database blob again if eviction removed it under a refreshed row.
* Shorter comments on the metadata last-access migration.
* Show evicted metadata on the analytics page next to evicted artifacts, and cover the new counter in the metric surface test.
* Evicting metadata now drops the row and queues its path in one transaction, so a crash before the blob is deleted leaves it to reclaim instead of orphaning it.
* The last-access backfill now runs in the background in batches of 1000 instead of inside the migration, resuming after a restart, and until a row is backfilled eviction orders it by its write time.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
- Keep a gzip copy of each rewrite cache entry when it is smaller, made once
per rewrite and counted against metadata_rewrite_cache_size
- Serve it with Content-Encoding: gzip to clients that accept gzip, with an
ETag of its own ("<sha256>-gzip") and Vary: Accept-Encoding
- Rewrites made outside the cache (cache disabled, cooldown on) stay
uncompressed
- rewriteCache.get and rewrite return rewrittenMetadata
With hosted packages enabled, the packument of a package with pushed versions lists each of them, built from its package.json with dist pointing at the proxy and carrying the archive's sha512, and the time it was pushed. A pushed version takes the place of an upstream entry with the same version. Cooldown and the denylist still filter only upstream versions, and dist-tags stay as upstream has them.
When the upstream doesn't know the package, the packument holds only the pushed versions, with latest set to the highest of them. Requests for a single pushed version are answered from its entry.
Rewrites that include pushed versions bypass the rewrite cache, which is keyed by the upstream document alone, so a push or delete shows on the next request and changes the ETag. Packages without pushed versions take the cached path as before, and with hosted packages disabled there is no extra lookup.
Refs #134
Dates were kept in each process's rewrite cache, so replicas sharing a
database and storage dated independently: one dated a document, upstream
changed in that second, and another gave the new document the same date,
so a client holding it got 304 and kept the old document.
- Date from metadata_dates instead of process memory: a package's date is
the second it started serving its current representation, recorded once
for every replica
- Switch a package to another representation only in a second later than
its recorded date, and serve it undated until then
- Date nothing in the second the process started, so dates clients hold
from before this change, or from the process it replaced, stay older
- Serve no date ahead of the local clock, recorded by a replica whose clock
is ahead
- Date rewrites without the rewrite cache too; the cache keeps only ETags
- Add two-replica, pre-fix date and clock skew regression tests
Two false 304s remained. A rewrite kept the date it was made, so when
upstream went from A to B and back to A, A's cached rewrite came back with
a date earlier than B's and a client holding B got 304. And a second
rewrite in the same second was dated a second ahead of the clock, a date
only that process remembered: after a restart a changed document could be
dated earlier than one the client held.
- Record which rewrite each package is serving and since when, and date a
rewrite by the second its package switched to it, including back to one
still cached
- Never date ahead of the clock: a rewrite switched to in a second that
already dated another for the package goes undated until the next second
- Date nothing in the second the cache starts, the last second the previous
process may have dated a rewrite in
- Drop a package's record when its current rewrite is evicted, so records
stay bounded by the cache; forgetting one dated in the current second
holds back dates for new packages until the next second
Last-Modified was upstream's date clamped to the process start, so
different documents could share a date: after a restart, a cached document
and the newer one a refresh fetched were both dated at startup, and a client
holding the first got 304 for the second.
- Date each rewrite by when the rewrite cache first made it, and keep that
date while the entry stays cached
- Give every new rewrite of a package a date later than the last one it got,
even within the same second or after its entry was evicted, so no two
representations share a date; dates are remembered only until the clock
passes them
- Send no Last-Modified when the rewrite cache is off, since nothing records
when an uncached rewrite was made; cooldown still sends none
- Drop the upstream Last-Modified from metadata lookups, which nothing uses
now
- With cooldown on every rewrite runs, and hashing each rewritten body for
its ETag made every npm metadata request about 20% more expensive
- Hash what fixes the output instead: the upstream digest, proxy URL,
package name, the versions cooldown and the denylist kept, and a
per-process nonce standing for the startup configuration
- Carry the stored document's digest and upstream Last-Modified with the
metadata lookup result, so neither is read from a second cache row lookup
that a concurrent refresh could have changed
- Fall back to hashing the body when the digest is unknown, and keep the
body hash with cooldown off, where it is computed once per cached rewrite
- Rewritten metadata went out with only Content-Type, so pnpm, npm and
Composer could never revalidate and downloaded every document in full
- ETag is the SHA-256 of the rewritten body, computed once per rewrite and
kept with the rewrite cache entry; exact under cooldown as well
- Last-Modified is the upstream Last-Modified, never earlier than the process
start, since a restart can change the rewrite; omitted with cooldown on
- Answer If-None-Match, and If-Modified-Since when there is no If-None-Match,
with 304 on the rewrite fast path and on the fetch path
Rewriting a packument decoded the whole document into map[string]any and
encoded it again. For a full packument of tens of megabytes that costs many
times the document size per request, and with cooldown enabled (which skips
the rewrite cache and forces full packuments) it happens on every request.
- Rewrite npm metadata in place: walk the raw JSON for member offsets, decode
only the version names, time and dist-tags for the existing filters, and
assemble the response from slices of the original bytes with only tarball
URLs (and filtered time/dist-tags) written fresh. Untouched values now reach
clients byte for byte instead of round-tripping through encoding/json.
- Look up versions[v].dist.tarball and time[v] directly on the tarball
download path instead of decoding every version.
- Record the stored metadata's SHA-256 in metadata_cache.content_digest and
key the rewrite cache by it, so a repeat npm or Composer request inside the
TTL is served from the rewrite cache without reading or hashing the stored
document.
On a synthetic 25 MB packument with 5000 versions, rewriteMetadata drops from
~150-174 MB / 706k allocations per call to ~31 MB / 90k, and runs in about
two thirds of the time. npmVersionTarball drops from ~1 MB / 10k allocations
to one allocation.
The npm and Composer handlers rewrite every metadata document they serve
so that download URLs point at the proxy: decode the whole document into
generic maps, change the URLs, encode it again, and for Composer expand
the minified format first. That ran on every request, cached metadata
included. With metadata caching on and upstream out of the picture, a
cached request still cost 7 ms for @babel/core, 17 ms for
symfony/console and 155 ms and 134 MB of allocations for typescript, and
an 8 vCPU VM serving cached npm packuments to 20 clients ran the proxy
at 450% CPU for about 1,100 requests a second.
Rewritten documents are now kept in memory, keyed by the ecosystem,
proxy URL, package and a SHA-256 of the raw document, so new bytes from
upstream are rewritten again and nothing is served stale. Requests that
arrive while a document is being rewritten wait for that rewrite rather
than running their own; a waiter leaves when its client does, and the
rewrite still completes and is cached. The denylist is fixed at startup,
so it needs no place in the key. Cooldown filtering depends on the
current time, so with cooldown on the cache is bypassed.
metadata_rewrite_cache_size bounds the cache (default "256MB", least
recently used out first, "0" to rewrite on every request). NewProxy
callers keep the old behaviour unless they set it.
Whole cached requests, measured locally:
@babel/core 7.1 ms -> 0.38 ms, 54,489 -> 119 allocations
typescript 155 ms -> 10.5 ms, 1.37M -> 133 allocations
symfony/console 17.4 ms -> 0.48 ms, 165,909 -> 123 allocations
What remains is reading the raw document from storage and hashing it.
