* Coalesce concurrent metadata fetches
Concurrent cache misses for one artifact already share a single upstream
fetch (#329), but metadata misses did not: every request for a package's
metadata went to the registry, even when the same document was already
being fetched. In CI that is the common case. Composer resolves versions
on every job, since MediaWiki core and its extensions commit no
composer.lock, and each package is one /p2/ document, so a burst of jobs
fetched each document once per job.
Metadata misses now follow the same pattern as artifacts: the first caller
fetches, the rest wait for its result, and the caller that takes the key
rechecks the cache first so a fetch that just finished is not repeated.
The key includes the Accept and Accept-Encoding headers, which change the
bytes upstream returns (npm's abbreviated and full documents), and whether
the caller validates the response.
One difference from artifacts: the shared fetch is detached from the first
caller's cancellation. Artifact fetches keep it for scanning and mirroring,
neither of which applies to metadata, and otherwise one job disconnecting
would fail every job waiting on the same document. runScan detaches the
same way, and the HTTP client's timeout still bounds the fetch.
This covers every ecosystem that goes through fetchOrCacheMetadata
(Composer, npm, PyPI, Maven, Cargo, Swift, Pub, Helm, NuGet), with
metadata caching on or off. Streamed metadata with caching off is not
coalesced.
On an 8 vCPU test instance with caching off, 10 concurrent clients
requesting the npm metadata of 1,263 packages went from 12,630 upstream
requests to 1,898. With cache_metadata on, 20 clients requesting 1,538
packages made exactly 1,538.
* Run each caller's validate on a shared metadata fetch
A caller that joined another's metadata fetch got the shared bytes
without running its own validate. NuGet's cooldown path uses validate to
decode the document for the request, so a joined download found no
publication date and was allowed, bypassing the cooldown.
coalesceMetadata now reports whether the caller joined, and
coalescedMetadataMiss runs that caller's validate on the shared bytes.
Validation before caching is unchanged: the first caller still runs it
inside the fetch.
TestNuGetCooldownConcurrentDownloads reproduces the bypass through
NuGetHandler.Routes() with two overlapping downloads of a version inside
its cooldown.
* feat(storage): expose efficient seek for local files
Open file-backed artifacts with os.Open so local readers support
efficient seeking. Leave cloud storage readers unchanged.
Keep the legacy sidecar mapping test focused on fileblob's reader:
Blob.Open now reads local files directly and no longer consults the
sidecar. Continue checking sidecar cleanup and post-store reads.
* feat(handler): preserve efficient seek through integrity checks
Keep the cache result's single Reader field and preserve io.Seeker
when the underlying storage reader supports efficient seeking.
Sequential reads continue through whole-object integrity verification.
A successful seek switches subsequent reads to the source for range
responses, where full-object verification is not possible.
Test both modes and verify the source is closed once.
* feat(handler): add request-aware byte range responses
Add a request-aware artifact serving helper for bounded, open-ended,
and suffix byte ranges, including If-Range handling, 206 responses,
and 416 responses for valid unsatisfiable ranges.
Ignore malformed and multi-range requests by serving the full artifact.
Advertise byte-range support only for seekable readers, preserving HEAD,
redirect, and non-seekable behavior. Clarify that net/http recovers
ErrAbortHandler per request and keeps the server running.
* feat(handler): enable byte ranges across artifact downloads
Forward each artifact request to the shared range-aware response helper,
including cached generic release assets and OCI blobs. This lets the
handler honor ranges only when its reader supports efficient seeking,
without changing direct-storage redirect behavior.
Add warm-cache endpoint tests that assert partial response headers and
bytes without an upstream refetch. Document supported range behavior,
unsupported reader fallback, and the integrity limitation of partial reads.
expandMinifiedVersions deep-copied every inherited field into every
version (b68184c) so that rewriting one version's dist URL in place
could not change the versions that inherited it. For a package with a
long history that means recursively copying require, autoload and the
rest hundreds of times per document, on every metadata request.
dist is the only field the proxy changes after expansion, so copy just
that: rewriteDistURL now gives the version its own dist map before
setting the URL, and expansion shares the other inherited values.
TestComposerExpandMinifiedSharedDistReferences still guards the original
bug, and fails if the copy in rewriteDistURL is removed.
On the symfony/console metadata from Packagist the rewritten output is
byte-identical, and the rewrite drops from about 20.7 ms to 13.8 ms,
with allocations down from 13.6 MB to 9 MB.
The npm and Composer handlers rewrite every metadata document they serve
so that download URLs point at the proxy: decode the whole document into
generic maps, change the URLs, encode it again, and for Composer expand
the minified format first. That ran on every request, cached metadata
included. With metadata caching on and upstream out of the picture, a
cached request still cost 7 ms for @babel/core, 17 ms for
symfony/console and 155 ms and 134 MB of allocations for typescript, and
an 8 vCPU VM serving cached npm packuments to 20 clients ran the proxy
at 450% CPU for about 1,100 requests a second.
Rewritten documents are now kept in memory, keyed by the ecosystem,
proxy URL, package and a SHA-256 of the raw document, so new bytes from
upstream are rewritten again and nothing is served stale. Requests that
arrive while a document is being rewritten wait for that rewrite rather
than running their own; a waiter leaves when its client does, and the
rewrite still completes and is cached. The denylist is fixed at startup,
so it needs no place in the key. Cooldown filtering depends on the
current time, so with cooldown on the cache is bypassed.
metadata_rewrite_cache_size bounds the cache (default "256MB", least
recently used out first, "0" to rewrite on every request). NewProxy
callers keep the old behaviour unless they set it.
Whole cached requests, measured locally:
@babel/core 7.1 ms -> 0.38 ms, 54,489 -> 119 allocations
typescript 155 ms -> 10.5 ms, 1.37M -> 133 allocations
symfony/console 17.4 ms -> 0.48 ms, 165,909 -> 123 allocations
What remains is reading the raw document from storage and hashing it.
The flag exists to withhold caller addresses. The by-client table carries
none, and the same figures are already public at /metrics, so gating it
left two metrics with no tile on the page in the default config.
* Add storage.passthrough to serve artifacts without storing them
With storage.passthrough enabled the proxy streams every artifact from
upstream to the client and never writes it to storage or the cache
database. Metadata filtering, cooldown and the denylist work as before,
so the proxy can sit behind another cache (e.g. an Artifactory remote)
purely as a policy layer without holding a second copy of every package.
Artifacts whose digest is known up front (OCI, Swift, Helm) are verified
while streaming. Their responses are sent chunked and the connection is
aborted on a mismatch, so a client never receives a tampered artifact as
a complete response.
Passthrough is rejected together with scanning, direct_serve and
mirror_api, and the mirror command refuses to run with it, since all of
them depend on stored artifacts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address review: abort truncated streams, guard every cache read
- serveArtifact aborts the response on any body read error, and when fewer
bytes than the declared size were written, not only on a digest mismatch.
A failed or short upstream body in passthrough mode no longer reaches the
client as a complete 200.
- Streamed upstream read failures are logged and counted as
stream_failed upstream errors before the response is aborted.
- checkCache reports a miss when artifact caching is off, so every cache
read path, including Swift archive HEAD requests, ignores entries stored
before the mode was enabled. Denylisted versions are still rejected.
- Rename storage.passthrough to storage.cache_artifacts (default true,
PROXY_STORAGE_CACHE_ARTIFACTS), which names what actually changes.
Tests cover an OCI blob shorter than its Content-Length and an npm tarball
whose chunked response ends without the final chunk, both served through
the HTTP handlers, plus a Swift HEAD request against a cached archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Rename Proxy.Passthrough to StreamArtifacts
Match the storage.cache_artifacts option: the field says what changes
(artifacts are streamed instead of stored) and its zero value keeps the
usual caching behaviour for every Proxy built without the server config,
such as the mirror command and tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
With proxies sharing a database, a batch can flush after a newer hit is
already written, and overwriting last_accessed_at and updated_at with
its older time could reorder LRU eviction. Both now keep the later of
the stored and incoming times. The count still adds up.
- Track per-caller request and byte counts in a bounded table
- Export proxy_client_requests_total and
proxy_client_response_bytes_total over a closed label set; addresses
are never labels
- Gate the sources card behind ui_request_sources, off by default,
since /ui carries no authentication of its own
- Move access_log.trust_forwarded_for to a top-level key
- Keep remote as the TCP peer and add remote_ip alongside it
Part 6 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- 33 panels, no hardcoded data source UID
- Three ecosystem variables, because the label means three different
things across /metrics: the package record, the request path, and
the handler's own name
Part 5 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Add downloaded_bytes, downloads and an ecosystems array, served from
the snapshot the gauges and the page already share
- Add stats_unavailable so a failed aggregation is distinguishable
from an idle proxy
- Regenerate the OpenAPI spec
Part 4 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Ring of download size by ecosystem, cache figures, per-ecosystem
table, vulnerability overview and a Runtime card
- Add metrics.Gather so the page can render counters that were never
in the database
- Add proxy_response_bytes_total and response-writer byte counting
- Serve a retained snapshot behind a staleness banner when the
aggregation fails, rather than rendering old figures as current
- Extract the security overview into a shared component
Part 3 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Add GetEcosystemStats, aggregating packages, versions, artifacts,
cache size, downloads and downloaded bytes per ecosystem
- Publish six proxy_ecosystem_* gauges from it on the existing
one-minute cache-stats tick
- Set and selectively delete rather than Reset, so no scrape lands on
a half-populated vector
- Report artifacts with no package row under "unattributed" rather
than dropping them, so the figures still add up
Part 2 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
TestRelayRoutes fails intermittently on CI with "write: connection reset
by peer" at the upstream handler's final Flush, on /hex/packages/demo and
/gem/info/demo.
Those two routes are the only ones in the table that fan out a second
upstream request: with cooldown enabled, the hex and gem handlers fetch
version timestamps concurrently with the artifact. The test server answers
that sidecar request with the same 64 KiB chunked 502, but the handler
returns as soon as it sees a non-200 and closes the body unread, so the
proxy drops the connection while the test server is still writing to it.
Whether the reset lands before or after the Flush is a scheduling race,
which is why it only surfaces on loaded runners.
The flush error says nothing about the response under test, which is
asserted on the downstream side, so stop reporting it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Map /apk to alpine, matching what the handler writes to the database
- Report /helm, /homebrew, /generic and /swift under their own names
- Test every one of the 23 mounts, so the next route added cannot
quietly fall through to "other"
Part 1 of 6 splitting #381 up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every cached download wrote its hit in its own UPDATE. SQLite runs on
one connection, so each download was a write transaction the rest
queued behind, holding a warm npm cache near 1,000 requests a second
with the CPU two thirds idle.
Hits are now counted in memory and written in one transaction every
database.hit_flush_interval, "1s" by default, "0" for a write per hit.
A failed write keeps the hits for the next flush, and Close writes
what is pending. The same test then served about 2,200 requests a
second.
Closes#322.
Eviction deleted an object before checking its record still pointed at
it. If a refetch had moved the record, that object was queued and a
request that read the record earlier could still be opening it. Eviction
now clears first, skips the delete if the record moved, and queues the
object when the delete fails after a clear.
Reclaim queues a failed delete again, which moves it behind the rest, so
objects the backend keeps refusing cannot fill every batch.
Also drop the clearArtifactCache passthrough and ClearCachedArtifact's
unused context, note that only tests call ArtifactPath, and document that
max_size does not count objects waiting to be reclaimed.
