A cache hit hashed the stored object with SHA-256 for content_hash and
again with every native algorithm in versions.integrity, SHA-512 for npm.
The native integrity was already checked when the bytes were stored and
content_hash was computed from the same bytes, so the second hash adds
CPU per byte served without catching anything more. Records without a
content hash still fall back to the native integrity.
Each download and metadata handler logged its own request line at info,
next to the request line LoggerMiddleware already writes for every
request with path, status and duration. That doubled the info log volume
on the hot path. The handler lines only repeat values parsed from the
path, so they move to debug.
Kept at info: the npm audit line (it carries the request body size), the
julia package line (it carries the name resolved from the UUID) and the
gem and hex cooldown lines.
Mirroring pkg:composer failed with unsupported ecosystem. A tagged release
is now mirrored with the dist archive upstream.composer lists, under the
file name the Composer handler serves it as and checked against the
listed shasum. The release's publish time is stored as the handler stores
it, so the archive is served from the cache without the metadata, with
cooldown on as well.
Branches are refused, since the handler reads the metadata for every
branch download, and so are archive URLs with a query and packages in
upstream.composer_routes.
The mirror now finds the whole version rather than its files, for the
dist type and publish time.
Part of #414.
A conda version is mirrored with every build anaconda.org lists for it,
each cached under the channel, name, version and subdir/filename the conda
download handler reads from its path. The channel comes from the namespace
or the channel qualifier; a package without one is refused, as retention
keys without one are. The subdir, build and type qualifiers select builds.
Which files a version has, which qualifiers select them and where the
handler caches them is now described per ecosystem, so PyPI and conda share
mirrorFiles.
Part of #414.
A streamed cache hit whose bytes don't match the row's digest now
records an integrity failure and deletes the row, queueing its blob for
deletion, as a corrupt cached artifact is handled. The next request
fetches the document from upstream instead of aborting until the TTL
runs out.
proxy mirror and mirror jobs failed for pypi with "unsupported
ecosystem", since the resolver they use has no registry for it. A PyPI
version has several files, so the mirror now lists them from
upstream.pypi's JSON API, through the same client as version listing,
and fetches each one, verified against its sha256. The file_name
qualifier limits it to one file.
Files are cached under the name and version the download handler reads
from the filename, which can differ from the PURL (zope.interface's
wheels are zope_interface), so pip is served them from the cache.
PyPI and npm downloads look up the publish time before the cache. When
that lookup fails, a file that is already cached is now served, and only
a cache miss fails with 502 or 503. The proxy keeps serving what it has
while the upstream is unreachable, and nothing new is fetched unchecked.
Also drops the unattached comment block in cooldown_fail_closed.go.
The case-variant lookup filtered on lower(name), which no index covers,
so a legacy row's first lookup scanned every metadata_cache row in its
ecosystem. Reclaim already skips a queued path a row still points at,
so each row now queues just its own legacy blob once it has moved off
it. On case-insensitive storage a case variant still at its legacy path
can lose its blob that way, and is then fetched again.
With cache_metadata on, every metadata cache hit read the whole stored
document into its own buffer before any coalescing, since
coalesceMetadata only covers upstream misses. metadata_max_size bounds
one request, not the sum: 30 concurrent hits on a 20 MB document held
about 820 MB.
The handlers that go through ProxyCached serve the stored document
unchanged, so a hit there no longer needs the bytes in memory.
serveCachedMetadata streams the blob from storage to the response
instead, with the row's Content-Type, Content-Encoding, size, ETag and
Last-Modified, and answers If-None-Match and If-Modified-Since with 304
as before. The bytes are checked against the row's content digest as
they are sent, and a mismatch aborts the response like a corrupt cached
artifact. A row without a digest or size, a document over the size
limit, or a blob that can't be opened takes the buffered path as before.
Handlers that rewrite or filter metadata keep reading it into memory,
and the rewrite cache stays in front of them. writeMetadataValidators
is split out of writeMetadataCachedResponseWithEncoding so both paths
share the conditional request handling.
The per-ecosystem rules for when two spellings name the same package
(PEP 503, NuGet case, Composer's leading v, conda channel, OCI paths,
Swift registry case) now live in purl. The denylist keeps only the
proxy's own pkg:generic/swift-registry spelling and the gem platform
match. Keys, cache paths and stored PURLs are unchanged.
Ignoring every row still at its unhashed path dropped the whole metadata
cache on upgrade, including on Linux and S3 where keys never collided,
and left nothing to serve while the upstream was down.
A legacy row's blob is now hashed once and compared with the row's
content_digest. When it matches, the blob is this key's document: it is
copied to the hashed path and the row moved there, so it is served
fresh or as a stale fallback as before. Only a mismatch, where another
key's document overwrote the shared file, or a row without a digest is
dropped and fetched again.
The legacy blobs are queued for deletion once no key differing only in
case still reads its legacy path, which on a case-insensitive
filesystem is the same file.
Debian, RPM, APK, PyPI, CRAN and Maven build upstream URLs from the
decoded request path, where a '?' or '#' would end the path and cache
one file under another's name. Only the server middleware refused them;
Arch and FreeBSD also check in the handler. Check in these handlers
too, so they stay safe when mounted without the middleware.
With cooldown on, the Hex handler read the /packages response, its decompressed payload and the Hex API's JSON without a limit, and the gem handler decoded the versions API's JSON the same way. A misbehaving upstream could make the proxy hold any amount in memory, and a small gzip payload could expand without bound.
These reads now go through ReadMetadata, so metadata_max_size applies to each, and to the Hex payload after decompression as the NuGet handler does. A response over the limit fails the request with 502.
RubyGems, Hex and PyPI keep publish times outside the metadata clients
read, so cooldown fetches them from a second API. When that fetch
failed, was rate limited, or returned something that didn't parse, the
compact index, Hex package or simple page was served unfiltered, and a
Hex package that couldn't be re-encoded was passed on as upstream sent
it. A flaky or rate-limited timestamp API let every version in
cooldown through.
These now fail with 502, or with 503 and Retry-After when the rate
limit gate held the request, as the Go handler already does. PyPI file
downloads check the same way. A PyPI package the JSON API answers 404
for still has no times and is served unfiltered, as before.
npm, Composer, pub.dev and PyPI JSON metadata that couldn't be
rewritten was also passed on unfiltered unless the package had denied
versions. It now fails with 502 whenever cooldown applies too. npm
tarball downloads whose packument couldn't be fetched skipped the
cooldown check and downloaded straight from upstream; they now fail.
All of this only applies when cooldown covers the package: packages
exempted with a duration of 0 skip the timestamp lookups, and the gem
and Hex handlers serve them through the metadata cache as when cooldown
is off.
upload-pack was started with its stdin attached to the request body, after
the read deadline had been extended to two hours, and ls-refs requests took
no slot. A client that sent part of a request and then nothing kept an
upload-pack process waiting, and any number of them could.
Stateless requests are complete once sent, so the handler now reads the
body in full first, within the server's read timeout, keeping bodies past
1 MiB in a temporary file. Only then are the deadlines extended for the
response. info/refs and ls-refs share their own slots, four per CPU, so they
still never wait behind clones. Each response write is bounded to two
minutes, so a client that stops reading a pack frees its slot instead of
holding it until the request timeout.
Conan, CRAN, Conda, RPM, the main Debian archive and the default Maven
repository keyed cached metadata by the request path with '/' replaced
by '_'. Names and versions may contain '_', so foo/bar_1.0 and
foo_bar/1.0 shared an entry, and with cache_metadata one reference's
recipe files were served for the other. Conan also appended the query
with '_', so a path without a query could collide with a shorter path
plus one.
Hash the upstream URL, the path and, for Conan, the query instead, as
the APK, Arch, generic and Helm handlers already do. Each part is
length-prefixed so no two inputs share a key.
Without artifact streaming, a chart was fetched with no expected
digest, committed to the cache, and only then compared with the
index's digest by serveChart, which cleared it again. Until then the
unverified chart was a cache entry like any other.
The index digest is now always passed to the fetch, so a mismatching
chart is discarded before it is committed, as it already was when
streaming. serveChart's check stays for charts that were cached before.
Manifests and digest-addressed artifacts were only checked against
sha256 digests. A sha512 digest, which the OCI spec also allows, or a
digest with any other algorithm was fetched and cached unchecked: a
manifest requested as sha512:<hex>, or a tag whose Docker-Content-Digest
header named one, was stored under that digest whatever the body was.
Manifests are now checked against any digest in the reference or the
Docker-Content-Digest header with the digest's own algorithm, and one
the proxy can't compute is answered with DIGEST_INVALID and not cached.
GetOrFetchArtifactFromURLWithDigest verifies sha512 (and sha384)
digests through the declared-integrity check, so a mismatch is never
committed, and refuses other algorithms with ErrUnsupportedDigest
instead of fetching unverified. Blob routes still only accept sha256.
Cached metadata was stored at _metadata/<ecosystem>/<key>/metadata. Keys
are case-sensitive, but on a case-insensitive filesystem such as APFS or
NTFS, file:// storage put keys that differ only in case in one file:
github.com/Sirupsen/logrus and github.com/sirupsen/logrus each had a
row, and whichever was cached last was served for both. Keys may also
hold names Windows reserves or end in a dot.
The path now carries a SHA-256 of the exact key. Keys that already are
SHA-256 hex digests, as most hashed keys are, keep their path.
A row still pointing at an unhashed path may hold another key's bytes,
so it is neither served nor revalidated; the document is fetched again
and stored at the new path, and the old blob is queued for deletion.
An uncached tarball is checked against the integrity its packument
declares. When the packument couldn't be fetched or was refused, for
example because it is larger than metadata_max_size or the registry
answered 503, the download went ahead with no integrity at all, and
the unverified tarball was cached and served to every later client.
Without the packument the download now fails like the Composer one
does: 502, or 404 when the registry says the package doesn't exist.
Nothing is fetched or cached. Tarballs already in the cache are still
served, since they were verified when they were stored, and a packument
that lists the version without a usable checksum still falls back to
an unverified fetch as before.
With cooldown on, /info/{name} is fetched from upstream and versions still in cooldown are removed line by line. The fetch passed on the client's Accept-Encoding, and setting that header turns off the transport's transparent decompression, so for any client that accepts gzip, Bundler included, the filter scanned gzip bytes, matched no versions and sent the compressed index on with upstream's Content-Encoding. Cooldown filtered nothing for those clients.
The fetch now passes on only Accept, so the transport decompresses the index before it is filtered. If-None-Match and If-Modified-Since are no longer passed on either: the client's validators came from a filtered copy, not from upstream's file.
The filtered index no longer carries upstream's ETag, Last-Modified, Digest, Repr-Digest or range headers, which describe the unfiltered file; Bundler checks the digest against the body it gets.
The index is now read through ReadMetadata before anything is written. bufio.Scanner stopped at a line over 64 KB and the handler sent what it had so far as a complete 200; a read error or an index over the metadata size limit now answers 502.
A generic upstream forwards any path below /generic/{name}/. The
documented mise setup puts a GITHUB_TOKEN on api.github.com to raise
the rate limit, so anyone who could reach the proxy could make any GET
request with that token, including for private repositories it can
read, and the answers went into the shared metadata cache. upstream.git
already refuses an upstream with credentials unless it lists its repos.
upstream.generic_paths now lists, per generic upstream, the paths it
may serve, with the same pattern syntax as git repos. Other paths
answer 404 without reaching the upstream, and so does a path holding
'%', '?', '#' or a backslash, which the upstream could read as a
different path than the one that matched. Config validation refuses a
generic upstream that upstream.auth has credentials for unless it has
generic_paths, which also covers a github.com token reaching the
generic github upstream. Upstreams without credentials keep serving
every path.
The docs and example config show the paths mise needs and recommend a
fine-grained token limited to public repositories.
parseGemFilename tries every "-<digit>" in the filename as the start of the version and matches the rest against two patterns each time, so its cost grows with the square of the filename's length. A path of a few hundred kilobytes kept a request busy for minutes, and the parse doesn't watch the request context.
Gem filenames longer than 512 bytes are now answered with 400 before parsing. Real ones are far shorter.
- Name the proxy in the User-Agent of publish-time HEAD requests, as its
artifact downloads do, rather than Go's default, which Maven Central
asks tools not to send.
- Take a version's time from its Gradle .module as well as its .pom, so
either one fetched first carries it.
- Serve filtered metadata through the shared rewrite path, with gzip and
the stale warning like the other ecosystems.
- Say in the docs that a refused .pom or .module is still cached.
A build fetching pinned versions never reads maven-metadata.xml, so each
version's time was read with a HEAD before its download, roughly
doubling the requests to Maven Central, which answered a fresh build
with 429 and a 30-minute Retry-After. The .pom's own response now gives
the time; Maven fetches the .pom before a version's other files, so
their check reads the stored time. A HEAD is only sent for a .pom served
from the cache or a file requested before its .pom.
A 429 on that HEAD now answers 503 with the upstream's Retry-After, as
other held upstream requests do, instead of 502.
A version's publish time is the Last-Modified of its .pom, read with a
HEAD request once per version and stored in the versions table. Named
repositories store theirs under a repository_url qualifier.
Versions in cooldown are left out of maven-metadata.xml, <latest> and
<release> are moved to the last versions left, and the metadata's
checksum files are computed from what is served. Files of a version in
cooldown get 403. Only versions that can still be in cooldown are
checked: none when <lastUpdated> is older than the window, otherwise
from the end of the list back to the first old enough version.
Part of #132.
A detached fetch waiting on another instance's lease ran the wait under
the request context, so a client leaving cancelled every caller joined
to it. The wait now runs past the client, bounded by the holder's fetch
and the lease expiry.
When the holder's fetch failed, each waiting instance took the lease in
turn and repeated it, so against a hanging upstream the k-th instance
waited about k timeouts. The holder now marks the lease failed. The
instances waiting on it then serve their stale metadata copy if they
have one, and otherwise fetch at the same time.
Clients resolve base_url and shards_base_url against the proxy's URL. A
scheme or host sends them around the proxy, and a leading slash leaves
its mount, so neither layout is served.
Instances sharing a Postgres database each fetched a missed artifact from
upstream; only callers within one instance shared a fetch. The instance
that fetches now holds a lease on the miss in a new fetch_leases table and
renews it while it downloads. Another instance missing the same key polls
the cache and serves the stored copy, or takes the lease over when the
holder releases it without storing anything or stops renewing it.
Metadata misses use the same lease when metadata caching is on with a TTL
above zero; without a stored copy there is nothing to wait for.
proxy_fetch_lease_waits_total counts the misses that waited. Reclaim
deletes leases left by stopped instances.
Refs #543
With cooldown on, every npm, Composer and Chef metadata request rewrote
the whole document, because a cached rewrite could keep hiding a
version after its cooldown ended. For a 24.5 MB packument that is about
45 ms and 31 MB of allocations per request, and the result was served
uncompressed and without Last-Modified, so Composer, which revalidates
p2 files with If-Modified-Since only, never got a 304.
The rewrite cache now keeps cooldown rewrites with an expiry of one
minute, so a version leaving cooldown shows up at most that late.
Concurrent requests share one rewrite, clients get the gzip copy, and
the rewrite is dated through metadataDate like any other. Its ETag is
the hash of its output, so replicas agree on it.
The cache is keyed by the digest the metadata cache already recorded,
instead of hashing the document again, which also lets storedRewrite
serve cooldown rewrites without reading the stored document.
Without the rewrite cache, cooldown rewrites behave as before.
Every Composer archive download resolved its dist from the package
metadata before checking the artifact cache. With cache_metadata off,
the default, that meant an uncoalesced upstream GET of the p2 document
for every download of an archive the proxy already had: three downloads
of a cached archive made three metadata requests. With the cache on,
each download after the TTL revalidated and rescanned the document.
The lookup is only needed for the publish time cooldown checks
downloads against, since lock files name archive URLs directly. Like
the npm handler, record that time in the versions table when the dist
is first resolved, and on later downloads of a tagged release read it
from there and serve the cached archive straight away. Without a stored
time, as for archives cached by an older proxy, the download takes the
metadata path, which then records it. The time is stored even with
cooldown off, so enabling cooldown later keeps the fast path.
The cache still enforces the denylist and verifies the stored digest,
and an archive is only ever cached under a name that was checked
against the metadata, so those guarantees hold. Branches, whose
archive and time change with every push, routed packages, whose
versions row would not say which repository the time came from, and
requests with a query string such as ?sha= keep resolving from the
metadata.
