* Delete cached artifacts through the API and the UI
cache_delete (PROXY_CACHE_DELETE) registers
DELETE /api/cache/{ecosystem}/{name}/{version}, which removes a
version's cached files, or only ?filename=, so the next request fetches
them from the upstream again. Records are cleared and the stored objects
queued for deletion like a replaced artifact. Pushed archives are left
alone: a whole-version delete skips them and naming one answers 409.
With it enabled, the version page shows a Delete button for each cached
file and one for the whole version. The UI calls /ui/api/cache so the
same ingress rules that guard /ui guard its deletes. Neither route has
authentication of its own, so the option is off by default.
* Load the cache delete script from a static file
Move the version page's delete-button JavaScript into
static/cache-delete.js and load it with a script tag, as
packages-list.js does.
---------
Co-authored-by: Giles Westwood <201443+gilesw@users.noreply.github.com>
* max_size now counts cached metadata as well as artifacts, and eviction drops whichever was used least recently, so metadata no longer piles up for good (#461). Metadata rows record when they were last served, an artifact not yet served counts from its download, and Arch stores a database blob again if eviction removed it under a refreshed row.
* Shorter comments on the metadata last-access migration.
* Show evicted metadata on the analytics page next to evicted artifacts, and cover the new counter in the metric surface test.
* Evicting metadata now drops the row and queues its path in one transaction, so a crash before the blob is deleted leaves it to reclaim instead of orphaning it.
* The last-access backfill now runs in the background in batches of 1000 instead of inside the migration, resuming after a restart, and until a row is backfilled eviction orders it by its write time.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
Every Composer archive download fetched the package's p2 metadata straight
from upstream to find the archive's URL, before the archive cache was even
checked. A cached archive cost one or two uncached Packagist requests per
package and install, and failed with a 502 while Packagist was down.
With cache_metadata on, downloads now look the dist up in the metadata the
proxy already caches for Composer, under the keys handlePackageMetadata uses.
Within the TTL that needs no upstream request, and when upstream fails the
stale copy still resolves the archive. A version or file the cached
documents do not list, such as one published since they were cached, is
looked up upstream as before.
* Add mirror.schedules, a list of image tags the server re-mirrors at a fixed interval. Digests are rejected since they never move.
* Run each schedule through the normal mirror, one run after the other. The first run starts at a random point within ten minutes and later runs get ±10 % jitter, so several instances don't hit the registry at once.
* Start the schedules with the server and let shutdown wait for a running refresh before storage and database are closed. Docs included.
* Check cached layers during a mirror without opening them, so a refresh no longer counts as a cache hit and doesn't keep scheduled images at the top of LRU eviction.
- Keep a gzip copy of each rewrite cache entry when it is smaller, made once
per rewrite and counted against metadata_rewrite_cache_size
- Serve it with Content-Encoding: gzip to clients that accept gzip, with an
ETag of its own ("<sha256>-gzip") and Vary: Accept-Encoding
- Rewrites made outside the cache (cache disabled, cooldown on) stay
uncompressed
- rewriteCache.get and rewrite return rewrittenMetadata
A 429, or a 503 with Retry-After, from an upstream host now stops the
proxy sending that host anything for as long as it asked (capped at an
hour), or for a minute doubling up to fifteen when no Retry-After came.
Held requests fail locally: cached metadata is served stale as before,
and anything uncached answers 503 with the remaining Retry-After so the
client paces itself too.
The gate is a RoundTripper under the auth transport, so metadata,
artifact and passthrough fetches all obey it. Metadata fetches had no
retry or circuit breaker, so a rate-limited upstream kept receiving one
request per cache miss for the whole block. /health lists held hosts
under upstream_holds and proxy_upstream_rate_limit_holds_total counts
holds.
Credentials configured for a token service were applied to any challenge
naming it. A host the proxy fetches from, such as one in a Composer dist
URL or a Helm index, could answer a /v2/ request with a Bearer challenge
naming that realm, and the proxy fetched a token with the configured
credentials and sent it to that host. A token cached from a legitimate
pull was handed over the same way.
- Follow a challenge to a realm with configured credentials only for a
registry that has configured credentials of its own, or is on the same
site (registrable domain) as the realm, as registry-1.docker.io is with
auth.docker.io; otherwise pass the 401 back
- Keep following anonymous challenges for any registry, as before
- Document the rule
proxy list and proxy delete sat next to serve, stats and mirror and read as cache operations. The commands are now proxy hosted push, proxy hosted list and proxy hosted delete, before a release makes the names hard to change. proxy hosted without a known subcommand prints the three.
Refs #133
A package the upstream doesn't know is served from its pushed versions, but the upstream is still asked about it. Once someone publishes the same name there, their versions are listed next to the pushed ones and a version range can resolve to them.
hosted.npm_scopes and hosted.composer_vendors (PROXY_HOSTED_NPM_SCOPES, PROXY_HOSTED_COMPOSER_VENDORS) name namespaces whose packages are served from pushed versions alone. Their metadata is built from what was pushed, npm latest included, and anything not pushed answers 404. The upstream is never asked about them. Packages in other namespaces are proxied as before.
The npm metadata handler's decisions about pushed versions move into serveLocalMetadata.
Refs #133
Refs #134
proxy push, proxy list and proxy delete call the /api/hosted endpoints of a running proxy. The proxy URL comes from -url or PROXY_URL and the token from PROXY_HOSTED_TOKEN, so it never ends up in shell history or process listings.
Document the hosted config block, the endpoints, the supported ecosystems and how pushed archives interact with eviction.
Refs #133
With hosted.enabled, PUT /api/hosted/{ecosystem}/{name}/{version} stores the request body as the archive for that version, marked local. It replaces any cached upstream copy and is served from the cache without contacting the upstream. GET /api/hosted/{ecosystem}/{name} lists what was pushed and DELETE removes a version again, after which clients get the upstream archive if there is one. Pushing and deleting need the bearer token; listing is open like the rest of the API.
Download handlers look archives up by the filename clients request, so a push is stored under the name its ecosystem derives from name and version: {name}-{version}.tgz for npm, without the scope, and {package}-{version}.zip for Composer. Other ecosystems are refused for now.
The Composer download handler resolves the upstream metadata before it looks at the cache and answers 404 for versions the upstream doesn't list. It now serves a pushed archive first. Only artifacts marked local take that path, so cached upstream archives still go through the metadata and commit checks.
The version's integrity is set to the pushed bytes, so the cache's integrity check verifies them rather than an upstream digest. Names and versions with path separators, dot segments or control characters are refused before anything is stored.
Refs #133
The compare endpoint buffers both archives whole, up to 512 MB each, and
anyone who can reach the UI can call it. A handful of parallel requests for
large cached versions was enough to exhaust the proxy's memory.
- Run at most two diffs at once; refuse more with 503, code BUSY and
Retry-After instead of queueing them
- Keep the per-archive cap, which legitimately large packages need
- Move the compare test's setup into a helper shared by the new tests
Artifacts had no size limit. A fetch was bounded only by the server's write
timeout, and storage.max_size evicts only after the bytes are written, so
any client could fill storage or keep fetches busy by requesting huge
artifacts.
- Add artifact_max_size (PROXY_ARTIFACT_MAX_SIZE), unlimited by default
- Refuse an artifact whose declared size is over the limit before reading
it, and fail one of unknown size once the download passes the limit
- Apply it in the fetcher so stored, streamed and mirrored artifacts all
honour it; nothing is cached and clients get 502
Packages are stored as pkg:deb/{name}, without a vendor and whichever configured archive they came from. Package rules may name a vendor the way PURLs usually do (pkg:deb/debian/curl, pkg:deb/ubuntu/curl); it's dropped, so a rule covers the package from every archive. The docs row warns that Debian and Ubuntu archives only keep the current version per suite.
The mirror can't fetch deb packages, so there's no mirror test.
Refs #306
On a build network where clients cannot reach sum.golang.org at all, the
proxy should keep answering for the database during an outage, since a 404
on supported would only send the go command somewhere it cannot go. With
the switch set, a cached tree of any age is enough; without it, the tree
has to be fresh, as before.
That does not make every cached module verify. The go command checks a
lookup against tiles of the tree that came with it, and sum.golang.org
attaches a different tree to almost every lookup, so it can ask for a tile
nobody fetched through the proxy before. configuration.md says so.
While the database is known to be unreachable, offline mode also serves
cached lookups and tiles straight away. Otherwise each of them would sit
through the upstream timeout before the fetch falls back to the cache,
and a build would take minutes for as long as the outage lasts. A lookup
or tile that falls back to the cache renews what the proxy knows about
the database, unless a probe has reached it since the request started.
The go command asks supported only once per process, so a long build would
otherwise be back to waiting a minute into the outage. recordTimeout is
now recordFailure, since lookups use it too.
The option needs cache_metadata, since without it there is no cached tree
to fall back on, and the config refuses the combination.
Two problems with the check behind supported. The fetch it waits on is
shared with other callers and only bounded by the upstream timeout, which
defaults to 30 seconds. A firewall that silently drops packets to
sum.golang.org would stall the first go command every minute for that long,
and the go command itself has no timeout on this request. Now we wait five
seconds at most; the fetch carries on in the background, records how it
went and fills the cache for the next check. A timeout only counts as a
failure if the fetch hasn't succeeded in the meantime.
The fetch reports no error when it falls back to the cached tree after the
database failed, so the metadata result now says when it did, and the
check treats that as the database not answering. A fresh cached tree
(within metadata_ttl) still lets a timed-out check say yes, since the
fetch only falls back once upstream has failed, which a silent database
delays past our timeout. An older tree does not count: once the proxy has
lost the database, clients go there directly, as they would without the
proxy.
And the check believed any 200. Egress proxies tend to answer blocked hosts
with a 200 HTML page, which we would have cached as the signed tree and taken
as proof that the database is reachable. latest now has to start with
"go.sum database tree", or it is refused like a bad tile.
Lookups had the same gap. A block page cached as a lookup would also
replace the good copy that is served during an outage, so a lookup now has
to be a record followed by a signed tree, the way the go command reads it.
Complete tiles are never fetched again once cached, so a bad response would
stick around until someone deletes it by hand. That could be an error page
from a proxy in between that comes back with 200.
A hash tile is always W*32 bytes, so anything else is refused now and not
stored, and the same check runs when a cached tile is served. Data tiles
have no fixed size and only must not be empty.
A cached tile that fails the check is fetched again without asking the
database whether it changed. Past the TTL the cache would ask, and a
database that sends an ETag would answer 304 and hand back the damaged
copy every time.
The fake database in the tests now guards its maps with the mutex, since the
tests change them between requests.
Once a proxy answers 200 on supported, the go command sends all checksum
requests there and never goes to the database directly. A proxy that cannot
reach sum.golang.org would therefore break every build that needs a lookup.
supported now fetches latest first, through the cache, and answers 404 if
that fails and nothing is cached, so clients fall back like before. A failed
check is remembered for a minute so an unreachable database doesn't make
each go command wait for an upstream timeout.
The first matching pattern names the package, so an upstream's own
patterns now take precedence over overlapping built-ins and keep the
package names, cache entries and rules already keyed on them.
parseTarballFilename split at the last hyphen before a digit, so amqp_client-4.1.5-1.tar was stored as package amqp_client-4.1.5. Hex names can't contain a hyphen, so it now splits at the first one. The docs caveat is gone.
Refs #306
The handler stores packages as {channel}/{name}, which purl writes as pkg:conda/conda-forge%2Fnumpy. Package rules name the channel as the namespace (pkg:conda/conda-forge/numpy) and get mapped to that form, with the package name lowercased the way conda treats it. A key without a channel or with an empty part could never match and is refused.
The mirror can't fetch conda packages today, so there's no mirror test.
Refs #306
Tarballs are stored under the name from the tarball filename, and purl keeps hex names as they are, so package rules use the default mapping with the name as published.
The filename parser splits versions like 4.1.5-1 at the wrong hyphen and files them under a package like amqp_client-4.1.5, so package rules don't reach them; the docs row says so until the parser is fixed.
Refs #306
The mirror passed ids through as written, so pkg:nuget/Newtonsoft.Json@13.0.3 was stored under a copy no client is served while the handler kept serving its lowercased one. GetOrFetchArtifact now lowercases nuget names, and the docs no longer ask for lowercase ids in mirror lists.
Refs #306
parseGemFilename split at the last hyphen before a digit, so sorbet-static-0.5.11002-universal-darwin-22.gem was stored as package sorbet-static-0.5.11002-universal-darwin. It now splits at the first one, which only misreads gem names with a -<digit> segment. The docs caveat is gone.
Refs #306
