The compare endpoint buffers both archives whole, up to 512 MB each, and
anyone who can reach the UI can call it. A handful of parallel requests for
large cached versions was enough to exhaust the proxy's memory.
- Run at most two diffs at once; refuse more with 503, code BUSY and
Retry-After instead of queueing them
- Keep the per-archive cap, which legitimately large packages need
- Move the compare test's setup into a helper shared by the new tests
Artifacts had no size limit. A fetch was bounded only by the server's write
timeout, and storage.max_size evicts only after the bytes are written, so
any client could fill storage or keep fetches busy by requesting huge
artifacts.
- Add artifact_max_size (PROXY_ARTIFACT_MAX_SIZE), unlimited by default
- Refuse an artifact whose declared size is over the limit before reading
it, and fail one of unknown size once the download passes the limit
- Apply it in the fetcher so stored, streamed and mirrored artifacts all
honour it; nothing is cached and clients get 502
Handlers set the configured upstream auth header on the request itself
before handing it to the client. When upstream redirected to another host,
Go's client copied that header to the next hop: it strips only
Authorization and Cookie, so a "header" type credential such as
X-JFrog-Art-Api reached the redirect target. The auth transport already
applies credentials per hop, scoped to each hop's URL.
- Remove Proxy.applyUpstreamAuth and Proxy.AuthForURL; auth comes only
from the transport
- Add a server test that redirects a header-authenticated upstream to
another host and asserts the header stays with the upstream
- Wire handler tests' auth through the transport, as the server does
The compare page loaded diff2html from cdn.jsdelivr.net without a version
or integrity hash, so whatever jsDelivr served as the latest release ran
with the proxy origin's privileges, and the page broke where the proxy
cannot reach the internet.
- Vendor diff2html 3.4.56 (MIT) under static/vendor like tailwind and lucide
- Load its script and stylesheet from /ui/static/vendor
- Fail a test when any template loads a script or stylesheet from another origin
The source browser built its file tree with onclick handlers such as
loadFile('${escapeHTML(file.path)}'). The browser decodes HTML entities in
an attribute before running the handler, so the ' that escapeHTML
produces turns back into a quote and ends the string. A cached package
with a crafted file name ran script on the proxy origin for anyone who
browsed it.
- Carry the path in data-path and dispatch clicks from one delegated
listener on the file tree
- Render the binary download button as a link instead of an onclick
that assigns window.location
- Fail the template tests when an inline handler interpolates data
Packages are stored as pkg:deb/{name}, without a vendor and whichever configured archive they came from. Package rules may name a vendor the way PURLs usually do (pkg:deb/debian/curl, pkg:deb/ubuntu/curl); it's dropped, so a rule covers the package from every archive. The docs row warns that Debian and Ubuntu archives only keep the current version per suite.
The mirror can't fetch deb packages, so there's no mirror test.
Refs #306
On a build network where clients cannot reach sum.golang.org at all, the
proxy should keep answering for the database during an outage, since a 404
on supported would only send the go command somewhere it cannot go. With
the switch set, a cached tree of any age is enough; without it, the tree
has to be fresh, as before.
That does not make every cached module verify. The go command checks a
lookup against tiles of the tree that came with it, and sum.golang.org
attaches a different tree to almost every lookup, so it can ask for a tile
nobody fetched through the proxy before. configuration.md says so.
While the database is known to be unreachable, offline mode also serves
cached lookups and tiles straight away. Otherwise each of them would sit
through the upstream timeout before the fetch falls back to the cache,
and a build would take minutes for as long as the outage lasts. A lookup
or tile that falls back to the cache renews what the proxy knows about
the database, unless a probe has reached it since the request started.
The go command asks supported only once per process, so a long build would
otherwise be back to waiting a minute into the outage. recordTimeout is
now recordFailure, since lookups use it too.
The option needs cache_metadata, since without it there is no cached tree
to fall back on, and the config refuses the combination.
Everything under /go/sumdb/ was passed on to the module upstream. But
proxy.golang.org doesn't serve the checksum database, so those requests
always came back 404 and every client had to talk to sum.golang.org itself.
The Go handler now serves /sumdb/{name}/ for the databases in
upstream.go_sumdbs, which defaults to sum.golang.org. Lookups, latest and
tiles come from the database through the metadata cache. Complete tiles
never change, so once they're cached we don't ask again. Unknown names and
paths that aren't part of the protocol get a 404 without going upstream.
If the database answers 404 we pass on up to 4 KiB of its text, because the
go command prints it and it tells you why a version is missing. For that the
not-found error from the metadata fetch now carries the start of the body.
Other handlers still just see ErrUpstreamNotFound.
The handler stores packages as {channel}/{name}, which purl writes as pkg:conda/conda-forge%2Fnumpy. Package rules name the channel as the namespace (pkg:conda/conda-forge/numpy) and get mapped to that form, with the package name lowercased the way conda treats it. A key without a channel or with an empty part could never match and is refused.
The mirror can't fetch conda packages today, so there's no mirror test.
Refs #306
Tarballs are stored under the name from the tarball filename, and purl keeps hex names as they are, so package rules use the default mapping with the name as published.
The filename parser splits versions like 4.1.5-1 at the wrong hyphen and files them under a package like amqp_client-4.1.5, so package rules don't reach them; the docs row says so until the parser is fixed.
Refs #306
The mirror passed ids through as written, so pkg:nuget/Newtonsoft.Json@13.0.3 was stored under a copy no client is served while the handler kept serving its lowercased one. GetOrFetchArtifact now lowercases nuget names, and the docs no longer ask for lowercase ids in mirror lists.
Refs #306
The history is process-wide, and tests cleared it only after running, and
not all of them. Records from earlier tests or an earlier -count repetition
reached later assertions: TestRecordUpstreamOperations left a record for
TestRecentFailuresNewestFirst, and the server tests never cleared it.
- Export ResetRecentFailures so tests outside the metrics package can use it
- Clear the history before and after every test that records or reads
failures directly, in metrics, server and handler
- Assert exactly one failure in the handler tests instead of looking only
at the newest
Source archives are stored under the registry identity, pkg:generic/swift-registry/{scope}.{name}, lowercased. Package rules write the scope as the namespace (pkg:swift/apple/swift-nio) or use the stored form, in any case. A source repository key like pkg:swift/github.com/apple/swift-nio doesn't name a registry identity and is refused.
The mirror can't fetch swift packages, so there's no mirror test.
Refs #306
Since #431 the handler stores Conan archives under pkg:conan/{name} with the version as the version and the revisions and any user/channel in the filename. purl keeps conan names as they are, so package rules use the default mapping, and pkg:conan/zlib covers every version, revision and user/channel of the recipe.
The mirror can't fetch conan packages, so there's no mirror test.
Refs #306
Archives are stored under pkg:composer/{vendor}/{package}, which purl lowercases, so case in a package rule doesn't matter. A key without a vendor could never match and is refused. Packages that git-pkgs wrote as "packagist" fall under the composer rules too.
The mirror can't fetch composer packages today (its resolver has no composer support, and #415 would file them under packagist), so there's no mirror test.
Refs #306
Crates are stored under the name they were published with, and purl keeps cargo names as they are. Package rules have to match that name exactly, so pkg:cargo/inflector doesn't cover Inflector.
Refs #306
The handler stores NuGet ids lowercased, but purl keeps nuget names as written, so package rules get lowercased before they're matched. Case in a rule doesn't matter.
The mirror stores ids as given, so a mixed-case PURL ends up as a copy clients never get. Until that's fixed, the docs ask for lowercase ids in mirror lists.
Refs #306
Packages are stored as pkg:rpm/{name}, without a vendor and with the name's case kept. Package rules may name a vendor the way PURLs usually do (pkg:rpm/fedora/curl); it's dropped. The docs row warns that Fedora and EPEL update repositories only keep the latest build.
The mirror can't fetch rpm packages, so there's no mirror test.
Refs #306
Packages are stored under the name from the download path, and purl keeps pub names as they are, so package rules use the default mapping with the name as published.
Refs #306
Gems are stored under the name from the gem filename, and purl keeps gem names as they are, so package rules use the default mapping and match the name as published, case included. Packages that git-pkgs wrote as "rubygems" fall under the gem rules too.
The filename parser splits platforms that end in a number (universal-darwin-22) at the wrong hyphen and files those gems under a package like sorbet-static-0.5.11002-universal-darwin, so package rules don't reach them; the docs row says so until the parser is fixed.
The mirror can't fetch gems today (it asks the resolver for "rubygems", see #415), so there's no mirror test.
Refs #306
Julia takes no package rules, since its PURLs need a uuid qualifier the cache doesn't store; configuring one already fails validation. The julia rule and the default cover package tarballs, artifacts and registry snapshots alike. Everything there is content-addressed, so an evicted tarball comes back unchanged.
Refs #306
Helm takes no package rules: charts are cached under the configured repository's name and the chart digest, not under the chart name, so a key like pkg:helm/nginx would name a repository. The helm rule and the default cover charts from every HTTP repository; charts pulled over OCI go through the container handler and fall under oci.
Refs #306
* Enable storage retention for npm
npm stores packages under the PURL purl builds from the full name, so package rules use the default mapping. Both pkg:npm/@babel/core and pkg:npm/%40babel/core work.
Refs #306
* Tighten npm retention tests and docs
The docs now say npm package rules match the name exactly, so pkg:npm/jsonstream doesn't cover JSONStream. The refetch, package rule and mirror tests were weaker than their names and now check what they claim.
Refs #306
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
* Enable storage retention for pypi
PyPI names are stored the way purl normalizes them, lowercased and with _ turned into -, so Typing_Extensions and typing-extensions hit the same package. The mirror can't fetch PyPI files today, so there's no mirror test.
Refs #306
* Correct the pypi package rule note
Dots in PyPI names aren't normalized, so zope.interface and zope_interface files end up under two packages. The docs now say to add a rule for each.
Refs #306
* Mention repeated separators in the pypi rule note
Runs like "__" aren't collapsed either, so a__b is stored as a--b.
Refs #306
Module paths are stored lowercased, so package rules match whatever case they're written in. A major version like /v2 is its own module and needs its own rule. Keys in the !-escaped form from proxy URLs and the module cache are refused, since they could never match.
Refs #306
Blobs are stored under the repository name of the pull, which purl writes as one name (pkg:oci/library%2Fnginx). Package rules write it as a path, pkg:oci/library/nginx, and get mapped to that form; keys with an empty segment, a tag or digest, or a bare upstream/{name} prefix are refused. Named upstreams keep their prefix, also when containerd reaches them through ns. Homebrew bottles and Helm charts pulled over OCI fall under oci too, bottles under Homebrew's image name (openssl@3 is homebrew/core/openssl/3). Manifests live in the metadata cache and aren't touched.
Refs #306
Package rules use pkg:maven/{group}/{artifact} with the case as published. Keys without a group or with the group written as a path can never match and are refused. Files from the Gradle Plugin Portal fallback fall under the maven rule; the Gradle build cache isn't affected.
The mirror can't fetch Maven packages today (it passes group/artifact where the resolver wants group:artifact), so there's no mirror test.
Refs #306
- Add ui_failure_details / PROXY_UI_FAILURE_DETAILS, off by default
- Each failure row expands to its latest failures, newest first
- Show a note naming the config key when failures exist and the flag is off
- Document the flag in README, configuration reference and example config
- Failure recorders take a FailureDetail: PURL, filename, URL, storage path, error
- Keep the last 20 failures per counter series in memory, reset on restart
- Strip userinfo, query and fragment from URLs, including those inside error text
- Pass the detail from upstream, storage, integrity, health probe and scan call sites
- Package names and URLs stay out of Prometheus labels
Source, archived and binary packages are stored under pkg:cran/{name}, the name from the filename with its case kept, and purl keeps cran names as they are. Package rules use the default mapping and match the name as published, so pkg:cran/rcpp doesn't cover Rcpp.
The mirror can't fetch cran packages, so there's no mirror test.
Refs #306
Packages are stored as pkg:apk/alpine/{name} whichever configured repository they came from, so a package rule covers the package in all of them. Rules may leave out the alpine namespace (pkg:apk/curl); another namespace could never match and is refused. The docs row warns that Alpine mirrors only keep the current build, so an evicted older one can't come back.
The mirror can't fetch apk packages, so there's no mirror test.
Refs #306
* Cache configured version-pinned paths of generic upstreams
The generic handler only knew GitHub release asset paths were immutable, so
any other download, such as Maven from archive.apache.org, went through the
metadata route: streamed uncached by default, or kept as metadata and never
listed as a package.
upstream.generic_artifacts maps a generic upstream to regular expressions
for its version-pinned paths. A pattern must match the whole path after
/generic/{name}/ and has version and file named groups, plus an optional
name group. Matching downloads take the release asset route: cached in the
artifact cache, served while the upstream is down, and listed under the
generic ecosystem. Other paths are unchanged.
* Fix generic artifact cache identity and query forwarding
* Serve generic release assets cached under the unhashed key
Release assets cached before hashed keys used {repository}/{asset} and
became unreachable after an upgrade, so an offline upstream returned 502.
Serve such an entry when its recorded upstream URL matches the request.
---------
Co-authored-by: Giles Westwood <201443+gilesw@users.noreply.github.com>
* Add age-based retention for cached artifacts
Cached artifacts stayed forever unless storage.max_size pushed them out.
storage.retention adds a default, per-ecosystem and per-package duration
after which an artifact nobody downloaded gets evicted.
Ecosystems opt in one at a time: a handler registers with
retention.Default, and until it does, configuring that ecosystem (or one
of its packages) fails validation, and the default doesn't touch it. This
commit registers none, so nothing changes yet. Startup logs which
ecosystems the default covers and which it doesn't.
An artifact counts as expired when both its fetch time and its last
access are older than the cutoff. Checking only the last access would
evict a refetched artifact again right away, since clearing a record
keeps its old access time and a cache miss doesn't record a hit.
The sweep scans artifacts in id windows, so no single query holds the
SQLite connection for long, and checks each row against its own rule.
Expired records are cleared and their files go through pending_deletes
instead of being deleted directly, because a buffered hit can make an
artifact that's being served right now look expired. Buffered hits are
flushed before each sweep, and one sweep clears at most 10,000
artifacts.
Since a big first sweep can queue a lot, reclaim now keeps working
through due entries for up to 30 seconds per tick instead of stopping
after 100, and pending_deletes gets an index on queued_at.
Also:
- proxy_artifacts_evicted_total{reason, ecosystem} counts LRU and
retention evictions.
- gradle.build_cache.max_age accepts "7d", as its comment always said.
Refs #306
* Document storage retention
Describes the new storage.retention block, the order rules apply in,
that the default only covers ecosystems that support retention yet, and
what to expect from the first sweep over an old cache. Also lists the
new eviction counter and notes that gradle max_age takes days.
Refs #306
* Fix retention timing on Postgres and tighten the sweep
On Postgres, fetched_at and last_accessed_at are TIMESTAMP columns
without a zone. They hold the local time the proxy wrote, but lib/pq
reads them back labelled UTC. East of UTC that makes every artifact look
hours younger than it is, so the sweep evicted late: two hours in
Berlin, nine in Tokyo. The candidates' times are now read back as local
time, and the database tests run on both SQLite and Postgres, with one
that sets the local zone to Asia/Tokyo.
A few smaller things:
- When a delete fails, reclaim stops for the rest of the minute instead
of trying every queued path. Otherwise a storage outage turns into
thirty seconds of failed deletes and warnings every minute.
- If the buffered hits can't be written, the sweep skips that round.
Running on stale access times could clear something that is being
downloaded right now.
- DefaultCanonical only accepts a package key when rebuilding it gives
the same PURL. For alpine and deb it would otherwise produce keys
that never match.
- sweep_interval has to be at least a minute.
- New tests cover the eviction counter and an artifact that gets a hit
between the scan and the clear.
- The docs no longer say nothing changes without retention, and they
point out that the example's ecosystem and package rules only pass
validation once those ecosystems support retention.
Refs #306
* Reject qualifiers and subpaths in retention package keys
A rule names a whole package. Qualifiers like ?type=pom were dropped on
the way to the stored PURL and the rule quietly covered every file of the
package. Now such a key fails validation.
Refs #306
* Say when a package key doesn't match how packages are stored
A package rule for an ecosystem that supports retention, written in a
form its packages aren't stored under (pkg:npm/babel/core without the
@), failed with "retention is not supported for npm packages". Once the
ecosystem is enabled that is no longer true. Such a key now fails with
"does not match how npm packages are stored" and points to the package
rule forms in the docs. Keys for ecosystems without retention support
keep the old message.
Refs #306
* Note that re-running the mirror keeps packages cached
A mirror run over a package that is still cached goes through the normal
cache lookup and records a download, so the package's age starts over.
The docs only said that mirrored packages age from the time they were
mirrored.
Refs #306
* Say when an ecosystem takes no package rules
Some ecosystems will support retention without package rules, because nothing in their stored records names a package the way a PURL would (conan, helm). A package key for one of them used to fail with "does not match how conan packages are stored; see the package rule forms", which sends people looking for a form that doesn't exist. It now says package rules aren't supported there and points to the ecosystem rule.
Refs #306
* Use helm as the example of an ecosystem without package rules
Conan gets package rules once its handler stores archives under the recipe name, so the config test that shows the refusal now uses helm, whose cache records name the repository, not the chart.
Refs #306
* Say that each cached file ages on its own
The retention docs said each artifact, each version of a package, ages on its own. A version can have several cached files (a jar and its pom, several wheels, Conan sources and binaries), and each of them ages separately.
Refs #306
* Clear an expired artifact and queue its file in one transaction
The sweep cleared the record and queued the file for deletion as two separate writes. If queueing failed, or the process stopped in between, the file stayed in storage with no record pointing at it and no queue entry, and no later sweep would find it again. Both writes now share a transaction, so a failed queue rolls the clear back and the next sweep retries.
Refs #306
* Show artifact evictions on the analytics page
proxy_artifacts_evicted_total had no place on /ui/analytics. The Runtime card now lists evicted artifacts by reason and ecosystem, without the failure tint, and the coverage test records an eviction so it doesn't depend on test order.
Refs #306
* Store artifact times in UTC
fetched_at and last_accessed_at are now written in UTC, and the retention cutoffs are compared in UTC, the same convention #434 uses for the metadata cache. lib/pq sends a time with its offset and a Postgres TIMESTAMP keeps only the wall clock, so UTC on write is what reads back correctly. That replaces reinterpreting the times as local on read.
Rows written before this still hold local time. East of UTC they look younger by the offset until they are rewritten, so retention evicts them a few hours late once; west of UTC that much early.
Refs #306
---------
Co-authored-by: t <t@t>
