Once a proxy answers 200 on supported, the go command sends all checksum
requests there and never goes to the database directly. A proxy that cannot
reach sum.golang.org would therefore break every build that needs a lookup.
supported now fetches latest first, through the cache, and answers 404 if
that fails and nothing is cached, so clients fall back like before. A failed
check is remembered for a minute so an unreachable database doesn't make
each go command wait for an upstream timeout.
Everything under /go/sumdb/ was passed on to the module upstream. But
proxy.golang.org doesn't serve the checksum database, so those requests
always came back 404 and every client had to talk to sum.golang.org itself.
The Go handler now serves /sumdb/{name}/ for the databases in
upstream.go_sumdbs, which defaults to sum.golang.org. Lookups, latest and
tiles come from the database through the metadata cache. Complete tiles
never change, so once they're cached we don't ask again. Unknown names and
paths that aren't part of the protocol get a 404 without going upstream.
If the database answers 404 we pass on up to 4 KiB of its text, because the
go command prints it and it tells you why a version is missing. For that the
not-found error from the metadata fetch now carries the start of the body.
Other handlers still just see ErrUpstreamNotFound.
The first matching pattern names the package, so an upstream's own
patterns now take precedence over overlapping built-ins and keep the
package names, cache entries and rules already keyed on them.
Downloads are routed as /{channel}/{arch}/{filename}, so the stored name is always {channel}/{name}. A key like pkg:conda/conda-forge/label/rc/numpy passed validation and matched nothing; it's now refused.
Refs #306
parseTarballFilename split at the last hyphen before a digit, so amqp_client-4.1.5-1.tar was stored as package amqp_client-4.1.5. Hex names can't contain a hyphen, so it now splits at the first one. The docs caveat is gone.
Refs #306
The handler stores packages as {channel}/{name}, which purl writes as pkg:conda/conda-forge%2Fnumpy. Package rules name the channel as the namespace (pkg:conda/conda-forge/numpy) and get mapped to that form, with the package name lowercased the way conda treats it. A key without a channel or with an empty part could never match and is refused.
The mirror can't fetch conda packages today, so there's no mirror test.
Refs #306
Tarballs are stored under the name from the tarball filename, and purl keeps hex names as they are, so package rules use the default mapping with the name as published.
The filename parser splits versions like 4.1.5-1 at the wrong hyphen and files them under a package like amqp_client-4.1.5, so package rules don't reach them; the docs row says so until the parser is fixed.
Refs #306
The mirror passed ids through as written, so pkg:nuget/Newtonsoft.Json@13.0.3 was stored under a copy no client is served while the handler kept serving its lowercased one. GetOrFetchArtifact now lowercases nuget names, and the docs no longer ask for lowercase ids in mirror lists.
Refs #306
parseGemFilename split at the last hyphen before a digit, so sorbet-static-0.5.11002-universal-darwin-22.gem was stored as package sorbet-static-0.5.11002-universal-darwin. It now splits at the first one, which only misreads gem names with a -<digit> segment. The docs caveat is gone.
Refs #306
The history is process-wide, and tests cleared it only after running, and
not all of them. Records from earlier tests or an earlier -count repetition
reached later assertions: TestRecordUpstreamOperations left a record for
TestRecentFailuresNewestFirst, and the server tests never cleared it.
- Export ResetRecentFailures so tests outside the metrics package can use it
- Clear the history before and after every test that records or reads
failures directly, in metrics, server and handler
- Assert exactly one failure in the handler tests instead of looking only
at the newest
Source archives are stored under the registry identity, pkg:generic/swift-registry/{scope}.{name}, lowercased. Package rules write the scope as the namespace (pkg:swift/apple/swift-nio) or use the stored form, in any case. A source repository key like pkg:swift/github.com/apple/swift-nio doesn't name a registry identity and is refused.
The mirror can't fetch swift packages, so there's no mirror test.
Refs #306
Since #431 the handler stores Conan archives under pkg:conan/{name} with the version as the version and the revisions and any user/channel in the filename. purl keeps conan names as they are, so package rules use the default mapping, and pkg:conan/zlib covers every version, revision and user/channel of the recipe.
The mirror can't fetch conan packages, so there's no mirror test.
Refs #306
Archives are stored under pkg:composer/{vendor}/{package}, which purl lowercases, so case in a package rule doesn't matter. A key without a vendor could never match and is refused. Packages that git-pkgs wrote as "packagist" fall under the composer rules too.
The mirror can't fetch composer packages today (its resolver has no composer support, and #415 would file them under packagist), so there's no mirror test.
Refs #306
Crates are stored under the name they were published with, and purl keeps cargo names as they are. Package rules have to match that name exactly, so pkg:cargo/inflector doesn't cover Inflector.
Refs #306
Generic upstreams pointed at archive.apache.org, nodejs.org or
dl.google.com/go now cache their versioned downloads as artifacts without
any generic_artifacts patterns. Configured patterns add to the built-in ones.
- Start from the schema before migration 011 on SQLite and PostgreSQL, with
npm and Composer metadata cached, and check the rows and stored bytes
survive the migration unchanged
- After the upgrade, check two replicas serve the cached document with
matching ETag and Last-Modified and answer conditional requests with 304
- Change the upstream document and check neither replica answers the old
validators with 304, and both agree on the new ones
- Give the PostgreSQL handler test a database of its own, so it shares no
tables with the database package tests on the same server
The handler stores NuGet ids lowercased, but purl keeps nuget names as written, so package rules get lowercased before they're matched. Case in a rule doesn't matter.
The mirror stores ids as given, so a mixed-case PURL ends up as a copy clients never get. Until that's fixed, the docs ask for lowercase ids in mirror lists.
Refs #306
Packages are stored as pkg:rpm/{name}, without a vendor and with the name's case kept. Package rules may name a vendor the way PURLs usually do (pkg:rpm/fedora/curl); it's dropped. The docs row warns that Fedora and EPEL update repositories only keep the latest build.
The mirror can't fetch rpm packages, so there's no mirror test.
Refs #306
Packages are stored under the name from the download path, and purl keeps pub names as they are, so package rules use the default mapping with the name as published.
Refs #306
Gems are stored under the name from the gem filename, and purl keeps gem names as they are, so package rules use the default mapping and match the name as published, case included. Packages that git-pkgs wrote as "rubygems" fall under the gem rules too.
The filename parser splits platforms that end in a number (universal-darwin-22) at the wrong hyphen and files those gems under a package like sorbet-static-0.5.11002-universal-darwin, so package rules don't reach them; the docs row says so until the parser is fixed.
The mirror can't fetch gems today (it asks the resolver for "rubygems", see #415), so there's no mirror test.
Refs #306
Julia takes no package rules, since its PURLs need a uuid qualifier the cache doesn't store; configuring one already fails validation. The julia rule and the default cover package tarballs, artifacts and registry snapshots alike. Everything there is content-addressed, so an evicted tarball comes back unchanged.
Refs #306
Helm takes no package rules: charts are cached under the configured repository's name and the chart digest, not under the chart name, so a key like pkg:helm/nginx would name a repository. The helm rule and the default cover charts from every HTTP repository; charts pulled over OCI go through the container handler and fall under oci.
Refs #306
Dates were kept in each process's rewrite cache, so replicas sharing a
database and storage dated independently: one dated a document, upstream
changed in that second, and another gave the new document the same date,
so a client holding it got 304 and kept the old document.
- Date from metadata_dates instead of process memory: a package's date is
the second it started serving its current representation, recorded once
for every replica
- Switch a package to another representation only in a second later than
its recorded date, and serve it undated until then
- Date nothing in the second the process started, so dates clients hold
from before this change, or from the process it replaced, stay older
- Serve no date ahead of the local clock, recorded by a replica whose clock
is ahead
- Date rewrites without the rewrite cache too; the cache keeps only ETags
- Add two-replica, pre-fix date and clock skew regression tests
Replicas sharing a database need one place to agree on the Last-Modified
of rewritten metadata, which process memory cannot give them.
- Add metadata_dates, keyed by ecosystem, proxy URL and package, holding the
ETag being dated and its date in Unix seconds, so no zone is involved
- Insert a package's first date only if no replica recorded one first
- Swap a date only if it is still the one read, so of two replicas
switching a package at once only one succeeds
- Create it in both schemas and in migration 011
Two false 304s remained. A rewrite kept the date it was made, so when
upstream went from A to B and back to A, A's cached rewrite came back with
a date earlier than B's and a client holding B got 304. And a second
rewrite in the same second was dated a second ahead of the clock, a date
only that process remembered: after a restart a changed document could be
dated earlier than one the client held.
- Record which rewrite each package is serving and since when, and date a
rewrite by the second its package switched to it, including back to one
still cached
- Never date ahead of the clock: a rewrite switched to in a second that
already dated another for the package goes undated until the next second
- Date nothing in the second the cache starts, the last second the previous
process may have dated a rewrite in
- Drop a package's record when its current rewrite is evicted, so records
stay bounded by the cache; forgetting one dated in the current second
holds back dates for new packages until the next second
Last-Modified was upstream's date clamped to the process start, so
different documents could share a date: after a restart, a cached document
and the newer one a refresh fetched were both dated at startup, and a client
holding the first got 304 for the second.
- Date each rewrite by when the rewrite cache first made it, and keep that
date while the entry stays cached
- Give every new rewrite of a package a date later than the last one it got,
even within the same second or after its entry was evicted, so no two
representations share a date; dates are remembered only until the clock
passes them
- Send no Last-Modified when the rewrite cache is off, since nothing records
when an uncached rewrite was made; cooldown still sends none
- Drop the upstream Last-Modified from metadata lookups, which nothing uses
now
- With cooldown on every rewrite runs, and hashing each rewritten body for
its ETag made every npm metadata request about 20% more expensive
- Hash what fixes the output instead: the upstream digest, proxy URL,
package name, the versions cooldown and the denylist kept, and a
per-process nonce standing for the startup configuration
- Carry the stored document's digest and upstream Last-Modified with the
metadata lookup result, so neither is read from a second cache row lookup
that a concurrent refresh could have changed
- Fall back to hashing the body when the digest is unknown, and keep the
body hash with cooldown off, where it is computed once per cached rewrite
- Rewritten metadata went out with only Content-Type, so pnpm, npm and
Composer could never revalidate and downloaded every document in full
- ETag is the SHA-256 of the rewritten body, computed once per rewrite and
kept with the rewrite cache entry; exact under cooldown as well
- Last-Modified is the upstream Last-Modified, never earlier than the process
start, since a restart can change the rewrite; omitted with cooldown on
- Answer If-None-Match, and If-Modified-Since when there is no If-None-Match,
with 304 on the rewrite fast path and on the fetch path
* npm: version lookup returns 502
Looking up a specific version will trigger a 502 instead of return the version metadata.
* scope only not handled explicitly
* honor cooldown in version lookups
tested (set default cooldown to 7d and 8.0.7 is released four days ago or something):
```
$ curl -sS 'http://localhost:8080/npm/@babel%2Fcore/8.0.7' | jq
{
"error": "package not found"
}
```
* package not found -> version not found
* gocognit
