Every npm and Composer download ran GetLocalArtifact, a full artifact
row lookup, even with hosted packages disabled. localVersions already
returns early in that case, so metadata does not list pushes, and
/generic/hosted/ is only served while hosting is enabled.
GetLocalArtifact now returns nil without a lookup when hosting is off.
Downloads then take the normal proxy path. An archive pushed earlier is
still answered from storage by that path and is never replaced by the
upstream copy; only cooldown now applies to it, as metadata no longer
lists it either. The hosted docs say so.
Composer names are case-insensitive, so acme/private and acme/Private
share the version record and its integrity but get different filenames.
A second push under the other case was accepted and broke the integrity
check of the first archive. For ecosystems with one file per version,
any pushed file of the version now refuses the push, both before the
upload and atomically when recording it.
- Record a fetched artifact with its package and version in one
transaction that refuses to replace a pushed artifact, so a fetch that
finishes after a push keeps both the pushed file and its version's
integrity. Pushes and cache fills share this write, including the
compare-and-swap on the storage path that queues every replaced object.
- Don't open the stored file for ecosystems without a manifest.
- Derive multi-file versions from the file being named in the path.
- Validate DELETE paths like PUT paths, answering 400 for malformed ones.
- Parse ?sha256= with integrity.ParseHex.
- Record the package, version and artifact of a push in one transaction
that only inserts the artifact if no pushed copy exists. Of two
concurrent pushes of the same file exactly one wins, and the loser's
version integrity is rolled back with it.
- Check for an existing file before reading the request body, and send
Expect: 100-continue from proxy hosted push, so a refused re-push isn't
uploaded first.
- Leave an existing version record alone for multi-file ecosystems, so a
generic push doesn't reset a version cached from upstream.generic.
- Reserve the upstream.generic name "hosted" only while hosting is
enabled, and only intercept /generic/hosted/ then.
- Accept a file argument in proxy hosted delete for generic only.
- Mark multi-file ecosystems explicitly, and give parse a min/max range.
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
- GetLocalArtifact went through checkCache, which reports a miss when StreamArtifacts is set, so npm and Composer downloads fetched the upstream archive instead of the pushed one
- Split readStored out of lookupCache and read pushed artifacts through it regardless of StreamArtifacts
- Keep the denylist check and the 503/404 answers for unreadable or missing pushed artifacts
- Test npm and Composer downloads of pushed archives with StreamArtifacts set
Refs #133
Pushing and downloading refuse denied versions, but npm and Composer metadata listed every pushed version. A version denied after it was pushed stayed listed, so resolvers picked it and then got 403 on the download. localVersions now leaves denied versions out, which covers both ecosystems and the hosted-only namespaces; npm latest falls back to the highest allowed version. /api/hosted still lists them, so operators can see and delete them.
Refs #134
checkCache answered nil both when a stored object couldn't be opened and after an integrity failure had cleared the record. Download handlers then fetched the upstream archive for the same version and filename, and recording it turned the pushed row into an upstream one without its manifest and queued the pushed object for deletion. One storage error while downloading a patched version served the unpatched upstream archive and dropped the push.
checkCache now looks at the record on a miss. For a pushed artifact it returns ErrHostedArtifactUnavailable when the object is recorded but couldn't be served, and ErrHostedArtifactMissing when no object is recorded any more. Every path that would fetch upstream goes through it, so none of them can replace the pushed artifact. Downloads answer 503 and 404 for the two.
Refs #133
A package the upstream doesn't know is served from its pushed versions, but the upstream is still asked about it. Once someone publishes the same name there, their versions are listed next to the pushed ones and a version range can resolve to them.
hosted.npm_scopes and hosted.composer_vendors (PROXY_HOSTED_NPM_SCOPES, PROXY_HOSTED_COMPOSER_VENDORS) name namespaces whose packages are served from pushed versions alone. Their metadata is built from what was pushed, npm latest included, and anything not pushed answers 404. The upstream is never asked about them. Packages in other namespaces are proxied as before.
The npm metadata handler's decisions about pushed versions move into serveLocalMetadata.
Refs #133
Refs #134
Resolvers take a version's dependencies from the registry metadata, not from the archive, so listing a pushed version needs its manifest. A push now reads package.json from the top directory of an npm tarball, or composer.json from the root or the single top directory of a Composer zip, and keeps it in the new artifacts.manifest column (migration 013_add_artifact_manifest).
The manifest has to name the pushed package and version. A Composer manifest may leave the version out, and its name is compared case-insensitively. An archive that can't be read or whose manifest names something else is refused with 400 and removed from storage.
The version's integrity is now the archive's sha512, which npm clients expect in dist.integrity.
Refs #134
With hosted.enabled, PUT /api/hosted/{ecosystem}/{name}/{version} stores the request body as the archive for that version, marked local. It replaces any cached upstream copy and is served from the cache without contacting the upstream. GET /api/hosted/{ecosystem}/{name} lists what was pushed and DELETE removes a version again, after which clients get the upstream archive if there is one. Pushing and deleting need the bearer token; listing is open like the rest of the API.
Download handlers look archives up by the filename clients request, so a push is stored under the name its ecosystem derives from name and version: {name}-{version}.tgz for npm, without the scope, and {package}-{version}.zip for Composer. Other ecosystems are refused for now.
The Composer download handler resolves the upstream metadata before it looks at the cache and answers 404 for versions the upstream doesn't list. It now serves a pushed archive first. Only artifacts marked local take that path, so cached upstream archives still go through the metadata and commit checks.
The version's integrity is set to the pushed bytes, so the cache's integrity check verifies them rather than an upstream digest. Names and versions with path separators, dot segments or control characters are refused before anything is stored.
Refs #133
