A generic upstream forwards any path below /generic/{name}/. The
documented mise setup puts a GITHUB_TOKEN on api.github.com to raise
the rate limit, so anyone who could reach the proxy could make any GET
request with that token, including for private repositories it can
read, and the answers went into the shared metadata cache. upstream.git
already refuses an upstream with credentials unless it lists its repos.
upstream.generic_paths now lists, per generic upstream, the paths it
may serve, with the same pattern syntax as git repos. Other paths
answer 404 without reaching the upstream, and so does a path holding
'%', '?', '#' or a backslash, which the upstream could read as a
different path than the one that matched. Config validation refuses a
generic upstream that upstream.auth has credentials for unless it has
generic_paths, which also covers a github.com token reaching the
generic github upstream. Upstreams without credentials keep serving
every path.
The docs and example config show the paths mise needs and recommend a
fine-grained token limited to public repositories.
- Record the package, version and artifact of a push in one transaction
that only inserts the artifact if no pushed copy exists. Of two
concurrent pushes of the same file exactly one wins, and the loser's
version integrity is rolled back with it.
- Check for an existing file before reading the request body, and send
Expect: 100-continue from proxy hosted push, so a refused re-push isn't
uploaded first.
- Leave an existing version record alone for multi-file ecosystems, so a
generic push doesn't reset a version cached from upstream.generic.
- Reserve the upstream.generic name "hosted" only while hosting is
enabled, and only intercept /generic/hosted/ then.
- Accept a file argument in proxy hosted delete for generic only.
- Mark multi-file ecosystems explicitly, and give parse a min/max range.
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
The first matching pattern names the package, so an upstream's own
patterns now take precedence over overlapping built-ins and keep the
package names, cache entries and rules already keyed on them.
Generic upstreams pointed at archive.apache.org, nodejs.org or
dl.google.com/go now cache their versioned downloads as artifacts without
any generic_artifacts patterns. Configured patterns add to the built-in ones.
* Cache configured version-pinned paths of generic upstreams
The generic handler only knew GitHub release asset paths were immutable, so
any other download, such as Maven from archive.apache.org, went through the
metadata route: streamed uncached by default, or kept as metadata and never
listed as a package.
upstream.generic_artifacts maps a generic upstream to regular expressions
for its version-pinned paths. A pattern must match the whole path after
/generic/{name}/ and has version and file named groups, plus an optional
name group. Matching downloads take the release asset route: cached in the
artifact cache, served while the upstream is down, and listed under the
generic ecosystem. Other paths are unchanged.
* Fix generic artifact cache identity and query forwarding
* Serve generic release assets cached under the unhashed key
Release assets cached before hashed keys used {repository}/{asset} and
became unreachable after an upgrade, so an offline upstream returned 502.
Serve such an entry when its recorded upstream URL matches the request.
---------
Co-authored-by: Giles Westwood <201443+gilesw@users.noreply.github.com>
* feat(storage): expose efficient seek for local files
Open file-backed artifacts with os.Open so local readers support
efficient seeking. Leave cloud storage readers unchanged.
Keep the legacy sidecar mapping test focused on fileblob's reader:
Blob.Open now reads local files directly and no longer consults the
sidecar. Continue checking sidecar cleanup and post-store reads.
* feat(handler): preserve efficient seek through integrity checks
Keep the cache result's single Reader field and preserve io.Seeker
when the underlying storage reader supports efficient seeking.
Sequential reads continue through whole-object integrity verification.
A successful seek switches subsequent reads to the source for range
responses, where full-object verification is not possible.
Test both modes and verify the source is closed once.
* feat(handler): add request-aware byte range responses
Add a request-aware artifact serving helper for bounded, open-ended,
and suffix byte ranges, including If-Range handling, 206 responses,
and 416 responses for valid unsatisfiable ranges.
Ignore malformed and multi-range requests by serving the full artifact.
Advertise byte-range support only for seekable readers, preserving HEAD,
redirect, and non-seekable behavior. Clarify that net/http recovers
ErrAbortHandler per request and keeps the server running.
* feat(handler): enable byte ranges across artifact downloads
Forward each artifact request to the shared range-aware response helper,
including cached generic release assets and OCI blobs. This lets the
handler honor ranges only when its reader supports efficient seeking,
without changing direct-storage redirect behavior.
Add warm-cache endpoint tests that assert partial response headers and
bytes without an upstream refetch. Document supported range behavior,
unsupported reader fallback, and the integrity limitation of partial reads.
* Add generic HTTP download proxy for GitHub release assets
Adds a /generic/{name}/ route backed by a new upstream.generic named-upstream
map, so tools that download from fixed URL shapes (mise's aqua backend
fetching GitHub release assets, and its tag lookups on api.github.com) can be
pointed at the proxy with client-side URL rewriting. Only configured
upstreams are reachable, so this is not an open HTTP proxy.
Paths shaped like {owner}/{repo}/releases/download/{tag}/{asset} are
version-pinned and go through the artifact cache: fetched once, hashed,
served without revalidation, and still served when the upstream is down.
Every other path goes through the metadata cache with the client's Accept
header and query string replayed, so API responses are fresh within
metadata_ttl, revalidated after that, and served stale when the upstream
fails or rate-limits the request.
Tests cover path classification, unknown upstreams and traversal, cache
hits with the upstream down, HEAD, 404 pass-through, Accept and query
forwarding, stale-on-429, cache isolation between upstreams, and that an
upstream token scoped to the release host is not sent to the object store
it redirects to.
Closes#183.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDjDeq27CKzEP2o3GWBY7F
* Use fixed Accept header for generic metadata
---------
Co-authored-by: Giles Westwood <giles@gileswestwood.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
