mirror of
https://github.com/git-pkgs/proxy.git
synced 2026-11-09 15:49:16 -05:00
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Christian Heim
|
729138d778 |
Reject package requests with ? or # in the decoded path (#437)
* Reject package requests whose decoded path contains '?' or '#', in one middleware in front of every ecosystem mount. Handlers build the upstream URL from the decoded path, so busybox.apk%3F/foo-9.9-r0.apk fetched busybox but cached it as foo; PyPI could be poisoned the same way, and the other ecosystems stored files under mangled names past the denylist and cooldown. * Let Composer archive downloads through the '?'/'#' guard: they resolve the upstream URL from metadata, and with #433 a branch like dev-fix/#42 is requested as dev-fix%2F%2342. Adversarial review finding F1. |
||
Christian Heim
|
7b6c64af0a | proxy mirror and the mirror API now pre-fetch container images from pkg:docker and pkg:oci PURLs: the mirror walks index, platform manifests (with their attestations) and blobs through the container handler, so a later pull is served from cache even when the upstream is down. A --platform filter (platforms in the API) limits multi-arch images, repository_url only resolves to configured registries, and names, tags and digests are validated before anything is fetched. | ||
|
Christian Heim
|
cc18c889ad | proxy mirror now builds its upstream clients the same way as serve, so it follows OCI Bearer challenges, uses configured upstream credentials and the SSRF guard, and artifact downloads have no overall time limit. Before this the CLI used a plain client and Docker Hub answered every request with 401. | ||
|
Christian Heim
|
094a962697 | At most 64 fetches keep running after their client hangs up; beyond that a fetch is tied to its client again, so nobody can start an unbounded number of large downloads and leave. A server test now checks that client requests actually get the detach marker. | ||
|
Christian Heim
|
111b271273 | Arch Linux pacman repositories are now proxied at /arch/{name}/, with geo.mirror.pkgbuild.com as the built-in archlinux repository and more mirrors configurable under upstream.arch. Databases and file lists go through the metadata cache byte for byte, and a database signature is refreshed together with its database so the two always match. Packages and package signatures go to the artifact cache and stay available offline. The path is passed through unchanged, so Arch Linux ARM's $arch/$repo layout works too. README, config example, configuration docs and the dashboard have pacman instructions. | ||
|
Christian Heim
|
2ed0327946 | Keep fetching an artifact after the client that started the fetch disconnects. pacman gives up after 10 seconds without data, and since the proxy only starts sending once the whole file is stored, a large package on a slow upstream was cancelled on every try and never got cached. Client requests now mark their context so the shared fetch runs on context.WithoutCancel, still bounded by the artifact client timeout. Mirror jobs don't set the marker, so cancelling a job still aborts its fetches. | ||
simon chrzanowski
|
832a441781 |
Add package routes for private Composer and npm registries (#426)
* Add Composer package routes for private repositories
- Add upstream.composer_routes mapping package name patterns to repositories
- Serve matching packages only from their route, with no fallback to the default repository
- Qualify metadata and artifact cache keys with the route's upstream
- Give routed versions an empty notification-url so installs are not reported to notify-batch
- Document routes in configuration docs, example config and README
* Add npm package routes for private registries
- Add upstream.npm_routes mapping package name patterns to registries
- Serve matching packages only from their route, with no fallback to upstream.npm
- Resolve and validate tarball URLs against the route's registry
- Qualify metadata and artifact cache keys with the route's upstream
- Document npm routes and the audit endpoint limitation
* Share routed artifact filename qualification between handlers
* Match Composer routes without the ~dev suffix
Packagist serves a package's dev versions from {name}~dev.json, and the
handler matched routes against the name with ~dev still attached. An exact
route for example/library therefore missed /p2/example/library~dev.json, and
the private package's dev metadata came from the default repository.
- Match routes against the package name without ~dev, keeping it in the
upstream URL and the cache key so the two files stay apart
- Add a regression test with an exact route
* Keep routed npm publish times out of the shared versions row
The download cooldown check stores publish times under the plain version
PURL, which does not say which registry a version came from. A time read
from the default registry was trusted for a private version of the same
name, so a private release published an hour ago passed a seven-day
cooldown when the public one was a month old.
- Skip the stored row for routed packages, both reading and writing, and
take the publish time from the route's metadata, which is cached under
the route
- This also stops a private time from holding back the public version, and
a route's previous upstream from lending its times to the new one
- Cover adding a route after a public download, the reverse order, and
moving a route to another upstream
|
||
|
pinguinfuss
|
1559be63e2 |
Route containerd ns mirror requests to configured registries (#405)
* Route containerd ns mirror requests to configured registries
containerd's hosts.toml mirrors append ?ns=<registry-host> to every
request. The container handler ignored it, so a single _default mirror
entry could not serve more than one registry.
ns is a closed-world lookup key and is never dialed. Docker Hub aliases
and the host of upstream.oci_default select the default route, hosts of
upstream.oci entries select their named upstream. Unknown hosts return
NAME_UNKNOWN so containerd falls back to its next host. Registry URLs
with a path are not indexed, because ns names the registry at the root
of a host. Host collisions log a warning and resolve deterministically.
With ns the path is the verbatim upstream repository, the reserved
upstream/ prefix is rejected, and repository prefix routes such as
Homebrew's do not apply. Cache names match the unprefixed and
upstream/{name}/ routes, so all routes share cached blobs, manifests and
tag lists. ns is no longer forwarded on tag-list requests, and the
pagination Link is rewritten per request instead of at store time so
clients on different routes get links for their own route.
Refs #303
* Document containerd ns mirroring
Refs #303
* Version the tag-list cache identity for the raw-link format
Tag-list rows now store the upstream Link verbatim and rewrite it per
request. Rows written by earlier versions hold a Link already rewritten
for one route. Under the unchanged key an older binary running next to
this one (rolling update on shared Postgres, or a rollback) would serve
the raw Link unrewritten, sending clients to the wrong route for the
next page. A format marker in the key keeps the two apart at the cost of
one cache miss per tag list after the upgrade.
Adversarial review finding F1: tag-list cache rows hold raw upstream
Links under unchanged keys, so an older binary serves them unrewritten.
* Qualify the README claim about ns mirror coverage
Registries whose URL has a path and hosts shared by two entries are not
reachable through ns; point to the configuration guide for both rules.
Adversarial review finding F3: README says ns covers every upstream.oci
registry.
* Redact registry URLs in ns index warnings
Configured registry URLs may carry userinfo credentials, which the new
startup warnings wrote to the log verbatim. Log them with the password
masked, and say precisely what a path-prefixed default registry means:
its host is not indexed, while the Docker Hub aliases still select it.
Adversarial review finding F2: startup warnings print registry URLs
unredacted and the default-registry message is misleading.
* Keep non-default ports apart in the ns index
Dropping 80 and 443 regardless of scheme turned https://host:80 and
https://host into the same key, so a request for one of them could land
on the other registry. Only the default port of the URL's scheme is
optional now. A configured host is indexed with and without that port,
because image references spell it either way; any other port has to
match exactly, and the ns value is used as containerd sends it, apart
from case and IPv6 brackets.
Since a URL now yields two keys that can belong to different routes,
the collision warning lists the owner per key.
The handler test runs two registries on one host that differ only in
the port. Tests can't bind 80 or 443, so a dialer maps those addresses
to the fake servers.
* Allow upstream/{name}/ together with ns when they agree
containerd sends ns on every request to a mirror host, override_path or
not. Per-registry mirrors pointing at /v2/upstream/{name} therefore
arrive as upstream/{name}/...?ns=<registry>, and refusing every prefixed
name under ns broke them, including the only way to mirror an upstream
whose URL has a path.
Such requests are now handled like the prefix route without ns, as long
as ns names that upstream's own host under the same port rules as the
index. Any other ns on a prefixed name is still NAME_UNKNOWN, so ns
can't be used to create cache entries under another registry's name.
* README: containerd config_path and how to check it
A _default/hosts.toml does nothing while CRI has no hosts directory
configured, so add the config.toml snippet for containerd 2.x and 1.x.
To check the setup, look at containerd config dump and pull with
crictl, then find the request in the proxy log; ctr --hosts-dir reads
the directory on its own and would succeed even without config_path.
Also mention that existing per-registry override_path entries keep
working and that k3s generates the directory itself.
* Let the prefix route take ns for aliases and mirrored registries
With upstream.oci.hub pointing at registry-1.docker.io and a docker.io
hosts.toml mirror on /v2/upstream/hub, containerd sends ns=docker.io,
and the prefix check only accepted the upstream's own host. The same
happens whenever the upstream is a mirror of the registry the nodes
pull from, say an Artifactory remote for ghcr.io: ns carries ghcr.io,
the upstream host is something else, and the pull got a 404.
The prefix already decides where the content comes from and which
cache entries are used, so ns can't redirect anything there. The check
now only refuses an ns that contradicts the prefix, meaning a host the
proxy knows that belongs to another route. The Docker Hub aliases count
as one host, and a host the proxy doesn't know at all is accepted,
since a client can only arrive at the prefix through its own mirror
entry.
* Take docker.io on every prefix route
A Docker Hub mirror configured as a named upstream (mirror.gcr.io, an
Artifactory remote) with a docker.io hosts.toml pointing at
/v2/upstream/hub got a 404 again: docker.io is always in the index for
the default route, so the check saw a known host of another route. That
setup worked before this branch. Docker Hub repository names have
exactly two path components, so docker.io/upstream/... can never be a
real image, and accepting the aliases on every prefix route shadows
nothing.
While here: the collision warning claimed ns requests go to the other
route, which is only true without the prefix, and the README said pulls
for unknown registries always get a 404, which isn't the case for paths
under upstream/{name}/.
* Only take a foreign ns on a prefix route when it is configured
A containerd _default mirror sends the same request for a pull of
unconfigured.example/upstream/mirror/owner/app as a ghcr.io hosts.toml
with override_path does for a mirror upstream. Accepting any host the
proxy didn't know therefore let such a pull be answered with owner/app
from the mirror upstream, a different image, instead of a 404 that lets
containerd go to the registry the image actually names.
The prefix route now accepts ns only for the upstream's own host, for
Docker Hub, and for hosts listed in the new upstream.oci_mirrors setting.
That setting is how an Artifactory remote of ghcr.io keeps working:
upstream:
oci_mirrors:
ghcr: ["ghcr.io"]
The listed hosts also select the upstream for unprefixed ns requests,
which makes a mirror with a path in its URL reachable through the
_default entry as well. Hosts from configured URLs still win over
mirror entries; the proxy warns about such overlaps at startup.
* Fix the docs sentence on which ns a prefix route takes
configuration.md still said a prefix request with ns is only refused
when ns names another route's host. Since the last commit it is the
other way round: only the upstream's own host, its oci_mirrors entries
and Docker Hub get through.
* Note what a host accepted on a prefix route gives up
Listing ghcr.io for an upstream (or the upstream's own host) still has
the ambiguity from the wildcard case: ghcr.io/upstream/ghcr/owner/app
lands on that upstream. That is the price of an explicit entry and
fine in practice, but it should be written down.
* Test that oci_mirrors entries stay with their own upstream
Two gaps in the coverage of the last change: nothing failed if a host
listed for one upstream was also accepted on another upstream's prefix,
and nothing checked that the server actually hands upstream.oci_mirrors
to the container handler. The handler test now has a second upstream
that must refuse ghcr.io, and a small server test pulls through a
mirror entry on both the prefix and the unprefixed route.
* Log when a prefix route refuses an ns
Setups with a ghcr.io hosts.toml pointing at a mirror upstream worked
before this branch and now need an oci_mirrors entry. Without one,
containerd gets a 404, quietly pulls from ghcr.io directly, and nobody
notices the proxy is being skipped. The refusal is now logged as a
warning that names the upstream and the ns and points at
upstream.oci_mirrors.
* Drop a check in validateOCIMirrors that could never fire
user@ghcr.io parses with ghcr.io as its host, so the parsed.Host != host
comparison already rejects it and the extra parsed.User test was dead.
A test case for a host with credentials keeps that path covered.
* Reject an oci_mirrors host that is listed for two upstreams
With ghcr.io under both art and nexus, both prefix routes took ns=ghcr.io
while unprefixed requests went to whichever name sorted first, with
nothing but a startup warning. Config validation now refuses the second
entry, comparing hosts without case and without the default :443.
* Log each refused ns once per upstream and refuse overlong values
The refusal warning fired on every request, and a node with a missing
oci_mirrors entry asks for the manifest and each layer, so the same line
piled up. Each upstream/ns pair is now logged once, for up to 1024
pairs; after that every refusal is logged again.
ns comes straight from the client and can be as long as the request
line allows, so remembering it as is would let anyone pin a lot of
memory with a handful of requests. No registry host is longer than 259
characters (a 253-character name plus a port), so anything longer is
refused right away, before it is logged or remembered.
* Compare oci_mirrors hosts the way the handler does
The duplicate check lowercased the host and trimmed :443, while the
handler splits host and port, brackets IPv6 literals and drops an empty
port. So ghcr.io and ghcr.io:, or [::1] and [::1]:, passed as different
hosts in two upstreams and then collided in the handler. The check now
builds the same key the handler uses.
* Rewrap the oci_mirrors paragraphs in the configuration docs
Two sentences had been dropped into the middle of wrapped lines, and the
prefix-route paragraph named Docker Hub twice in a row. Same content,
shorter lines, one mention.
* Stop logging ns refusals once the set is full
Past 1024 different upstream/ns pairs every refusal was logged again,
so anyone could fill the set with made-up hosts and then flood the log
at warning level. When the set is full the proxy now says so once and
keeps quiet after that. A real misconfiguration shows up long before
that, with the first pull.
* Test the limit on remembered ns refusals
Nothing covered what happens once 1024 pairs are remembered. The new
test fills the set, then sends a known pair and two new ones: the proxy
must log the limit notice exactly once, log none of the refusals and
keep the set at its size.
|
||
Abhinav Gautam
|
8b3b933fd4 |
Stream archives when listing and reading browsed files (#408)
* Stream archives when listing and reading browsed files * Fail browse file responses on archive read errors |
||
Ondrej Kokes
|
129e6c6c1b |
Filter cached packages by ecosystem using pills (#319)
* experimental filtering pills * align data in the dashboard table more nicely (#318) * Bump zizmorcore/zizmor-action from 0.6.2 to 0.6.3 (#333) Bumps [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) from 0.6.2 to 0.6.3. - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits]( |
||
Andrew Nesbitt
|
866e559285 |
Add cooldown package pattern overrides (#407)
* Add cooldown package pattern overrides * Expose full cooldown decisions through pattern policy * Reject unsafe cooldown pattern syntax |
||
montehurd
|
8cc63c40ee |
Cache rewritten npm and Composer metadata (#402)
The npm and Composer handlers rewrite every metadata document they serve so that download URLs point at the proxy: decode the whole document into generic maps, change the URLs, encode it again, and for Composer expand the minified format first. That ran on every request, cached metadata included. With metadata caching on and upstream out of the picture, a cached request still cost 7 ms for @babel/core, 17 ms for symfony/console and 155 ms and 134 MB of allocations for typescript, and an 8 vCPU VM serving cached npm packuments to 20 clients ran the proxy at 450% CPU for about 1,100 requests a second. Rewritten documents are now kept in memory, keyed by the ecosystem, proxy URL, package and a SHA-256 of the raw document, so new bytes from upstream are rewritten again and nothing is served stale. Requests that arrive while a document is being rewritten wait for that rewrite rather than running their own; a waiter leaves when its client does, and the rewrite still completes and is cached. The denylist is fixed at startup, so it needs no place in the key. Cooldown filtering depends on the current time, so with cooldown on the cache is bypassed. metadata_rewrite_cache_size bounds the cache (default "256MB", least recently used out first, "0" to rewrite on every request). NewProxy callers keep the old behaviour unless they set it. Whole cached requests, measured locally: @babel/core 7.1 ms -> 0.38 ms, 54,489 -> 119 allocations typescript 155 ms -> 10.5 ms, 1.37M -> 133 allocations symfony/console 17.4 ms -> 0.48 ms, 165,909 -> 123 allocations What remains is reading the raw document from storage and hashing it. |
||
|
Andrew Nesbitt
|
e7817dc86c |
Merge pull request #393 from wickedOne/analytics-6-source-attribution
[6/6] Attribute requests to a caller and a client tool |
||
|
Andrew Nesbitt
|
4261bcbc3f |
Show the by-client table regardless of ui_request_sources
The flag exists to withhold caller addresses. The by-client table carries none, and the same figures are already public at /metrics, so gating it left two metrics with no tile on the page in the default config. |
||
|
Andrew Nesbitt
|
7c129f9adb |
Merge pull request #391 from wickedOne/analytics-4-stats-breakdown
[4/6] Report the per-ecosystem breakdown from GET /stats |
||
|
Andrew Nesbitt
|
31b73784f5 |
Merge pull request #390 from wickedOne/analytics-3-page
[3/6] Add the /ui/analytics page |
||
|
Andrew Nesbitt
|
9dff45d36c |
Merge remote-tracking branch 'origin/main' into analytics-2-ecosystem-stats
# Conflicts: # README.md |
||
|
Andrew Nesbitt
|
82f5e66d8f |
Merge pull request #388 from wickedOne/analytics-1-route-coverage
[1/6] Name every mounted route in requestEcosystem |
||
|
Andrew Nesbitt
|
2124e8c8ca |
Merge pull request #383 from montehurd/batch-hit-updates
Batch cache hit writes |
||
Daniil Skripchenko
|
b40a50aad8 |
Add storage.cache_artifacts to serve artifacts without storing them (#387)
* Add storage.passthrough to serve artifacts without storing them With storage.passthrough enabled the proxy streams every artifact from upstream to the client and never writes it to storage or the cache database. Metadata filtering, cooldown and the denylist work as before, so the proxy can sit behind another cache (e.g. an Artifactory remote) purely as a policy layer without holding a second copy of every package. Artifacts whose digest is known up front (OCI, Swift, Helm) are verified while streaming. Their responses are sent chunked and the connection is aborted on a mismatch, so a client never receives a tampered artifact as a complete response. Passthrough is rejected together with scanning, direct_serve and mirror_api, and the mirror command refuses to run with it, since all of them depend on stored artifacts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address review: abort truncated streams, guard every cache read - serveArtifact aborts the response on any body read error, and when fewer bytes than the declared size were written, not only on a digest mismatch. A failed or short upstream body in passthrough mode no longer reaches the client as a complete 200. - Streamed upstream read failures are logged and counted as stream_failed upstream errors before the response is aborted. - checkCache reports a miss when artifact caching is off, so every cache read path, including Swift archive HEAD requests, ignores entries stored before the mode was enabled. Denylisted versions are still rejected. - Rename storage.passthrough to storage.cache_artifacts (default true, PROXY_STORAGE_CACHE_ARTIFACTS), which names what actually changes. Tests cover an OCI blob shorter than its Content-Length and an npm tarball whose chunked response ends without the final chunk, both served through the HTTP handlers, plus a Swift HEAD request against a cached archive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rename Proxy.Passthrough to StreamArtifacts Match the storage.cache_artifacts option: the field says what changes (artifacts are streamed instead of stored) and its zero value keeps the usual caching behaviour for every Proxy built without the server config, such as the mirror command and tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
Wicliff Wolda
|
3655000b16 |
Attribute requests to a caller and a client tool
- Track per-caller request and byte counts in a bounded table - Export proxy_client_requests_total and proxy_client_response_bytes_total over a closed label set; addresses are never labels - Gate the sources card behind ui_request_sources, off by default, since /ui carries no authentication of its own - Move access_log.trust_forwarded_for to a top-level key - Keep remote as the TCP peer and add remote_ip alongside it Part 6 of 6 splitting #381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Wicliff Wolda
|
4b8c5a89fb |
Report the per-ecosystem breakdown from GET /stats
- Add downloaded_bytes, downloads and an ecosystems array, served from the snapshot the gauges and the page already share - Add stats_unavailable so a failed aggregation is distinguishable from an idle proxy - Regenerate the OpenAPI spec Part 4 of 6 splitting #381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Wicliff Wolda
|
05015344cb |
Add the /ui/analytics page
- Ring of download size by ecosystem, cache figures, per-ecosystem table, vulnerability overview and a Runtime card - Add metrics.Gather so the page can render counters that were never in the database - Add proxy_response_bytes_total and response-writer byte counting - Serve a retained snapshot behind a staleness banner when the aggregation fails, rather than rendering old figures as current - Extract the security overview into a shared component Part 3 of 6 splitting #381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Wicliff Wolda
|
4ebd805ffc |
Per-ecosystem cache and download statistics
- Add GetEcosystemStats, aggregating packages, versions, artifacts, cache size, downloads and downloaded bytes per ecosystem - Publish six proxy_ecosystem_* gauges from it on the existing one-minute cache-stats tick - Set and selectively delete rather than Reset, so no scrape lands on a half-populated vector - Report artifacts with no package row under "unattributed" rather than dropping them, so the figures still add up Part 2 of 6 splitting #381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Wicliff Wolda
|
fb04bb8dcb |
Name every mounted route in requestEcosystem
- Map /apk to alpine, matching what the handler writes to the database - Report /helm, /homebrew, /generic and /swift under their own names - Test every one of the 23 mounts, so the next route added cannot quietly fall through to "other" Part 1 of 6 splitting #381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
montehurd
|
4827cb9d76 |
Batch cache hit writes
Every cached download wrote its hit in its own UPDATE. SQLite runs on one connection, so each download was a write transaction the rest queued behind, holding a warm npm cache near 1,000 requests a second with the CPU two thirds idle. Hits are now counted in memory and written in one transaction every database.hit_flush_interval, "1s" by default, "0" for a write per hit. A failed write keeps the hits for the next flush, and Close writes what is pending. The same test then served about 2,200 requests a second. Closes #322. |
||
|
Andrew Nesbitt
|
c3540fbcdf |
Merge pull request #380 from abhinavgautam01/fix/response-relay-326
fix: centralize pass-through response relaying |
||
|
Andrew Nesbitt
|
eb4350568c |
Log and count requests aborted mid-relay
A truncated relay panics with http.ErrAbortHandler, which skipped the request log line, the request metrics and the access log entry. |
||
|
montehurd
|
436155e50c |
Clear before deleting on eviction, and requeue failed reclaims
Eviction deleted an object before checking its record still pointed at it. If a refetch had moved the record, that object was queued and a request that read the record earlier could still be opening it. Eviction now clears first, skips the delete if the record moved, and queues the object when the delete fails after a clear. Reclaim queues a failed delete again, which moves it behind the rest, so objects the backend keeps refusing cannot fill every batch. Also drop the clearArtifactCache passthrough and ClearCachedArtifact's unused context, note that only tests call ArtifactPath, and document that max_size does not count objects waiting to be reclaimed. |
||
|
abhinavgautam01
|
ac261cc34d |
fix: centralize pass-through response relaying | ||
|
montehurd
|
4ce6c37bd9 |
Give each fetch its own storage path
Every fetch of an artifact wrote one path, so fetches with different URLs
or digests, which do not share a coalescing key, could overwrite each
other's bytes, serve the wrong ones, or delete them on a digest mismatch
while the other was about to open them.
storeArtifact now writes {ecosystem}/{name}/{version}/{fetch id}/{filename},
with a random id per fetch. Most ecosystems learn the digest only after
the fetch, and storage has no rename, so the id is the one rule that fits
all of them. Existing records keep their paths and stay readable. On a
file:// bucket, Delete now removes an emptied fetch directory, which
fileblob leaves behind, and no other directory, since another fetch may
be creating its own inside it.
A path a record stops pointing at is queued in pending_deletes rather
than deleted, since a request that read the record may still open it. A
loop deletes queued paths after max(1h, direct_serve_ttl), whether or not
max_size is set. UpsertArtifact and clears apply only if the record still
holds the path the caller read, so racing commits each queue the path
they replaced and a clear never orphans a newer commit. Eviction still
deletes inline, and counts space as freed only when the record still
pointed at what it deleted.
|
||
|
montehurd
|
50e5cc334e |
Merge branch 'git-pkgs:main' into bound-eviction-pass | ||
|
Andrew Nesbitt
|
65f2bf52b1 |
Widen wall-clock bound in flaky probe timeout test
Loaded CI runners exceeded the 500ms bound; 5s still proves the probe timeout fired instead of blocking on Store. |
||
|
montehurd
|
1001cd772b |
End an eviction pass that cannot evict anything
A failed delete leaves the record eligible, so the same batch came back on the next iteration while the recorded size never dropped. Against a backend refusing deletes the pass retried forever, burning CPU and flooding the log. A canceled context made it faster: every delete fails at once, so the loop spun as quickly as the database could answer. The pass now ends when a batch clears no record, and returns as soon as the context is canceled. Progress counts records rather than bytes, because a cached record can have no size recorded. A pass can therefore end with the cache still above its limit, leaving the records that failed for the next pass to retry. |
||
|
Andrew Nesbitt
|
e28745742c |
Merge pull request #353 from andrewmrich/arich/debian-repositories
enable multiple debian repositories |
||
|
Andrew Nesbitt
|
1da2dc0bf9 |
Merge pull request #359 from abhinavgautam01/fix/helm-oci-index-320
fix(helm): support OCI references in HTTP repository indexes |
||
|
abhinavgautam01
|
548a50d56e |
feat: add exact-version package denylist | ||
|
abhinavgautam01
|
429e7458e4 |
fix(helm): support OCI references in HTTP repository indexes | ||
Andrew Rich
|
8805cd9283 |
Merge branch 'git-pkgs:main' into arich/debian-repositories | ||
|
Andrew Nesbitt
|
0bab9bd0af |
Separate OCI request timeout from readiness probes in loopback test (#347)
The 250ms client is meant for the readiness poll, where a timeout is retried. Reusing it for the OCI manifest request makes the test flake under -race on Windows CI when fetch and cache I/O take longer, as seen on #328. The request checks upstream routing, not latency. |
||
|
pinguinfuss
|
be2a1f05d2 |
fix(server): tune the shared upstream transport defaults (#351)
* fix(server): tune the shared upstream transport defaults server.serve builds the shared client with safehttp.New, which clones Go's default transport: MaxIdleConnsPerHost stays 0 (an effective limit of two idle connections per host) and ResponseHeaderTimeout stays 0. Handing that client to fetch.NewFetcher via fetch.WithHTTPClient replaces the fetcher's own defaults of 10 idle connections per host and a 60-second response-header timeout. Set both on the shared transport, matching the fetcher defaults: a second burst of concurrent cache misses to one registry now reuses its connections instead of re-dialling most of them, and an upstream that accepts a request but stalls before sending headers is cut off after 60 seconds rather than only by the client's overall timeout. Tests measure connection reuse across two concurrent bursts against a TLS upstream that counts accepted connections (Go default: at most two reused; tuned: all eight) and assert that a stall before headers fails with the response-header timeout. Fixes #327 * fix(server): reconcile http_timeout docs and tighten the transport tests The http_timeout documentation and the config comment said "0" disables the upstream timeout entirely. With a fixed 60-second ResponseHeaderTimeout on the shared transport that is no longer the whole story, so both now say that waiting for response headers stays bounded independently of the setting. Test cleanup from review: the 50ms settle between bursts was dead time (the transport returns a connection to the idle pool before the body's final Read returns, so burst returning already means the pool is settled); the maxNewInBurst sentinel became explicit min/max bounds per case; the stall test dropped the client.Timeout override and the elapsed-time assertion, which was redundant with the error-text check in any realistic run and whose failure message misattributed the cause, and its comment now says plainly that the field assertions pin production while the behavioural half runs at a lowered timeout. * test(server): hold each burst at the upstream instead of sleeping The reuse test kept a burst in flight with a 100ms handler sleep, so a process stall longer than that between spawning the goroutines and their dials let a request finish early and hand its connection to a sibling. Review reproduced this with forced stalls: the default-transport case then dialled 5 instead of 6 connections. The handler now answers only once burstSize requests are waiting at the same time. With HTTP/1.1 pinned that forces every burst onto burstSize distinct connections regardless of scheduling, and the measured counts stay exactly 6 new for Go's default and 0 for the tuned transport, also under the same forced stalls. |
||
|
Andrew Rich
|
8a51d04011 |
feat(debian): serve additional APT archives at /debian/{name}/
A Debian release is served by more than one archive. Security updates live on
a different host than the main archive, so a single upstream.debian URL cannot
serve a complete suite set and -security suites are unreachable.
Add upstream.debian_repositories, a name-to-URL map served at /debian/{name}/,
modelled on upstream.apk. The field is additive: upstream.debian keeps serving
/debian/pool/... and /debian/dists/... with unchanged cache identities, so
existing deployments and their warm caches are unaffected, and the scalar
field and PROXY_UPSTREAM_DEBIAN are untouched.
Named repositories scope both caches by name, since the same filename can hold
different bytes in different archives. Metadata keys are hashed over name,
upstream URL, and path, as APKHandler does. Names are validated through
validateNamedUpstreams, and "pool" and "dists" are refused because they would
shadow the main archive's own paths.
An unconfigured first path segment stays a main-archive path rather than
returning 404 as the APK handler does: the main archive is unnamed and serves
paths of its own at the root.
|
||
|
Ondrej Kokes
|
94c11b4b3b |
align data in the dashboard table more nicely (#318) | ||
|
Ondrej Kokes
|
bb2205ad76 |
docker host is trimmed too much in the docs (#317) | ||
|
Andrew Nesbitt
|
b67cfb1014 |
Add Homebrew JSON API and bottle proxy support (#254)
* Add Homebrew JSON API and bottle proxy support * Fix Homebrew HEAD offline fallback and non-sha256 OCI manifest handling Route Homebrew API HEAD requests through ProxyCached so a warm cache answers without an upstream call and stale entries are served when the upstream is unreachable. HEAD still reaches upstream as HEAD when metadata caching is disabled. Limit OCI manifest digest verification to sha256 references and Docker-Content-Digest headers so other digest algorithms are proxied instead of rejected, and log the failing expected value. * Reconcile with #280 and #301 after rebase Compute real manifest digests in #280's fixture upstreams so the new verification accepts them, and add headerETag / headerLastModified to * Send fixed Accept for Homebrew API and match If-None-Match properly The Homebrew API cache key does not include Accept, so replaying the client header could serve one representation under another; the API does not negotiate anyway. Compare If-None-Match with weak comparison, list splitting and "*" per RFC 7232 instead of string equality, and apply the same helper to the metadata and swift responders. * Reconcile with #298 and #304 after rebase Move the configureScanning doc comment back to its function after the auto-merge stacked it on mountProtocolHandlers, and drop the second ETag/Last-Modified set in writeMetadataCachedResponse now that the pre-304 set covers both response paths. |
||
Giles Westwood
|
30e54a1e0c |
Add generic HTTP download proxy for GitHub release assets (mise/aqua) (#302)
* Add generic HTTP download proxy for GitHub release assets
Adds a /generic/{name}/ route backed by a new upstream.generic named-upstream
map, so tools that download from fixed URL shapes (mise's aqua backend
fetching GitHub release assets, and its tag lookups on api.github.com) can be
pointed at the proxy with client-side URL rewriting. Only configured
upstreams are reachable, so this is not an open HTTP proxy.
Paths shaped like {owner}/{repo}/releases/download/{tag}/{asset} are
version-pinned and go through the artifact cache: fetched once, hashed,
served without revalidation, and still served when the upstream is down.
Every other path goes through the metadata cache with the client's Accept
header and query string replayed, so API responses are fresh within
metadata_ttl, revalidated after that, and served stale when the upstream
fails or rate-limits the request.
Tests cover path classification, unknown upstreams and traversal, cache
hits with the upstream down, HEAD, 404 pass-through, Accept and query
forwarding, stale-on-429, cache isolation between upstreams, and that an
upstream token scoped to the release host is not sent to the object store
it redirects to.
Closes #183.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDjDeq27CKzEP2o3GWBY7F
* Use fixed Accept header for generic metadata
---------
Co-authored-by: Giles Westwood <giles@gileswestwood.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
|
||
aarnaud
|
5499837084 |
Add pre-cache artifact scanning hook (trivy/ClamAV/Wiz/custom) (#298)
Runs fetched artifacts through pluggable external scanners after they're staged in storage but before they're committed to the cache DB, so a block verdict deletes the object instead of ever exposing it to a client. Scanners pull the staged bytes themselves via a short-lived HMAC-signed internal route rather than the proxy pushing bytes to them, keeping the mechanism storage-backend-agnostic and avoiding uploading potentially huge artifacts through the proxy's own egress. Hardening baked in from the start: the internal scan-fetch route is gated both at router-mount time and in the handler so it's inert whenever scanning is disabled or unsigned; the signing key is mandatory whenever scanning is enabled, enforced directly in scanner.NewGroup rather than relying on callers to invoke config validation; the scan call and the delete-on-block cleanup both run on a context detached from the client's, so a client disconnecting mid-scan can't be mistaken for a scanner failure, doesn't cause a legitimate artifact to be deleted, and doesn't leave a genuinely blocked artifact's bytes orphaned in storage; and scanner infrastructure errors (connection failures, internal hostnames) are never forwarded verbatim to anonymous clients, only a generic message. The scan-error metric also correctly distinguishes a scanner's own timeout from being cancelled because a sibling scanner already decided the verdict. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Andrew Nesbitt
|
7743417c72 |
Add Swift package registry support (#259)
* Add Swift package registry support * Fix Swift archive integrity handling * Fix Swift registry pagination and archive HEAD requests * Canonicalize Swift package identifiers * Handle Swift registry cache identities * Use stored PURLs for cache lookups * Address review: upstream-hash refetch and cache PURL cleanup - Re-fetch instead of 502 when a cached artifact's hash disagrees with the upstream-declared checksum; log and discard the stale entry. - Rename expectedHash to upstreamHash and document how the check differs from checkCache's stream integrity verification. - Drop the repository_url qualifier from swift cache PURLs so cache entries survive an upstream.swift change, matching other ecosystems. - Pass name to handleSourceArchiveHead instead of re-deriving it. * Drop BulkCheckVulnerabilities coverage after #279 removed it The rebase over #279 (dead-code cleanup) drops BulkCheckVulnerabilities; remove the tests and helper that exercised the swift-identity-filtering path through it, and the imports they pulled in. |
||
wickedOne
|
cad1a9226a |
Report upstream circuit breaker state in /health and /metrics (#275)
* Report upstream circuit breaker state in /health and /metrics * applied requested changes * apply review changes * Bumped github.com/git-pkgs/registries to v0.9.0 |
||
|
Andrew Nesbitt
|
b5ee6dd96c |
Add ECR auto-refreshing upstream authentication (#278)
* Add ECR auto-refreshing upstream authentication - Add "ecr" auth type to upstream.auth config with optional region - Cache ecr:GetAuthorizationToken results per region and refresh shortly before expiry via the AWS SDK default credential chain - Route type: ecr through the token cache in Server.authForURL - Document the new type in config.example.yaml and docs/configuration.md Fixes #276 * Collapse ecrTokens.header to a single return path Drops the internal/server package below the goconst min-occurrences threshold for the Authorization literal. * Coalesce concurrent ECR token fetches with singleflight Concurrent cache misses for the same region now share a single GetAuthorizationToken call instead of each issuing their own, avoiding a request burst against the ECR API at cold start and at each 12-hour refresh. golang.org/x/sync is already a direct dependency. * Improve ECR token refresh and region inference * Back off failed ECR token refreshes |