Eviction deleted an object before checking its record still pointed at
it. If a refetch had moved the record, that object was queued and a
request that read the record earlier could still be opening it. Eviction
now clears first, skips the delete if the record moved, and queues the
object when the delete fails after a clear.
Reclaim queues a failed delete again, which moves it behind the rest, so
objects the backend keeps refusing cannot fill every batch.
Also drop the clearArtifactCache passthrough and ClearCachedArtifact's
unused context, note that only tests call ArtifactPath, and document that
max_size does not count objects waiting to be reclaimed.
Every fetch of an artifact wrote one path, so fetches with different URLs
or digests, which do not share a coalescing key, could overwrite each
other's bytes, serve the wrong ones, or delete them on a digest mismatch
while the other was about to open them.
storeArtifact now writes {ecosystem}/{name}/{version}/{fetch id}/{filename},
with a random id per fetch. Most ecosystems learn the digest only after
the fetch, and storage has no rename, so the id is the one rule that fits
all of them. Existing records keep their paths and stay readable. On a
file:// bucket, Delete now removes an emptied fetch directory, which
fileblob leaves behind, and no other directory, since another fetch may
be creating its own inside it.
A path a record stops pointing at is queued in pending_deletes rather
than deleted, since a request that read the record may still open it. A
loop deletes queued paths after max(1h, direct_serve_ttl), whether or not
max_size is set. UpsertArtifact and clears apply only if the record still
holds the path the caller read, so racing commits each queue the path
they replaced and a clear never orphans a newer commit. Eviction still
deletes inline, and counts space as freed only when the record still
pointed at what it deleted.
A failed delete leaves the record eligible, so the same batch came back
on the next iteration while the recorded size never dropped. Against a
backend refusing deletes the pass retried forever, burning CPU and
flooding the log. A canceled context made it faster: every delete fails
at once, so the loop spun as quickly as the database could answer.
The pass now ends when a batch clears no record, and returns as soon as
the context is canceled. Progress counts records rather than bytes,
because a cached record can have no size recorded. A pass can therefore
end with the cache still above its limit, leaving the records that
failed for the next pass to retry.
The 250ms client is meant for the readiness poll, where a timeout is
retried. Reusing it for the OCI manifest request makes the test flake
under -race on Windows CI when fetch and cache I/O take longer, as
seen on #328. The request checks upstream routing, not latency.
* fix(server): tune the shared upstream transport defaults
server.serve builds the shared client with safehttp.New, which clones
Go's default transport: MaxIdleConnsPerHost stays 0 (an effective limit
of two idle connections per host) and ResponseHeaderTimeout stays 0.
Handing that client to fetch.NewFetcher via fetch.WithHTTPClient
replaces the fetcher's own defaults of 10 idle connections per host and
a 60-second response-header timeout.
Set both on the shared transport, matching the fetcher defaults: a
second burst of concurrent cache misses to one registry now reuses its
connections instead of re-dialling most of them, and an upstream that
accepts a request but stalls before sending headers is cut off after 60
seconds rather than only by the client's overall timeout.
Tests measure connection reuse across two concurrent bursts against a
TLS upstream that counts accepted connections (Go default: at most two
reused; tuned: all eight) and assert that a stall before headers fails
with the response-header timeout.
Fixes#327
* fix(server): reconcile http_timeout docs and tighten the transport tests
The http_timeout documentation and the config comment said "0" disables
the upstream timeout entirely. With a fixed 60-second
ResponseHeaderTimeout on the shared transport that is no longer the
whole story, so both now say that waiting for response headers stays
bounded independently of the setting.
Test cleanup from review: the 50ms settle between bursts was dead time
(the transport returns a connection to the idle pool before the body's
final Read returns, so burst returning already means the pool is
settled); the maxNewInBurst sentinel became explicit min/max bounds per
case; the stall test dropped the client.Timeout override and the
elapsed-time assertion, which was redundant with the error-text check
in any realistic run and whose failure message misattributed the cause,
and its comment now says plainly that the field assertions pin
production while the behavioural half runs at a lowered timeout.
* test(server): hold each burst at the upstream instead of sleeping
The reuse test kept a burst in flight with a 100ms handler sleep, so a
process stall longer than that between spawning the goroutines and
their dials let a request finish early and hand its connection to a
sibling. Review reproduced this with forced stalls: the default-transport
case then dialled 5 instead of 6 connections.
The handler now answers only once burstSize requests are waiting at the
same time. With HTTP/1.1 pinned that forces every burst onto burstSize
distinct connections regardless of scheduling, and the measured counts
stay exactly 6 new for Go's default and 0 for the tuned transport, also
under the same forced stalls.
A Debian release is served by more than one archive. Security updates live on
a different host than the main archive, so a single upstream.debian URL cannot
serve a complete suite set and -security suites are unreachable.
Add upstream.debian_repositories, a name-to-URL map served at /debian/{name}/,
modelled on upstream.apk. The field is additive: upstream.debian keeps serving
/debian/pool/... and /debian/dists/... with unchanged cache identities, so
existing deployments and their warm caches are unaffected, and the scalar
field and PROXY_UPSTREAM_DEBIAN are untouched.
Named repositories scope both caches by name, since the same filename can hold
different bytes in different archives. Metadata keys are hashed over name,
upstream URL, and path, as APKHandler does. Names are validated through
validateNamedUpstreams, and "pool" and "dists" are refused because they would
shadow the main archive's own paths.
An unconfigured first path segment stays a main-archive path rather than
returning 404 as the APK handler does: the main archive is unnamed and serves
paths of its own at the root.
* Add Homebrew JSON API and bottle proxy support
* Fix Homebrew HEAD offline fallback and non-sha256 OCI manifest handling
Route Homebrew API HEAD requests through ProxyCached so a warm cache
answers without an upstream call and stale entries are served when the
upstream is unreachable. HEAD still reaches upstream as HEAD when
metadata caching is disabled.
Limit OCI manifest digest verification to sha256 references and
Docker-Content-Digest headers so other digest algorithms are proxied
instead of rejected, and log the failing expected value.
* Reconcile with #280 and #301 after rebase
Compute real manifest digests in #280's fixture upstreams so the new
verification accepts them, and add headerETag / headerLastModified to
* Send fixed Accept for Homebrew API and match If-None-Match properly
The Homebrew API cache key does not include Accept, so replaying the
client header could serve one representation under another; the API
does not negotiate anyway. Compare If-None-Match with weak comparison,
list splitting and "*" per RFC 7232 instead of string equality, and
apply the same helper to the metadata and swift responders.
* Reconcile with #298 and #304 after rebase
Move the configureScanning doc comment back to its function after the
auto-merge stacked it on mountProtocolHandlers, and drop the second
ETag/Last-Modified set in writeMetadataCachedResponse now that the
pre-304 set covers both response paths.
* Add generic HTTP download proxy for GitHub release assets
Adds a /generic/{name}/ route backed by a new upstream.generic named-upstream
map, so tools that download from fixed URL shapes (mise's aqua backend
fetching GitHub release assets, and its tag lookups on api.github.com) can be
pointed at the proxy with client-side URL rewriting. Only configured
upstreams are reachable, so this is not an open HTTP proxy.
Paths shaped like {owner}/{repo}/releases/download/{tag}/{asset} are
version-pinned and go through the artifact cache: fetched once, hashed,
served without revalidation, and still served when the upstream is down.
Every other path goes through the metadata cache with the client's Accept
header and query string replayed, so API responses are fresh within
metadata_ttl, revalidated after that, and served stale when the upstream
fails or rate-limits the request.
Tests cover path classification, unknown upstreams and traversal, cache
hits with the upstream down, HEAD, 404 pass-through, Accept and query
forwarding, stale-on-429, cache isolation between upstreams, and that an
upstream token scoped to the release host is not sent to the object store
it redirects to.
Closes#183.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDjDeq27CKzEP2o3GWBY7F
* Use fixed Accept header for generic metadata
---------
Co-authored-by: Giles Westwood <giles@gileswestwood.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Runs fetched artifacts through pluggable external scanners after they're
staged in storage but before they're committed to the cache DB, so a
block verdict deletes the object instead of ever exposing it to a
client. Scanners pull the staged bytes themselves via a short-lived
HMAC-signed internal route rather than the proxy pushing bytes to them,
keeping the mechanism storage-backend-agnostic and avoiding uploading
potentially huge artifacts through the proxy's own egress.
Hardening baked in from the start: the internal scan-fetch route is
gated both at router-mount time and in the handler so it's inert
whenever scanning is disabled or unsigned; the signing key is mandatory
whenever scanning is enabled, enforced directly in scanner.NewGroup
rather than relying on callers to invoke config validation; the scan
call and the delete-on-block cleanup both run on a context detached
from the client's, so a client disconnecting mid-scan can't be mistaken
for a scanner failure, doesn't cause a legitimate artifact to be
deleted, and doesn't leave a genuinely blocked artifact's bytes
orphaned in storage; and scanner infrastructure errors (connection
failures, internal hostnames) are never forwarded verbatim to anonymous
clients, only a generic message. The scan-error metric also correctly
distinguishes a scanner's own timeout from being cancelled because a
sibling scanner already decided the verdict.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Add Swift package registry support
* Fix Swift archive integrity handling
* Fix Swift registry pagination and archive HEAD requests
* Canonicalize Swift package identifiers
* Handle Swift registry cache identities
* Use stored PURLs for cache lookups
* Address review: upstream-hash refetch and cache PURL cleanup
- Re-fetch instead of 502 when a cached artifact's hash disagrees with
the upstream-declared checksum; log and discard the stale entry.
- Rename expectedHash to upstreamHash and document how the check
differs from checkCache's stream integrity verification.
- Drop the repository_url qualifier from swift cache PURLs so cache
entries survive an upstream.swift change, matching other ecosystems.
- Pass name to handleSourceArchiveHead instead of re-deriving it.
* Drop BulkCheckVulnerabilities coverage after #279 removed it
The rebase over #279 (dead-code cleanup) drops BulkCheckVulnerabilities;
remove the tests and helper that exercised the swift-identity-filtering
path through it, and the imports they pulled in.
* Add ECR auto-refreshing upstream authentication
- Add "ecr" auth type to upstream.auth config with optional region
- Cache ecr:GetAuthorizationToken results per region and refresh
shortly before expiry via the AWS SDK default credential chain
- Route type: ecr through the token cache in Server.authForURL
- Document the new type in config.example.yaml and docs/configuration.md
Fixes#276
* Collapse ecrTokens.header to a single return path
Drops the internal/server package below the goconst min-occurrences
threshold for the Authorization literal.
* Coalesce concurrent ECR token fetches with singleflight
Concurrent cache misses for the same region now share a single
GetAuthorizationToken call instead of each issuing their own, avoiding
a request burst against the ECR API at cold start and at each 12-hour
refresh. golang.org/x/sync is already a direct dependency.
* Improve ECR token refresh and region inference
* Back off failed ECR token refreshes
* Add Alpine APK repository proxy support
- Serve named APK repositories at /apk/{repository}/ with the official
Alpine mirror as the default repository
- Cache v2 APKINDEX.tar.gz and v3 Packages.adb indexes and detached
signatures via the metadata cache, serving stored bytes unchanged so
apk signature verification keeps working
- Cache .apk packages in the shared artifact cache keyed by the full
repository path, since APK filenames do not include the architecture
- Add configurable upstream repositories via upstream.apk with
validation, plus dashboard registry instructions
- Add tests for index/signature byte fidelity, per-arch caching, cache
hits, offline reads, upstream authentication, and 404 handling
- Document apk usage in README, config example, and configuration docs
* Serve APK package HEAD requests without a body
Use the method-aware serveArtifact helper (as container.go does) so HEAD
responses carry Content-Length but omit the body; add a regression test.
* Drop doubled blank line in docs/configuration.md
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
- Request application/json from the npm upstream when the option is set,
independent of cooldown, so served packuments carry the "time" map
- Wire the option through the shared Proxy struct and the
PROXY_UPSTREAM_NPM_FULL_METADATA environment override
- Document it in config.example.yaml and docs/configuration.md
- Test that the option forces full metadata with cooldown disabled
* Add GCS storage backend with Workload Identity support
Register gocloud.dev/blob/gcsblob so gs:// URLs are accepted as a storage
backend. Authentication uses Application Default Credentials, which makes
GKE Workload Identity work out of the box; signed URLs (direct_serve)
fall back to the IAM Credentials signBlob API when no private key is
available.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Replace gcsblob with lightweight GCS backend
* Extract GCS client into standalone module
---------
Co-authored-by: Anthony A. <github@anthony-arnaud.fr>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Every built-in ecosystem upstream can now be set via the upstream
config block or PROXY_UPSTREAM_* env vars, with allow_private_hosts
and allow_loopback controlling access to non-public addresses.
* Bump go tool golangci-lint to v2.13.1
The .golangci.yml goconst.ignore-tests setting was added in v2.12.0
(golangci/golangci-lint#6480). On the previously pinned v2.10.1,
config verify fails with "additional properties 'ignore-tests' not
allowed" and the setting is silently ignored at run time, so goconst
counts test-file literals toward min-occurrences.
* Apply gofmt and CutSuffix simplification
- gofmt -w internal/server/health_test.go
- Replace HasSuffix+TrimSuffix with CutSuffix in ParseSize
* Remove dead code and migrate tests off legacy Filesystem storage
Migrate the three test call sites of storage.NewFilesystem to
storage.OpenBucket("file://...") and drop the deprecated
StorageConfig.Path field from test configs, then delete code that
deadcode reports as unreachable from cmd/proxy:
- internal/storage/filesystem.go and its tests
- storage.HashingReader
- enrichment.Service.BulkCheckVulnerabilities and NormalizeLicense
- server.ActiveRequestsMiddleware (no-op body; the real tracking
is the inline r.Use at server.go:226)
- mirror.RegistrySource (unimplemented stub)
metrics.UpdateCircuitBreakerState and RecordCircuitBreakerTrip are
kept because #275 wires them.
Update the CONTRIBUTING.md storage section to reflect blob.go.
* Show build information in web UI
Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
* Fix footer build info shadowed by page Version fields
Shared footer templates were reading .Version and .Commit, which resolve
to package data on VersionShowData and BrowseSourceData. Point the footer
at Layout.BuildInfo and cover both pages so the proxy version stays visible.
Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix Layout build info field promotion
---------
Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Refactor LoadFromEnv to use helpers
Avoid triggering the linter about the cyclomatic complexity of the LoadFromEnv
function in later changes by refactoring it to use setEnvString/setEnvBool
helpers.
No functional change, just collapse ~29 repetitive if-blocks into single-line
calls to two small helpers.
* Make Debian upstream repository configurable
Support overriding the Debian handler's upstream (e.g. Ubuntu archives)
via PROXY_UPSTREAM_DEBIAN or upstream.debian in the config file.
Tested with PROXY_UPSTREAM_DEBIAN=http://archive.ubuntu.com/ubuntu
to get Ubuntu Resolute packages.
* resolve name and version for PEP 658 metadata sidecars
* fix(pypi): parse Windows installer and egg filenames separately
The bdist_wininst and bdist_msi layout joins the platform to the version
with a '.' rather than a '-', so treating .exe/.msi like a wheel folded
the platform into the version: foo-1.0.win32-py2.0.exe resolved to
version "1.0.win32". Eggs shared the problem, as setuptools' hyphen
escaping is not universal: aws-sdk-1.0.0-py3.11.egg resolved to name
"aws", version "sdk".
Give each format its own parser. Wheels keep the PEP 427
spec-guaranteed field positions, eggs locate the version relative to the
py{X.Y} interpreter field, and Windows installers strip the platform and
interpreter fields before splitting name from version.
A PEP 658 sidecar resolves to the same name and version as the
distribution it describes, so it is cached under that version. Browse and
compare took the first cached artifact without checking its extension,
handing openArchive plain text: a version pip had only fetched metadata
for reported hasCached and then 500'd.
Add firstBrowsableArtifact, replacing five duplicated selection loops,
and export PyPIMetadataSuffix so the suffix has a single definition.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add http_timeout config option (and PROXY_HTTP_TIMEOUT env var) to set
the per-request timeout on the shared HTTP client used by protocol
handlers for upstream metadata fetches and pass-through file requests.
Defaults to the previous hardcoded 30s; "0" disables the timeout.
Fixes#187
The startup log and /stats endpoint always reported Database.Path,
which is the sqlite default even when PROXY_DATABASE_DRIVER=postgres
is set and a postgres URL is in use. Add DatabaseConfig.String() that
returns the sqlite path or the postgres URL with the password redacted,
and use it in both places.
Fixes#173
- Move third-party JS into static/vendor/ and add lucide for icons; refine
.gitignore so embedded vendor dirs aren't caught by the Go vendor rule.
- Replace folder/file emojis in the source browser with lucide icons.
- Wrap the header logo and title in a single anchor so the icon is clickable.
- Drop the redundant "Powered by git-pkgs" footer block; add a GitHub repo
link to the About column and bump the ecosystem count from 16+ to 17+.
- Sticky footer pattern: body is min-h-full flex column with main growing to
fill, so the footer sits at the bottom of short pages.
- Hamburger menu under md: search and nav links collapse into a drawer
toggled by a menu button; theme toggle stays visible at both sizes.
Adds UIBaseURL (env PROXY_UI_URL), advertised separately from BaseURL
for deployments where the UI is reached on a public domain while build
machines hit a Docker network alias for the package endpoints. Defaults
to BaseURL when unset.
Consumers:
- Logged alongside base_url at startup.
- <link rel="canonical"> and og:url / og:title / og:site_name in every
UI page, omitted when UIBaseURL is empty.
- Banner on the install guide when UIBaseURL differs from BaseURL,
clarifying that the URLs in the snippets are the package endpoint and
that the UI itself lives elsewhere.
Docs call out that the proxy serves UI and package endpoints on the same
listener, so reverse proxies fronting the UI publicly must restrict the
public route to PathPrefix(/ui) to avoid exposing /npm, /pypi, etc.
nginx and Traefik examples both show the path-split pattern.
* Mount web UI under /ui (closes#123)
The UI now lives under /ui so reverse proxies can apply different
access rules to it (e.g. require auth) while leaving the package
endpoints (/npm, /pypi, /v2, ...) open to build machines.
- GET / redirects to /ui/
- /api/browse and /api/compare move to /ui/api/browse and
/ui/api/compare since only the browser JS calls them
- /health, /stats, /metrics, /openapi.json and /api/* stay at root
* README: nginx example for splitting UI auth from package endpoints
The /ui prefix lets a reverse proxy apply different access rules to the
web UI than to the package endpoints. Adds the snippet that motivated
the prefix change so deployers don't have to derive it.
* Bump github.com/go-chi/chi/v5 from 5.2.5 to 5.3.0
Bumps [github.com/go-chi/chi/v5](https://github.com/go-chi/chi) from 5.2.5 to 5.3.0.
- [Release notes](https://github.com/go-chi/chi/releases)
- [Changelog](https://github.com/go-chi/chi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/go-chi/chi/compare/v5.2.5...v5.3.0)
---
updated-dependencies:
- dependency-name: github.com/go-chi/chi/v5
dependency-version: 5.3.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Drop deprecated middleware.RealIP
chi 5.3.0 deprecated middleware.RealIP because it trusts X-Forwarded-For,
True-Client-IP, and X-Real-IP unconditionally, which is spoofable when
the service is not strictly behind a trusted proxy (GHSA-3fxj-6jh8-hvhx,
GHSA-rjr7-jggh-pgcp, GHSA-9g5q-2w5x-hmxf). The only consumer of
r.RemoteAddr in this codebase is the request log; no auth, rate limiting,
or other security decisions depend on it, so removing the middleware is
safe. If we ever need real client IPs in logs behind an LB, add a
trusted-proxy-aware middleware then.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
