The history is process-wide, and tests cleared it only after running, and
not all of them. Records from earlier tests or an earlier -count repetition
reached later assertions: TestRecordUpstreamOperations left a record for
TestRecentFailuresNewestFirst, and the server tests never cleared it.
- Export ResetRecentFailures so tests outside the metrics package can use it
- Clear the history before and after every test that records or reads
failures directly, in metrics, server and handler
- Assert exactly one failure in the handler tests instead of looking only
at the newest
- Failure recorders take a FailureDetail: PURL, filename, URL, storage path, error
- Keep the last 20 failures per counter series in memory, reset on restart
- Strip userinfo, query and fragment from URLs, including those inside error text
- Pass the detail from upstream, storage, integrity, health probe and scan call sites
- Package names and URLs stay out of Prometheus labels
* feat(storage): expose efficient seek for local files
Open file-backed artifacts with os.Open so local readers support
efficient seeking. Leave cloud storage readers unchanged.
Keep the legacy sidecar mapping test focused on fileblob's reader:
Blob.Open now reads local files directly and no longer consults the
sidecar. Continue checking sidecar cleanup and post-store reads.
* feat(handler): preserve efficient seek through integrity checks
Keep the cache result's single Reader field and preserve io.Seeker
when the underlying storage reader supports efficient seeking.
Sequential reads continue through whole-object integrity verification.
A successful seek switches subsequent reads to the source for range
responses, where full-object verification is not possible.
Test both modes and verify the source is closed once.
* feat(handler): add request-aware byte range responses
Add a request-aware artifact serving helper for bounded, open-ended,
and suffix byte ranges, including If-Range handling, 206 responses,
and 416 responses for valid unsatisfiable ranges.
Ignore malformed and multi-range requests by serving the full artifact.
Advertise byte-range support only for seekable readers, preserving HEAD,
redirect, and non-seekable behavior. Clarify that net/http recovers
ErrAbortHandler per request and keeps the server running.
* feat(handler): enable byte ranges across artifact downloads
Forward each artifact request to the shared range-aware response helper,
including cached generic release assets and OCI blobs. This lets the
handler honor ranges only when its reader supports efficient seeking,
without changing direct-storage redirect behavior.
Add warm-cache endpoint tests that assert partial response headers and
bytes without an upstream refetch. Document supported range behavior,
unsupported reader fallback, and the integrity limitation of partial reads.
Every fetch of an artifact wrote one path, so fetches with different URLs
or digests, which do not share a coalescing key, could overwrite each
other's bytes, serve the wrong ones, or delete them on a digest mismatch
while the other was about to open them.
storeArtifact now writes {ecosystem}/{name}/{version}/{fetch id}/{filename},
with a random id per fetch. Most ecosystems learn the digest only after
the fetch, and storage has no rename, so the id is the one rule that fits
all of them. Existing records keep their paths and stay readable. On a
file:// bucket, Delete now removes an emptied fetch directory, which
fileblob leaves behind, and no other directory, since another fetch may
be creating its own inside it.
A path a record stops pointing at is queued in pending_deletes rather
than deleted, since a request that read the record may still open it. A
loop deletes queued paths after max(1h, direct_serve_ttl), whether or not
max_size is set. UpsertArtifact and clears apply only if the record still
holds the path the caller read, so racing commits each queue the path
they replaced and a clear never orphans a newer commit. Eviction still
deletes inline, and counts space as freed only when the record still
pointed at what it deleted.
* Return the stored artifact from storeArtifact, not a reader
storeArtifact returned a CacheResult holding an open file handle. A
handle has one read position, so it can only ever serve a single caller,
which is what blocks sharing one fetch between concurrent requests.
Return the artifact and its storage path instead, and let each caller
open its own reader through openStoredArtifact. Threading that type
through fetchAndCache, fetchAndCacheFromURL and their error paths is
mechanical; behaviour is unchanged.
* Coalesce concurrent cache misses
A cache miss went from checkCache straight to an upstream fetch with
nothing tracking in-flight work, so N concurrent requests for one
uncached artifact produced N upstream fetches and N stores to the same
key. That is the CI shape: parallel jobs installing overlapping
dependencies against a cold cache. The duplicate stores also fail
requests, racing fileblob's per-key ".attrs" sidecar into a partial read
served as a 502. Over 12 runs of 8 simultaneous requests for one
uncached tarball, against bb2205a: before, 8 fetches per run and 12 of
96 responses were 502; after, 1 fetch per run and none failed.
Route both miss paths through a shared in-flight map keyed on the
artifact, including the download URL and upstream-declared hash so
callers expecting different bytes never share a fetch.
singleflight does not fit: Do gives waiters no way to leave, while
DoChan lets the caller running the fetch abandon it, breaking
storeArtifact's scan-on-disconnect contract. Deciding roles under a
mutex gives both behaviours. The fetch runs on the first caller's
context and is seen through; waiters leave when their own clients do.
This removes the sidecar trigger on this path. The race is in fileblob
and three writers bypass this path entirely, so it is fixed separately.
Fewer failures now reach the circuit breaker, so it trips later.
Sixteen concurrent callers against real file:// storage fail 10 of 10
runs on main and pass 10 of 10 here. Other tests pin key discrimination,
failure propagation, resolver-path coalescing, per-caller readers,
waiter cancellation, key release and panic safety. allocs/op is
unchanged. mockStorage gains a mutex so concurrent tests can use it.
* Normalize digest case in the coalescing key
artifactHashMatches compares digests with strings.EqualFold, but the
coalescing key used the hash verbatim. The same digest in two casings
produced two keys, so two callers for one artifact each ran their own
upstream fetch and store, which is what the coalescing is meant to
prevent.
* Make the panic coalescing test deterministic
The test timed the second caller's arrival with a sleep, so which caller
became the leader was left to the scheduler. When it lost that race the
second caller ran the fetch itself, and its panic was not recovered, so
the test binary died instead of the test failing.
Whether a caller has reached the wait is not observable from outside:
it runs a cache lookup against the database first, so releasing the
leader on a timer races that query. Drive coalesceFetch directly and
hold the shared entry instead, which removes the timing entirely. The
panicking fetcher is no longer needed.
* Recheck the cache before running a shared fetch
A caller checks the cache before it reaches coalesceFetch, so a fetch
that commits in that gap is invisible to it. Arriving after the sharing
entry is gone, it became a new leader and fetched, stored and scanned an
artifact the cache already held.
The leader now rechecks the committed record first. It serves that
record only if its bytes still open, because a record can outlive them,
and refetching is the recovery the cache lookup already makes for that
case. Waiters are unaffected: the record fills the same shared value a
fetch would, and each caller opens its own reader from it.
The recheck is the leader's alone. A waiter has a fetch in flight to
wait on, and rechecking would race it for no gain.
* Lock the mock fetcher's bookkeeping
Coalescing tests call the handler from many goroutines. The key keeps
the fetch itself serialized, but the mock should not lean on that: it
now locks the fields it records, so any concurrency the handler applies
is safe under the race detector.
* Wait for the leader's fetch instead of sleeping
The canceled-waiter test slept 200ms and assumed the leader had taken
the key by then. On a slow scheduler the canceled call could become the
leader and the test would no longer cover waiter cancellation. The
fetcher now signals when its first fetch begins, which happens only once
the key is held.
* Use shared artifacts at cache boundaries
* Address artifact cache review
* Defer cached artifact validation to checkCache
Construct artifacts.Artifact from the row without validation so a
malformed content hash reaches newIntegrityChecks in checkCache, which
clears the row and treats the request as a miss. Erroring at the DB
boundary instead surfaced a 500 and left the bad row in place.
* Build stored Artifact after digest and scan checks
Construct the shared artifact struct from trusted storage output as a
literal, after the hash-mismatch and scanner paths that delete failed
downloads, so no path between Store and updateCacheDB can leave bytes
in storage without a database row.
Runs fetched artifacts through pluggable external scanners after they're
staged in storage but before they're committed to the cache DB, so a
block verdict deletes the object instead of ever exposing it to a
client. Scanners pull the staged bytes themselves via a short-lived
HMAC-signed internal route rather than the proxy pushing bytes to them,
keeping the mechanism storage-backend-agnostic and avoiding uploading
potentially huge artifacts through the proxy's own egress.
Hardening baked in from the start: the internal scan-fetch route is
gated both at router-mount time and in the handler so it's inert
whenever scanning is disabled or unsigned; the signing key is mandatory
whenever scanning is enabled, enforced directly in scanner.NewGroup
rather than relying on callers to invoke config validation; the scan
call and the delete-on-block cleanup both run on a context detached
from the client's, so a client disconnecting mid-scan can't be mistaken
for a scanner failure, doesn't cause a legitimate artifact to be
deleted, and doesn't leave a genuinely blocked artifact's bytes
orphaned in storage; and scanner infrastructure errors (connection
failures, internal hostnames) are never forwarded verbatim to anonymous
clients, only a generic message. The scan-error metric also correctly
distinguishes a scanner's own timeout from being cancelled because a
sibling scanner already decided the verdict.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Add Swift package registry support
* Fix Swift archive integrity handling
* Fix Swift registry pagination and archive HEAD requests
* Canonicalize Swift package identifiers
* Handle Swift registry cache identities
* Use stored PURLs for cache lookups
* Address review: upstream-hash refetch and cache PURL cleanup
- Re-fetch instead of 502 when a cached artifact's hash disagrees with
the upstream-declared checksum; log and discard the stale entry.
- Rename expectedHash to upstreamHash and document how the check
differs from checkCache's stream integrity verification.
- Drop the repository_url qualifier from swift cache PURLs so cache
entries survive an upstream.swift change, matching other ecosystems.
- Pass name to handleSourceArchiveHead instead of re-deriving it.
* Drop BulkCheckVulnerabilities coverage after #279 removed it
The rebase over #279 (dead-code cleanup) drops BulkCheckVulnerabilities;
remove the tests and helper that exercised the swift-identity-filtering
path through it, and the imports they pulled in.
Every built-in ecosystem upstream can now be set via the upstream
config block or PROXY_UPSTREAM_* env vars, with allow_private_hosts
and allow_loopback controlling access to non-public addresses.
NormalizedPackages runs config keys through purl.Parse().String(), which
applies per-type rules like lowercasing pypi names. The handler side was
building lookup keys with MakePURLString, which does not, so a config
entry for pkg:pypi/Django would be normalized to pkg:pypi/django and
never match the runtime key pkg:pypi/Django.
Route both sides through the same canonical form: a new
canonicalPackagePURL helper calls Normalize() on the constructed PURL
before stringifying, and all cooldown IsAllowed call sites use it.
When the proxy reaches storage at an internal address (127.0.0.1, a
Docker service name) the presigned URLs it generates point there too,
which is useless to external clients. This adds an optional base URL
that replaces the scheme and host of signed URLs before they're returned,
keeping the signed path and query intact.
When storage.direct_serve is enabled and the backend supports it (S3,
Azure), cached artifact downloads return a 302 redirect to a presigned
URL instead of streaming bytes through the proxy. Falls back to
streaming when the backend can't sign (fileblob, local filesystem) or
signing fails.
Adds the azureblob driver so azblob:// storage URLs work.
Cache-hit accounting already happened before io.Copy so redirects are
counted correctly; the metrics calls are pulled into a helper so both
paths share them.
Closes#96
Cached metadata is now served directly within a configurable TTL window
(default 5m) without contacting upstream, reducing latency and upstream
load. When upstream is unreachable and the cache is past its TTL, stale
content is served with a Warning: 110 header per RFC 7234.
New config: `metadata_ttl` (YAML) / `PROXY_METADATA_TTL` (env).
Set to "0" to always revalidate with upstream.
- ProxyCached now stores upstream Last-Modified in the cache and uses it
(along with ETag) for conditional request handling, returning 304 when
client validators match. Adds Content-Length to cached responses.
- Handlers calling FetchOrCacheMetadata (pypi, composer, pub, nuget) now
check for ErrUpstreamNotFound and return 404 instead of 502, matching
the existing npm and cargo behavior.
- Mirror jobs report live progress via a periodic callback while running,
so API polls return real counts instead of zeroed progress.
- Registry mirroring removed from CLI flags, API acceptance, README, and
docs since every enumerator was a stub returning "not yet implemented".
- Added tests for the conditional metadata path (ETag/If-None-Match,
Last-Modified/If-Modified-Since, 304 responses, header omission).
* Fix startup message and add connectivity check for S3 storage
When S3 storage is configured, the startup log incorrectly showed the
default local path (./cache/artifacts) instead of the actual S3 URL.
This also adds a lightweight connectivity check at startup so bad
credentials or endpoints fail immediately rather than on first request.
Add URL() and Close() to the Storage interface so all backends report
their URL and can be cleaned up properly. Rename the stats JSON field
from storage_path to storage_url. Close storage in error paths and
during graceful shutdown.
Fixes#49
* Fix Windows test assertion for file:// URL format
OpenBucket normalizes Windows paths to file:///C:/path (three slashes)
but the test expected file://C:/path (two slashes).
* Fix container blob caching by passing auth token to fetcher
The container handler was calling GetOrFetchArtifactFromURL without
authentication headers, causing Docker Hub to return 401. The fallback
proxyBlobWithAuth path had auth but bypassed the cache entirely.
Now passes the Bearer token through GetOrFetchArtifactFromURLWithHeaders
so blobs are both authenticated and cached.
Fixesgit-pkgs/proxy#43
* Update registries to v0.4.0
Replace pre-release pseudo-version with the released v0.4.0 now that
git-pkgs/registries#13 has been merged.
* Fix all golangci-lint issues across the codebase
Resolve 77 lint issues reported by golangci-lint with gocritic, gocognit,
gocyclo, maintidx, dupl, mnd, unparam, ireturn, goconst, and errcheck
enabled. Net reduction of ~175 lines through shared helpers and
deduplication.
* Suppress staticcheck SA1019 for intentional deprecated field usage
The Storage.Path field is deprecated but still read for backwards
compatibility with existing configs that haven't migrated to the URL field.
