Concurrent cache misses for the same region now share a single
GetAuthorizationToken call instead of each issuing their own, avoiding
a request burst against the ECR API at cold start and at each 12-hour
refresh. golang.org/x/sync is already a direct dependency.
- Add "ecr" auth type to upstream.auth config with optional region
- Cache ecr:GetAuthorizationToken results per region and refresh
shortly before expiry via the AWS SDK default credential chain
- Route type: ecr through the token cache in Server.authForURL
- Document the new type in config.example.yaml and docs/configuration.md
Fixes#276
* Bump go tool golangci-lint to v2.13.1
The .golangci.yml goconst.ignore-tests setting was added in v2.12.0
(golangci/golangci-lint#6480). On the previously pinned v2.10.1,
config verify fails with "additional properties 'ignore-tests' not
allowed" and the setting is silently ignored at run time, so goconst
counts test-file literals toward min-occurrences.
* Apply gofmt and CutSuffix simplification
- gofmt -w internal/server/health_test.go
- Replace HasSuffix+TrimSuffix with CutSuffix in ParseSize
* Remove dead code and migrate tests off legacy Filesystem storage
Migrate the three test call sites of storage.NewFilesystem to
storage.OpenBucket("file://...") and drop the deprecated
StorageConfig.Path field from test configs, then delete code that
deadcode reports as unreachable from cmd/proxy:
- internal/storage/filesystem.go and its tests
- storage.HashingReader
- enrichment.Service.BulkCheckVulnerabilities and NormalizeLicense
- server.ActiveRequestsMiddleware (no-op body; the real tracking
is the inline r.Use at server.go:226)
- mirror.RegistrySource (unimplemented stub)
metrics.UpdateCircuitBreakerState and RecordCircuitBreakerTrip are
kept because #275 wires them.
Update the CONTRIBUTING.md storage section to reflect blob.go.
* Show build information in web UI
Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
* Fix footer build info shadowed by page Version fields
Shared footer templates were reading .Version and .Commit, which resolve
to package data on VersionShowData and BrowseSourceData. Point the footer
at Layout.BuildInfo and cover both pages so the proxy version stays visible.
Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix Layout build info field promotion
---------
Signed-off-by: WilliamK112 <164879897+WilliamK112@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Refactor LoadFromEnv to use helpers
Avoid triggering the linter about the cyclomatic complexity of the LoadFromEnv
function in later changes by refactoring it to use setEnvString/setEnvBool
helpers.
No functional change, just collapse ~29 repetitive if-blocks into single-line
calls to two small helpers.
* Make Debian upstream repository configurable
Support overriding the Debian handler's upstream (e.g. Ubuntu archives)
via PROXY_UPSTREAM_DEBIAN or upstream.debian in the config file.
Tested with PROXY_UPSTREAM_DEBIAN=http://archive.ubuntu.com/ubuntu
to get Ubuntu Resolute packages.
* resolve name and version for PEP 658 metadata sidecars
* fix(pypi): parse Windows installer and egg filenames separately
The bdist_wininst and bdist_msi layout joins the platform to the version
with a '.' rather than a '-', so treating .exe/.msi like a wheel folded
the platform into the version: foo-1.0.win32-py2.0.exe resolved to
version "1.0.win32". Eggs shared the problem, as setuptools' hyphen
escaping is not universal: aws-sdk-1.0.0-py3.11.egg resolved to name
"aws", version "sdk".
Give each format its own parser. Wheels keep the PEP 427
spec-guaranteed field positions, eggs locate the version relative to the
py{X.Y} interpreter field, and Windows installers strip the platform and
interpreter fields before splitting name from version.
A PEP 658 sidecar resolves to the same name and version as the
distribution it describes, so it is cached under that version. Browse and
compare took the first cached artifact without checking its extension,
handing openArchive plain text: a version pip had only fetched metadata
for reported hasCached and then 500'd.
Add firstBrowsableArtifact, replacing five duplicated selection loops,
and export PyPIMetadataSuffix so the suffix has a single definition.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add http_timeout config option (and PROXY_HTTP_TIMEOUT env var) to set
the per-request timeout on the shared HTTP client used by protocol
handlers for upstream metadata fetches and pass-through file requests.
Defaults to the previous hardcoded 30s; "0" disables the timeout.
Fixes#187
The startup log and /stats endpoint always reported Database.Path,
which is the sqlite default even when PROXY_DATABASE_DRIVER=postgres
is set and a postgres URL is in use. Add DatabaseConfig.String() that
returns the sqlite path or the postgres URL with the password redacted,
and use it in both places.
Fixes#173
- Move third-party JS into static/vendor/ and add lucide for icons; refine
.gitignore so embedded vendor dirs aren't caught by the Go vendor rule.
- Replace folder/file emojis in the source browser with lucide icons.
- Wrap the header logo and title in a single anchor so the icon is clickable.
- Drop the redundant "Powered by git-pkgs" footer block; add a GitHub repo
link to the About column and bump the ecosystem count from 16+ to 17+.
- Sticky footer pattern: body is min-h-full flex column with main growing to
fill, so the footer sits at the bottom of short pages.
- Hamburger menu under md: search and nav links collapse into a drawer
toggled by a menu button; theme toggle stays visible at both sizes.
Adds UIBaseURL (env PROXY_UI_URL), advertised separately from BaseURL
for deployments where the UI is reached on a public domain while build
machines hit a Docker network alias for the package endpoints. Defaults
to BaseURL when unset.
Consumers:
- Logged alongside base_url at startup.
- <link rel="canonical"> and og:url / og:title / og:site_name in every
UI page, omitted when UIBaseURL is empty.
- Banner on the install guide when UIBaseURL differs from BaseURL,
clarifying that the URLs in the snippets are the package endpoint and
that the UI itself lives elsewhere.
Docs call out that the proxy serves UI and package endpoints on the same
listener, so reverse proxies fronting the UI publicly must restrict the
public route to PathPrefix(/ui) to avoid exposing /npm, /pypi, etc.
nginx and Traefik examples both show the path-split pattern.
* Mount web UI under /ui (closes#123)
The UI now lives under /ui so reverse proxies can apply different
access rules to it (e.g. require auth) while leaving the package
endpoints (/npm, /pypi, /v2, ...) open to build machines.
- GET / redirects to /ui/
- /api/browse and /api/compare move to /ui/api/browse and
/ui/api/compare since only the browser JS calls them
- /health, /stats, /metrics, /openapi.json and /api/* stay at root
* README: nginx example for splitting UI auth from package endpoints
The /ui prefix lets a reverse proxy apply different access rules to the
web UI than to the package endpoints. Adds the snippet that motivated
the prefix change so deployers don't have to derive it.
* Bump github.com/go-chi/chi/v5 from 5.2.5 to 5.3.0
Bumps [github.com/go-chi/chi/v5](https://github.com/go-chi/chi) from 5.2.5 to 5.3.0.
- [Release notes](https://github.com/go-chi/chi/releases)
- [Changelog](https://github.com/go-chi/chi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/go-chi/chi/compare/v5.2.5...v5.3.0)
---
updated-dependencies:
- dependency-name: github.com/go-chi/chi/v5
dependency-version: 5.3.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Drop deprecated middleware.RealIP
chi 5.3.0 deprecated middleware.RealIP because it trusts X-Forwarded-For,
True-Client-IP, and X-Real-IP unconditionally, which is spoofable when
the service is not strictly behind a trusted proxy (GHSA-3fxj-6jh8-hvhx,
GHSA-rjr7-jggh-pgcp, GHSA-9g5q-2w5x-hmxf). The only consumer of
r.RemoteAddr in this codebase is the request log; no auth, rate limiting,
or other security decisions depend on it, so removing the middleware is
safe. If we ever need real client IPs in logs behind an LB, add a
trusted-proxy-aware middleware then.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
* config: add Health.StorageProbeInterval
* metrics: add proxy_health_probe_failures_total counter
* server: add storageProbe with happy-path test
* server: add storageProbe failure-mode tests
* server: add healthCache with TTL, single-flight, transition logging
* server: wire storage probe into /health
* server: update TestHealthEndpoint for JSON; wire healthCache into newTestServer
Also fix Windows file-locking issue in storageProbe: close the reader
explicitly before Delete so the file handle is released prior to os.Remove.
* server: clean up stale comment in storageProbe
* docs: document storage health probe and new metric
* docs: regenerate Swagger for /health JSON response
* server: simplify rc.Close error handling in storageProbe
* server: defer probe cleanup so size/open/read/verify failures don't leak objects
Previously, storageProbe only called Delete on the success path. Any
failure between Store and the final Delete (size mismatch, Open error,
mid-stream read failure, content mismatch) left the probe object orphaned
in the storage backend. With caching disabled and Kubernetes-rate probing,
the leak could accumulate noticeably on backends like S3.
Use a named return + defer to attempt Delete after every successful Store.
The earlier-step failure remains the primary error; Delete failure only
surfaces as step="delete" when nothing else went wrong. Add a table-driven
test that asserts cleanup runs for each non-delete failure path.
Reported by Copilot on #119.
* config: validate health.storage_probe_interval in Config.Validate
The new duration field was only validated at use time in newHealthCache.
The existing codebase already validates other duration fields
(MetadataTTL, DirectServeTTL, Gradle.MaxAge, Gradle.SweepInterval) in
Config.Validate() so misconfiguration fails fast at startup with a
config-key-specific error.
Match that pattern. The parse-at-use code in newHealthCache stays as
a safety net, mirroring the MetadataTTL precedent.
Reported by Copilot on #119.
* docs: lowercase "counter" in metrics table for consistency
Other rows in the table use lowercase type names (counter/gauge/histogram).
Match that style.
Reported by Copilot on #119.
* docs: include size-check step in /health probe description
The probe is write → size-check → read → verify → delete; the
architecture note was missing the size-check step.
Reported by Copilot on #119.
* server: address andrew's review on #119
- Drop unused callerCtx parameter from healthCache.Check (Check is now
parameter-less; the comment-only "accepted for symmetry" justification
wasn't carrying its weight).
- Emit "storage": {"status": "skipped"} on DB short-circuit instead of
omitting the key, so monitors expecting a fixed key set keep working.
- Reject negative storage_probe_interval at config validation time
(previously parsed and silently behaved like "0").
- Extract HealthConfig.Validate to keep Config.Validate under the
gocognit threshold and match the existing GradleBuildCacheConfig pattern.
- README Health Check section: note that /health is intended as a
readiness probe rather than a liveness probe (Check holds a mutex
for up to the 10s probe timeout).
- cmd/proxy/main.go godoc: column-align the new env var with the
surrounding Gradle entries.
Reported by andrew on #119.
- Implement /julia/* handler for the Pkg server protocol
(registries, registry, package, artifact, meta)
- Resolve package UUIDs to names by parsing Registry.toml from
the General registry tarball, with a hash-guarded background
refresh on registry updates
- Wire into router, ecosystem list, install page, badge styles
- Update README and architecture docs
- Bump github.com/git-pkgs/registries to v0.6.0: the fetcher now
honours HTTP_PROXY, gates dialled IPs against the safehttp block
list, and Version.Integrity is populated for pub, julia and nuget
- Replace internal/cooldown with github.com/git-pkgs/cooldown v0.1.1
(identical surface, lifted from this repo)
- Update docs/architecture.md to point at the external package
Bake the extended linter set into a project config so plain
golangci-lint run matches what we check locally, with goconst tuned
to ignore tests and bare lowercase words to drop ~200 ecosystem-name
and test-literal false positives.
Clear the remaining real findings: extract GradleBuildCacheConfig.Validate
from Config.Validate, pull the eviction sort comparator into
sortOldestFirst (zero time.Time already sorts first via Before so the
switch was redundant), add headerAcceptEncoding and SQL column-type
constants, and drop a dead empty-key recheck in the gradle handler.
* add Gradle Build Cache support with handler and tests
* linting issue
* MR Suggestions: Add Gradle HTTP Build Cache configuration to README
* implement minor stuff: Refactor Gradle handler to remove unnecessary URL parameter and update related tests
Co-authored-by: Copilot <copilot@github.com>
* Add Gradle build cache configuration and eviction support
- Introduced configuration options for Gradle build cache in config files and documentation.
- Implemented read-only mode and upload size limits for the Gradle build cache.
- Added cache eviction logic based on age and size, with corresponding tests.
- Enhanced storage interfaces to support listing objects by prefix.
* implement minor stuff: Refactor Gradle handler to remove unnecessary URL parameter and update related tests
* last finding fix
* fix tests and implement PR suggestions
Co-authored-by: Copilot <copilot@github.com>
* unify path
---------
Co-authored-by: Mateusz (Mati) Kepa <m.kepa@sportradar.com>
Co-authored-by: Copilot <copilot@github.com>
* Structured JSON error responses for API endpoints
API handlers returned errors via http.Error (text/plain) with ad-hoc
strings, while the mirror API used a different {"error": "..."} shape
and leaked internal err.Error() text to clients.
Add ErrorResponse{Code, Message} with stable codes (BAD_REQUEST,
NOT_FOUND, UPSTREAM_ERROR, INTERNAL_ERROR) and writeError/badRequest/
notFound/internalError helpers. Convert all JSON API handlers in
api.go, browse.go, mirror_api.go and the /stats endpoint. Enrichment
failures now report 502 UPSTREAM_ERROR rather than 500.
Protocol handlers in internal/handler/ are deliberately unchanged
since npm/pip/cargo clients expect their own response formats, not
JSON. HTML page handlers in server.go also keep text/plain.
Swagger @Failure annotations updated and docs regenerated.
Fixes#76
* Convert validatePackagePath errors to JSON in API handlers
The wildcard package routes (/packages/{ecosystem}/*, /api/package/*,
/api/vulns/*, /api/browse/*, /api/compare/*) only checked for an empty
path before passing user input to GetPackageByEcosystemName and the
enrichment service.
Add validatePackagePath as a coarse first-line filter: reject null
bytes, other control characters, and paths over 512 bytes. Wired into
all five entry handlers immediately after the chi wildcard is read.
This is the generic layer; ecosystem-specific name format rules (npm
scoped name shape, Maven coordinate structure, etc.) can be added on
top per #75.
Fixes#75
The browse and compare handlers buffer the full artifact into memory for
prefix detection. Without a cap, a single request for a large cached
artifact could exhaust server memory.
These file types were served with executable content types (text/html,
image/svg+xml) allowing stored XSS via package archive contents.
Also adds Content-Security-Policy: sandbox and X-Content-Type-Options:
nosniff headers to all browse file responses.
Closes#99. The max_size storage config was parsed and validated but
never enforced. This adds a background eviction loop that periodically
checks total cache size and evicts least recently used artifacts when
the limit is exceeded.
When the proxy reaches storage at an internal address (127.0.0.1, a
Docker service name) the presigned URLs it generates point there too,
which is useless to external clients. This adds an optional base URL
that replaces the scheme and host of signed URLs before they're returned,
keeping the signed path and query intact.
When storage.direct_serve is enabled and the backend supports it (S3,
Azure), cached artifact downloads return a 302 redirect to a presigned
URL instead of streaming bytes through the proxy. Falls back to
streaming when the backend can't sign (fileblob, local filesystem) or
signing fails.
Adds the azureblob driver so azblob:// storage URLs work.
Cache-hit accounting already happened before io.Copy so redirects are
counted correctly; the metrics calls are pulled into a helper so both
paths share them.
Closes#96
Cached metadata is now served directly within a configurable TTL window
(default 5m) without contacting upstream, reducing latency and upstream
load. When upstream is unreachable and the cache is past its TTL, stale
content is served with a Warning: 110 header per RFC 7234.
New config: `metadata_ttl` (YAML) / `PROXY_METADATA_TTL` (env).
Set to "0" to always revalidate with upstream.
- Fix race where runJob could overwrite canceled state set by Cancel()
- Fix Debian ecosystem name inconsistency ("deb" -> "debian")
- Stream metadata responses when caching is disabled to avoid buffering
- Add metadata_cache table to initial schema strings for consistency
- Gate mirror API behind mirror_api config flag (disabled by default)
- Fix goconst lint in metadata_cache_test.go
- Wire job contexts to server shutdown context so jobs are canceled on
server stop instead of running indefinitely
- Defer context cancel in runJob so completed jobs don't leak contexts
- Cap error accumulation in progressTracker to 1000 entries to prevent
OOM on large mirror operations with many failures
- Add panic recovery in errgroup workers to prevent process crashes
- Use defer for db.Close() in runMirror CLI to ensure cleanup on all
error paths
Add a `proxy mirror` CLI command and `/api/mirror` API endpoints that
pre-populate the cache from various input sources: individual PURLs,
SBOM files (CycloneDX and SPDX), or full registry enumeration.
The mirror reuses the existing handler.Proxy.GetOrFetchArtifact()
pipeline so cached artifacts are identical to those fetched on demand.
A bounded worker pool controls download parallelism.
Metadata caching is opt-in via `cache_metadata: true` in config (or
PROXY_CACHE_METADATA=true). The mirror command always enables it. When
enabled, upstream metadata responses are stored for offline fallback
with ETag-based conditional revalidation.
New internal/mirror package with Source interface, PURLSource,
SBOMSource, RegistrySource, and async JobStore. New metadata_cache
database table for offline metadata serving.
GitHub zipballs wrap all files in a repo-hash/ directory. Instead of
hardcoding prefixes per ecosystem, open the archive once to check if all
files share a single root directory and strip it automatically. The npm
package/ prefix is still handled as a special case.
GitHub zipball URLs end in a bare commit hash with no file extension.
rewriteDistURL now appends .zip when the filename has no extension and
the dist type is zip. expandMinifiedVersions deep copies inherited
values so in-place URL rewriting no longer corrupts shared references.
browse.go infers .zip for extensionless filenames so existing cached
artifacts can still be opened.
`fileblob` creates temp files in `os.TempDir()` (`/tmp`) by default,
then uses `os.Rename` to move them to the final path. When the storage
directory is on a different filesystem (e.g. a Docker volume mount at
`/data`), the rename fails with "invalid cross-device link".
Set `no_tmp_dir=true` on file:// bucket URLs so fileblob creates temp
files next to the final destination instead.
Fixes#65
