2026-01-29 16:33:09 +00:00
# Configuration
The proxy can be configured via command line flags, environment variables, or a configuration file. Command line flags take precedence over environment variables, which take precedence over the configuration file.
## Configuration File
Create a YAML or JSON file and pass it with `-config` :
```bash
proxy serve -config config.yaml
```
See `config.example.yaml` in the repository root for a complete example.
## Server Settings
| Config | Environment | Flag | Default | Description |
|--------|-------------|------|---------|-------------|
| `listen` | `PROXY_LISTEN` | `-listen` | `:8080` | Address to listen on |
2026-06-07 16:12:43 +01:00
| `base_url` | `PROXY_BASE_URL` | `-base-url` | `http://localhost:8080` | Public URL package managers use to reach this proxy |
| `ui_base_url` | `PROXY_UI_URL` | - | (defaults to `base_url` ) | Public URL where the web UI is reached. Set separately when the UI lives behind a different hostname than package endpoints (e.g. public domain vs Docker network alias). Used for canonical/og:url tags and the install guide banner. The proxy still serves package endpoints on the same listener, so any reverse proxy fronting the UI publicly should restrict the public route to `PathPrefix(/ui)` to avoid exposing package endpoints. |
2026-10-01 14:32:21 +02:00
| `trust_forwarded_for` | `PROXY_TRUST_FORWARDED_FOR` | - | `false` | Attribute requests to the leftmost `X-Forwarded-For` entry instead of the TCP peer address, in the structured log, the access log and the request-source table on `/ui/analytics` . |
| `ui_request_sources` | `PROXY_UI_REQUEST_SOURCES` | - | `false` | Show the request-source table on `/ui/analytics` , which reports caller addresses, the tool each ran and how much each pulled. |
2026-10-12 09:41:42 +02:00
| `ui_failure_details` | `PROXY_UI_FAILURE_DETAILS` | - | `false` | List the latest failures behind each failure count on `/ui/analytics` : the package, its upstream URL or storage path, and the error. URLs are shown without credentials or query strings. |
2026-10-19 15:54:21 +08:00
| `http_timeout` | `PROXY_HTTP_TIMEOUT` | - | `30s` | Timeout for upstream metadata and pass-through requests. |
| `write_timeout` | `PROXY_WRITE_TIMEOUT` | - | `5m` | Response write deadline, including download and scanning time before the response starts. |
| `artifact_timeout` | `PROXY_ARTIFACT_TIMEOUT` | - | `5m` | Timeout for each upstream artifact download while serving requests; excludes subsequent scanning. Does not limit `proxy mirror` . |
These timeouts accept Go durations such as `30s` , `5m` , or `1h` . An omitted value uses the default; `"0"` disables that timeout. Invalid or negative values are rejected at startup. Environment variables override file values.
For long scans, increase `write_timeout` to cover the download, scan, and response transfer. `scanning.timeout` controls the scan separately; changing it alone does not extend the response deadline. `artifact_timeout` is independent of both. For example:
```yaml
write_timeout: "18m"
artifact_timeout: "10m"
scanning:
timeout: "16m"
```
These are independent limits, not reserved time slices: a download approaching 10 minutes followed by a 16-minute scan would still exceed an 18-minute response deadline. Any reverse proxy in front of the server also needs sufficient timeout allowance. The upstream transport's existing response-header timeout remains unchanged.
2026-01-29 16:33:09 +00:00
2026-10-25 17:25:56 +01:00
### Memory limit
On Linux, when the process runs under a cgroup memory limit, such as a container's `--memory` or a Kubernetes memory limit, `proxy serve` sets Go's soft memory limit to 80% of it and log the value at startup (`set Go memory limit from cgroup` ). The garbage collector then runs more often as memory use approaches the limit, rather than letting the heap grow until the container is killed. The remaining 20% is left for memory Go doesn't manage, such as `git` processes the proxy runs. Set the `GOMEMLIMIT` environment variable to choose the value yourself, or `GOMEMLIMIT=off` to disable it.
2026-01-29 16:33:09 +00:00
## Storage
The proxy stores cached artifacts using gocloud.dev/blob, supporting local filesystem and S3-compatible storage.
### Local Filesystem
```yaml
storage:
url: "file:///var/cache/proxy"
```
Or using the legacy path option:
```yaml
storage:
path: "./cache/artifacts"
```
| Config | Environment | Flag | Description |
|--------|-------------|------|-------------|
| `storage.url` | `PROXY_STORAGE_URL` | `-storage-url` | Storage URL (file:// or s3://) |
| `storage.path` | `PROXY_STORAGE_PATH` | `-storage-path` | Local path (deprecated, use url) |
| `storage.max_size` | `PROXY_STORAGE_MAX_SIZE` | - | Max cache size (e.g., "10GB") |
Add storage.cache_artifacts to serve artifacts without storing them (#387)
* Add storage.passthrough to serve artifacts without storing them
With storage.passthrough enabled the proxy streams every artifact from
upstream to the client and never writes it to storage or the cache
database. Metadata filtering, cooldown and the denylist work as before,
so the proxy can sit behind another cache (e.g. an Artifactory remote)
purely as a policy layer without holding a second copy of every package.
Artifacts whose digest is known up front (OCI, Swift, Helm) are verified
while streaming. Their responses are sent chunked and the connection is
aborted on a mismatch, so a client never receives a tampered artifact as
a complete response.
Passthrough is rejected together with scanning, direct_serve and
mirror_api, and the mirror command refuses to run with it, since all of
them depend on stored artifacts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address review: abort truncated streams, guard every cache read
- serveArtifact aborts the response on any body read error, and when fewer
bytes than the declared size were written, not only on a digest mismatch.
A failed or short upstream body in passthrough mode no longer reaches the
client as a complete 200.
- Streamed upstream read failures are logged and counted as
stream_failed upstream errors before the response is aborted.
- checkCache reports a miss when artifact caching is off, so every cache
read path, including Swift archive HEAD requests, ignores entries stored
before the mode was enabled. Denylisted versions are still rejected.
- Rename storage.passthrough to storage.cache_artifacts (default true,
PROXY_STORAGE_CACHE_ARTIFACTS), which names what actually changes.
Tests cover an OCI blob shorter than its Content-Length and an npm tarball
whose chunked response ends without the final chunk, both served through
the HTTP handlers, plus a Swift HEAD request against a cached archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Rename Proxy.Passthrough to StreamArtifacts
Match the storage.cache_artifacts option: the field says what changes
(artifacts are streamed instead of stored) and its zero value keeps the
usual caching behaviour for every Proxy built without the server config,
such as the mirror command and tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 15:28:20 +07:00
| `storage.cache_artifacts` | `PROXY_STORAGE_CACHE_ARTIFACTS` | - | Store fetched artifacts (default: true); `false` streams them from upstream |
Add age-based retention for cached artifacts (#428)
* Add age-based retention for cached artifacts
Cached artifacts stayed forever unless storage.max_size pushed them out.
storage.retention adds a default, per-ecosystem and per-package duration
after which an artifact nobody downloaded gets evicted.
Ecosystems opt in one at a time: a handler registers with
retention.Default, and until it does, configuring that ecosystem (or one
of its packages) fails validation, and the default doesn't touch it. This
commit registers none, so nothing changes yet. Startup logs which
ecosystems the default covers and which it doesn't.
An artifact counts as expired when both its fetch time and its last
access are older than the cutoff. Checking only the last access would
evict a refetched artifact again right away, since clearing a record
keeps its old access time and a cache miss doesn't record a hit.
The sweep scans artifacts in id windows, so no single query holds the
SQLite connection for long, and checks each row against its own rule.
Expired records are cleared and their files go through pending_deletes
instead of being deleted directly, because a buffered hit can make an
artifact that's being served right now look expired. Buffered hits are
flushed before each sweep, and one sweep clears at most 10,000
artifacts.
Since a big first sweep can queue a lot, reclaim now keeps working
through due entries for up to 30 seconds per tick instead of stopping
after 100, and pending_deletes gets an index on queued_at.
Also:
- proxy_artifacts_evicted_total{reason, ecosystem} counts LRU and
retention evictions.
- gradle.build_cache.max_age accepts "7d", as its comment always said.
Refs #306
* Document storage retention
Describes the new storage.retention block, the order rules apply in,
that the default only covers ecosystems that support retention yet, and
what to expect from the first sweep over an old cache. Also lists the
new eviction counter and notes that gradle max_age takes days.
Refs #306
* Fix retention timing on Postgres and tighten the sweep
On Postgres, fetched_at and last_accessed_at are TIMESTAMP columns
without a zone. They hold the local time the proxy wrote, but lib/pq
reads them back labelled UTC. East of UTC that makes every artifact look
hours younger than it is, so the sweep evicted late: two hours in
Berlin, nine in Tokyo. The candidates' times are now read back as local
time, and the database tests run on both SQLite and Postgres, with one
that sets the local zone to Asia/Tokyo.
A few smaller things:
- When a delete fails, reclaim stops for the rest of the minute instead
of trying every queued path. Otherwise a storage outage turns into
thirty seconds of failed deletes and warnings every minute.
- If the buffered hits can't be written, the sweep skips that round.
Running on stale access times could clear something that is being
downloaded right now.
- DefaultCanonical only accepts a package key when rebuilding it gives
the same PURL. For alpine and deb it would otherwise produce keys
that never match.
- sweep_interval has to be at least a minute.
- New tests cover the eviction counter and an artifact that gets a hit
between the scan and the clear.
- The docs no longer say nothing changes without retention, and they
point out that the example's ecosystem and package rules only pass
validation once those ecosystems support retention.
Refs #306
* Reject qualifiers and subpaths in retention package keys
A rule names a whole package. Qualifiers like ?type=pom were dropped on
the way to the stored PURL and the rule quietly covered every file of the
package. Now such a key fails validation.
Refs #306
* Say when a package key doesn't match how packages are stored
A package rule for an ecosystem that supports retention, written in a
form its packages aren't stored under (pkg:npm/babel/core without the
@), failed with "retention is not supported for npm packages". Once the
ecosystem is enabled that is no longer true. Such a key now fails with
"does not match how npm packages are stored" and points to the package
rule forms in the docs. Keys for ecosystems without retention support
keep the old message.
Refs #306
* Note that re-running the mirror keeps packages cached
A mirror run over a package that is still cached goes through the normal
cache lookup and records a download, so the package's age starts over.
The docs only said that mirrored packages age from the time they were
mirrored.
Refs #306
* Say when an ecosystem takes no package rules
Some ecosystems will support retention without package rules, because nothing in their stored records names a package the way a PURL would (conan, helm). A package key for one of them used to fail with "does not match how conan packages are stored; see the package rule forms", which sends people looking for a form that doesn't exist. It now says package rules aren't supported there and points to the ecosystem rule.
Refs #306
* Use helm as the example of an ecosystem without package rules
Conan gets package rules once its handler stores archives under the recipe name, so the config test that shows the refusal now uses helm, whose cache records name the repository, not the chart.
Refs #306
* Say that each cached file ages on its own
The retention docs said each artifact, each version of a package, ages on its own. A version can have several cached files (a jar and its pom, several wheels, Conan sources and binaries), and each of them ages separately.
Refs #306
* Clear an expired artifact and queue its file in one transaction
The sweep cleared the record and queued the file for deletion as two separate writes. If queueing failed, or the process stopped in between, the file stayed in storage with no record pointing at it and no queue entry, and no later sweep would find it again. Both writes now share a transaction, so a failed queue rolls the clear back and the next sweep retries.
Refs #306
* Show artifact evictions on the analytics page
proxy_artifacts_evicted_total had no place on /ui/analytics. The Runtime card now lists evicted artifacts by reason and ecosystem, without the failure tint, and the coverage test records an eviction so it doesn't depend on test order.
Refs #306
* Store artifact times in UTC
fetched_at and last_accessed_at are now written in UTC, and the retention cutoffs are compared in UTC, the same convention #434 uses for the metadata cache. lib/pq sends a time with its offset and a Postgres TIMESTAMP keeps only the wall clock, so UTC on write is what reads back correctly. That replaces reinterpreting the times as local on read.
Rows written before this still hold local time. East of UTC they look younger by the offset until they are rewritten, so retention evicts them a few hours late once; west of UTC that much early.
Refs #306
---------
Co-authored-by: t <t@t>
2026-10-12 10:06:01 +02:00
| `storage.retention.default` | `PROXY_STORAGE_RETENTION_DEFAULT` | - | Evict artifacts not served for this long (e.g. "30d"); see [Retention ](#retention ) |
| `storage.retention.sweep_interval` | `PROXY_STORAGE_RETENTION_SWEEP_INTERVAL` | - | How often expired artifacts are looked for (default: "10m", at least "1m") |
2026-01-29 16:33:09 +00:00
2026-10-20 11:22:55 +02:00
`storage.max_size` counts cached artifacts and cached metadata documents (`cache_metadata` ). When the total goes over the limit, the least recently served entries of either kind are evicted first; an evicted metadata document is fetched from upstream again on the next request for it. Pushed artifacts are not counted or evicted. An artifact replaced by a refetch stays in storage for at least an hour, or `storage.direct_serve_ttl` if longer, so requests already reading it can finish, and storage use can exceed the limit by what was replaced in that time.
2026-09-29 10:24:22 -07:00
2026-10-25 20:29:03 +01:00
### Cache misses
On a cache miss the proxy sends the artifact to the client while it stores it, so the client gets its first bytes as soon as the upstream sends them, rather than after the whole artifact is stored. Clients that give up after a while without data, such as git-lfs (30 seconds) or pacman (10 seconds), no longer time out on large artifacts. The response goes out chunked, without `Content-Length` , as with `cache_artifacts: false` ; once stored, the artifact is served with its size.
2026-10-26 16:44:11 +01:00
The bytes are checked against the digest the upstream declares, where it declares one, as they are sent. On a mismatch the connection is aborted before the response completes, so the client never receives a tampered artifact as a good one, and nothing is cached. The upstream is read at its own speed whatever the client's: a client more than 8 MB behind continues from the stored copy once it is stored, so a slow client doesn't hold back the download, other requests waiting for it, or `artifact_timeout` . If the client goes away, the artifact is still stored. If storing it fails, for example because the disk is full, the client still gets the whole artifact; the failure is logged at error level and counted in `proxy_artifacts_served_uncached_total` , so a cache that has stopped filling doesn't go unnoticed while every request succeeds.
2026-10-25 20:29:03 +01:00
Requests for the same artifact that arrive while it is being stored wait for the stored copy, as do `HEAD` requests and requests with a `Range` header, which are answered from it with its size. With a `block` -mode scanner configured for the ecosystem, a miss is stored and scanned before anything is sent, as before (see [Artifact Scanning ](#artifact-scanning )).
Add storage.cache_artifacts to serve artifacts without storing them (#387)
* Add storage.passthrough to serve artifacts without storing them
With storage.passthrough enabled the proxy streams every artifact from
upstream to the client and never writes it to storage or the cache
database. Metadata filtering, cooldown and the denylist work as before,
so the proxy can sit behind another cache (e.g. an Artifactory remote)
purely as a policy layer without holding a second copy of every package.
Artifacts whose digest is known up front (OCI, Swift, Helm) are verified
while streaming. Their responses are sent chunked and the connection is
aborted on a mismatch, so a client never receives a tampered artifact as
a complete response.
Passthrough is rejected together with scanning, direct_serve and
mirror_api, and the mirror command refuses to run with it, since all of
them depend on stored artifacts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address review: abort truncated streams, guard every cache read
- serveArtifact aborts the response on any body read error, and when fewer
bytes than the declared size were written, not only on a digest mismatch.
A failed or short upstream body in passthrough mode no longer reaches the
client as a complete 200.
- Streamed upstream read failures are logged and counted as
stream_failed upstream errors before the response is aborted.
- checkCache reports a miss when artifact caching is off, so every cache
read path, including Swift archive HEAD requests, ignores entries stored
before the mode was enabled. Denylisted versions are still rejected.
- Rename storage.passthrough to storage.cache_artifacts (default true,
PROXY_STORAGE_CACHE_ARTIFACTS), which names what actually changes.
Tests cover an OCI blob shorter than its Content-Length and an npm tarball
whose chunked response ends without the final chunk, both served through
the HTTP handlers, plus a Swift HEAD request against a cached archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Rename Proxy.Passthrough to StreamArtifacts
Match the storage.cache_artifacts option: the field says what changes
(artifacts are streamed instead of stored) and its zero value keeps the
usual caching behaviour for every Proxy built without the server config,
such as the mirror command and tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 15:28:20 +07:00
### Serving artifacts without storing them
With `storage.cache_artifacts: false` the proxy streams every artifact download from upstream to the client and stores no artifacts. Metadata is still filtered and cached, so cooldown and the denylist apply as usual, and the database still holds the publish times cooldown needs. Use it when another caching layer, such as an Artifactory remote repository, sits in front of the proxy and caching artifacts twice only costs storage.
```yaml
storage:
cache_artifacts: false
```
Every download is a fresh upstream fetch, and concurrent requests for the same artifact are not combined. Artifacts with a digest known up front (OCI blobs, Swift archives, Helm charts) are verified while streaming: the response is sent chunked, and on a mismatch the connection is aborted before the response completes so the client never receives a tampered artifact as a good one. The same happens when the upstream connection fails mid-download.
2026-10-18 19:33:13 +02:00
`cache_artifacts: false` cannot be combined with `scanning.enabled` , `storage.direct_serve` , `mirror_api` or `mirror.schedules` , which all need stored artifacts, and the `mirror` command refuses to run with it.
Add storage.cache_artifacts to serve artifacts without storing them (#387)
* Add storage.passthrough to serve artifacts without storing them
With storage.passthrough enabled the proxy streams every artifact from
upstream to the client and never writes it to storage or the cache
database. Metadata filtering, cooldown and the denylist work as before,
so the proxy can sit behind another cache (e.g. an Artifactory remote)
purely as a policy layer without holding a second copy of every package.
Artifacts whose digest is known up front (OCI, Swift, Helm) are verified
while streaming. Their responses are sent chunked and the connection is
aborted on a mismatch, so a client never receives a tampered artifact as
a complete response.
Passthrough is rejected together with scanning, direct_serve and
mirror_api, and the mirror command refuses to run with it, since all of
them depend on stored artifacts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address review: abort truncated streams, guard every cache read
- serveArtifact aborts the response on any body read error, and when fewer
bytes than the declared size were written, not only on a digest mismatch.
A failed or short upstream body in passthrough mode no longer reaches the
client as a complete 200.
- Streamed upstream read failures are logged and counted as
stream_failed upstream errors before the response is aborted.
- checkCache reports a miss when artifact caching is off, so every cache
read path, including Swift archive HEAD requests, ignores entries stored
before the mode was enabled. Denylisted versions are still rejected.
- Rename storage.passthrough to storage.cache_artifacts (default true,
PROXY_STORAGE_CACHE_ARTIFACTS), which names what actually changes.
Tests cover an OCI blob shorter than its Content-Length and an npm tarball
whose chunked response ends without the final chunk, both served through
the HTTP handlers, plus a Swift HEAD request against a cached archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Rename Proxy.Passthrough to StreamArtifacts
Match the storage.cache_artifacts option: the field says what changes
(artifacts are streamed instead of stored) and its zero value keeps the
usual caching behaviour for every Proxy built without the server config,
such as the mirror command and tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 15:28:20 +07:00
Add age-based retention for cached artifacts (#428)
* Add age-based retention for cached artifacts
Cached artifacts stayed forever unless storage.max_size pushed them out.
storage.retention adds a default, per-ecosystem and per-package duration
after which an artifact nobody downloaded gets evicted.
Ecosystems opt in one at a time: a handler registers with
retention.Default, and until it does, configuring that ecosystem (or one
of its packages) fails validation, and the default doesn't touch it. This
commit registers none, so nothing changes yet. Startup logs which
ecosystems the default covers and which it doesn't.
An artifact counts as expired when both its fetch time and its last
access are older than the cutoff. Checking only the last access would
evict a refetched artifact again right away, since clearing a record
keeps its old access time and a cache miss doesn't record a hit.
The sweep scans artifacts in id windows, so no single query holds the
SQLite connection for long, and checks each row against its own rule.
Expired records are cleared and their files go through pending_deletes
instead of being deleted directly, because a buffered hit can make an
artifact that's being served right now look expired. Buffered hits are
flushed before each sweep, and one sweep clears at most 10,000
artifacts.
Since a big first sweep can queue a lot, reclaim now keeps working
through due entries for up to 30 seconds per tick instead of stopping
after 100, and pending_deletes gets an index on queued_at.
Also:
- proxy_artifacts_evicted_total{reason, ecosystem} counts LRU and
retention evictions.
- gradle.build_cache.max_age accepts "7d", as its comment always said.
Refs #306
* Document storage retention
Describes the new storage.retention block, the order rules apply in,
that the default only covers ecosystems that support retention yet, and
what to expect from the first sweep over an old cache. Also lists the
new eviction counter and notes that gradle max_age takes days.
Refs #306
* Fix retention timing on Postgres and tighten the sweep
On Postgres, fetched_at and last_accessed_at are TIMESTAMP columns
without a zone. They hold the local time the proxy wrote, but lib/pq
reads them back labelled UTC. East of UTC that makes every artifact look
hours younger than it is, so the sweep evicted late: two hours in
Berlin, nine in Tokyo. The candidates' times are now read back as local
time, and the database tests run on both SQLite and Postgres, with one
that sets the local zone to Asia/Tokyo.
A few smaller things:
- When a delete fails, reclaim stops for the rest of the minute instead
of trying every queued path. Otherwise a storage outage turns into
thirty seconds of failed deletes and warnings every minute.
- If the buffered hits can't be written, the sweep skips that round.
Running on stale access times could clear something that is being
downloaded right now.
- DefaultCanonical only accepts a package key when rebuilding it gives
the same PURL. For alpine and deb it would otherwise produce keys
that never match.
- sweep_interval has to be at least a minute.
- New tests cover the eviction counter and an artifact that gets a hit
between the scan and the clear.
- The docs no longer say nothing changes without retention, and they
point out that the example's ecosystem and package rules only pass
validation once those ecosystems support retention.
Refs #306
* Reject qualifiers and subpaths in retention package keys
A rule names a whole package. Qualifiers like ?type=pom were dropped on
the way to the stored PURL and the rule quietly covered every file of the
package. Now such a key fails validation.
Refs #306
* Say when a package key doesn't match how packages are stored
A package rule for an ecosystem that supports retention, written in a
form its packages aren't stored under (pkg:npm/babel/core without the
@), failed with "retention is not supported for npm packages". Once the
ecosystem is enabled that is no longer true. Such a key now fails with
"does not match how npm packages are stored" and points to the package
rule forms in the docs. Keys for ecosystems without retention support
keep the old message.
Refs #306
* Note that re-running the mirror keeps packages cached
A mirror run over a package that is still cached goes through the normal
cache lookup and records a download, so the package's age starts over.
The docs only said that mirrored packages age from the time they were
mirrored.
Refs #306
* Say when an ecosystem takes no package rules
Some ecosystems will support retention without package rules, because nothing in their stored records names a package the way a PURL would (conan, helm). A package key for one of them used to fail with "does not match how conan packages are stored; see the package rule forms", which sends people looking for a form that doesn't exist. It now says package rules aren't supported there and points to the ecosystem rule.
Refs #306
* Use helm as the example of an ecosystem without package rules
Conan gets package rules once its handler stores archives under the recipe name, so the config test that shows the refusal now uses helm, whose cache records name the repository, not the chart.
Refs #306
* Say that each cached file ages on its own
The retention docs said each artifact, each version of a package, ages on its own. A version can have several cached files (a jar and its pom, several wheels, Conan sources and binaries), and each of them ages separately.
Refs #306
* Clear an expired artifact and queue its file in one transaction
The sweep cleared the record and queued the file for deletion as two separate writes. If queueing failed, or the process stopped in between, the file stayed in storage with no record pointing at it and no queue entry, and no later sweep would find it again. Both writes now share a transaction, so a failed queue rolls the clear back and the next sweep retries.
Refs #306
* Show artifact evictions on the analytics page
proxy_artifacts_evicted_total had no place on /ui/analytics. The Runtime card now lists evicted artifacts by reason and ecosystem, without the failure tint, and the coverage test records an eviction so it doesn't depend on test order.
Refs #306
* Store artifact times in UTC
fetched_at and last_accessed_at are now written in UTC, and the retention cutoffs are compared in UTC, the same convention #434 uses for the metadata cache. lib/pq sends a time with its offset and a Postgres TIMESTAMP keeps only the wall clock, so UTC on write is what reads back correctly. That replaces reinterpreting the times as local on read.
Rows written before this still hold local time. East of UTC they look younger by the offset until they are rewritten, so retention evicts them a few hours late once; west of UTC that much early.
Refs #306
---------
Co-authored-by: t <t@t>
2026-10-12 10:06:01 +02:00
### Retention
Retention evicts cached artifacts that nobody has downloaded for a while. It works alongside `storage.max_size` : retention removes what has gone unused, and the size limit still applies to whatever is left.
The `ecosystems` and `packages` entries below only pass validation once the ecosystems they name support retention; see the table further down.
```yaml
storage:
retention:
default: "30d"
ecosystems:
npm: "14d"
maven: "0"
packages:
"pkg:npm/lodash": "0"
sweep_interval: "10m"
```
Each cached file ages on its own, counted from the last time the proxy served it, or from when it was fetched if it has not been served since. A newer release of the same package does not shorten the life of older versions. Durations take a `d` suffix for days, and `"0"` or an empty value means never evict by age. With no retention configured, artifacts stay until `max_size` evicts them. `sweep_interval` must be at least `1m` .
2026-10-19 15:43:57 +08:00
A package rule wins over an ecosystem rule, which wins over `default` . Package rules are keyed by package PURL without a version, written in one of the forms the table below lists for the ecosystem. The ecosystems and package rules can only name ecosystems that support retention in the running version; anything else, including a typo, stops the proxy at startup with an error naming the setting. `default` applies only to supported ecosystems, except `generic` , which requires an explicit rule. Support is added one ecosystem at a time, so a later release can make an existing `default` cover more ecosystems. On startup the proxy logs which ecosystems retention supports (`covered` ) and which ones `default` does not reach (`default_not_applied` ), including `generic` .
Generic retention is opt-in because release assets may disappear upstream and cannot then be downloaded again after eviction. An existing `default` never starts evicting generic downloads. Set `ecosystems.generic: "30d"` to cover all generic artifacts, or configure only selected GitHub repositories:
```yaml
storage:
retention:
default: "30d" # Other supported ecosystems; never generic.
packages:
"pkg:generic/jqlang%2Fjq": "30d"
```
The package rule covers every release and asset in that repository, but each cached file ages separately. Other generic repositories remain exempt from age-based eviction. A package rule of `"0"` also exempts that repository when `ecosystems.generic` is enabled. The independent `storage.max_size` limit still applies.
Add age-based retention for cached artifacts (#428)
* Add age-based retention for cached artifacts
Cached artifacts stayed forever unless storage.max_size pushed them out.
storage.retention adds a default, per-ecosystem and per-package duration
after which an artifact nobody downloaded gets evicted.
Ecosystems opt in one at a time: a handler registers with
retention.Default, and until it does, configuring that ecosystem (or one
of its packages) fails validation, and the default doesn't touch it. This
commit registers none, so nothing changes yet. Startup logs which
ecosystems the default covers and which it doesn't.
An artifact counts as expired when both its fetch time and its last
access are older than the cutoff. Checking only the last access would
evict a refetched artifact again right away, since clearing a record
keeps its old access time and a cache miss doesn't record a hit.
The sweep scans artifacts in id windows, so no single query holds the
SQLite connection for long, and checks each row against its own rule.
Expired records are cleared and their files go through pending_deletes
instead of being deleted directly, because a buffered hit can make an
artifact that's being served right now look expired. Buffered hits are
flushed before each sweep, and one sweep clears at most 10,000
artifacts.
Since a big first sweep can queue a lot, reclaim now keeps working
through due entries for up to 30 seconds per tick instead of stopping
after 100, and pending_deletes gets an index on queued_at.
Also:
- proxy_artifacts_evicted_total{reason, ecosystem} counts LRU and
retention evictions.
- gradle.build_cache.max_age accepts "7d", as its comment always said.
Refs #306
* Document storage retention
Describes the new storage.retention block, the order rules apply in,
that the default only covers ecosystems that support retention yet, and
what to expect from the first sweep over an old cache. Also lists the
new eviction counter and notes that gradle max_age takes days.
Refs #306
* Fix retention timing on Postgres and tighten the sweep
On Postgres, fetched_at and last_accessed_at are TIMESTAMP columns
without a zone. They hold the local time the proxy wrote, but lib/pq
reads them back labelled UTC. East of UTC that makes every artifact look
hours younger than it is, so the sweep evicted late: two hours in
Berlin, nine in Tokyo. The candidates' times are now read back as local
time, and the database tests run on both SQLite and Postgres, with one
that sets the local zone to Asia/Tokyo.
A few smaller things:
- When a delete fails, reclaim stops for the rest of the minute instead
of trying every queued path. Otherwise a storage outage turns into
thirty seconds of failed deletes and warnings every minute.
- If the buffered hits can't be written, the sweep skips that round.
Running on stale access times could clear something that is being
downloaded right now.
- DefaultCanonical only accepts a package key when rebuilding it gives
the same PURL. For alpine and deb it would otherwise produce keys
that never match.
- sweep_interval has to be at least a minute.
- New tests cover the eviction counter and an artifact that gets a hit
between the scan and the clear.
- The docs no longer say nothing changes without retention, and they
point out that the example's ecosystem and package rules only pass
validation once those ecosystems support retention.
Refs #306
* Reject qualifiers and subpaths in retention package keys
A rule names a whole package. Qualifiers like ?type=pom were dropped on
the way to the stored PURL and the rule quietly covered every file of the
package. Now such a key fails validation.
Refs #306
* Say when a package key doesn't match how packages are stored
A package rule for an ecosystem that supports retention, written in a
form its packages aren't stored under (pkg:npm/babel/core without the
@), failed with "retention is not supported for npm packages". Once the
ecosystem is enabled that is no longer true. Such a key now fails with
"does not match how npm packages are stored" and points to the package
rule forms in the docs. Keys for ecosystems without retention support
keep the old message.
Refs #306
* Note that re-running the mirror keeps packages cached
A mirror run over a package that is still cached goes through the normal
cache lookup and records a download, so the package's age starts over.
The docs only said that mirrored packages age from the time they were
mirrored.
Refs #306
* Say when an ecosystem takes no package rules
Some ecosystems will support retention without package rules, because nothing in their stored records names a package the way a PURL would (conan, helm). A package key for one of them used to fail with "does not match how conan packages are stored; see the package rule forms", which sends people looking for a form that doesn't exist. It now says package rules aren't supported there and points to the ecosystem rule.
Refs #306
* Use helm as the example of an ecosystem without package rules
Conan gets package rules once its handler stores archives under the recipe name, so the config test that shows the refusal now uses helm, whose cache records name the repository, not the chart.
Refs #306
* Say that each cached file ages on its own
The retention docs said each artifact, each version of a package, ages on its own. A version can have several cached files (a jar and its pom, several wheels, Conan sources and binaries), and each of them ages separately.
Refs #306
* Clear an expired artifact and queue its file in one transaction
The sweep cleared the record and queued the file for deletion as two separate writes. If queueing failed, or the process stopped in between, the file stayed in storage with no record pointing at it and no queue entry, and no later sweep would find it again. Both writes now share a transaction, so a failed queue rolls the clear back and the next sweep retries.
Refs #306
* Show artifact evictions on the analytics page
proxy_artifacts_evicted_total had no place on /ui/analytics. The Runtime card now lists evicted artifacts by reason and ecosystem, without the failure tint, and the coverage test records an eviction so it doesn't depend on test order.
Refs #306
* Store artifact times in UTC
fetched_at and last_accessed_at are now written in UTC, and the retention cutoffs are compared in UTC, the same convention #434 uses for the metadata cache. lib/pq sends a time with its offset and a Postgres TIMESTAMP keeps only the wall clock, so UTC on write is what reads back correctly. That replaces reinterpreting the times as local on read.
Rows written before this still hold local time. East of UTC they look younger by the offset until they are rewritten, so retention evicts them a few hours late once; west of UTC that much early.
Refs #306
---------
Co-authored-by: t <t@t>
2026-10-12 10:06:01 +02:00
| Ecosystem key | Retention supported | Package rule form |
|---------------|---------------------|-------------------|
2026-10-11 01:07:07 +02:00
| `alpine` | yes | `pkg:apk/curl` or `pkg:apk/alpine/curl` ; case doesn't matter. A rule covers the package in every configured APK repository. Alpine mirrors keep only the current build of each package per release, so an evicted older one can't be fetched again. |
2026-10-10 22:10:31 +02:00
| `cargo` | yes | `pkg:cargo/serde` , the crate name as published; case, `-` and `_` have to match, so `pkg:cargo/inflector` doesn't cover `Inflector` . |
2026-10-11 00:25:33 +02:00
| `composer` | yes | `pkg:composer/symfony/console` ; case doesn't matter. A key without a vendor is refused. |
2026-10-11 18:01:15 +02:00
| `conan` | yes | `pkg:conan/zlib` , the recipe name; a rule covers every version, revision and user/channel of it. |
2026-10-10 23:37:32 +02:00
| `conda` | yes | `pkg:conda/conda-forge/numpy` , with the channel as the namespace; the package name's case doesn't matter. A key without a channel is refused. |
2026-10-11 01:50:16 +02:00
| `cran` | yes | `pkg:cran/ggplot2` , the package name as published, case included (`pkg:cran/Rcpp` ). A rule covers source and binary builds alike. |
2026-10-11 01:12:41 +02:00
| `deb` | yes | `pkg:deb/curl` ; case doesn't matter. A vendor namespace such as `pkg:deb/debian/curl` is accepted and dropped, so a rule covers the package from every configured archive. Debian and Ubuntu archives only keep the current version per suite, so an evicted older one can't be fetched again. |
2026-10-22 12:13:20 +02:00
| `freebsd` | yes | `pkg:bsd/freebsd/zsh` , the package name as published, case included (`pkg:bsd/freebsd/FreeBSD-acct` ). A rule covers the package in every configured repository and ABI. Only packages under `Hashed/` are cached, and mirrors keep only the current build, so an evicted older one can't be fetched again. |
2026-10-12 19:55:10 +02:00
| `gem` | yes | `pkg:gem/rails` , the gem name as published, case included. |
2026-10-19 15:43:57 +08:00
| `generic` | yes, explicit rules only | `pkg:generic/owner%2Frepo` for a GitHub Release repository such as `jqlang/jq` . The ecosystem rule covers all cached version-pinned generic downloads, including GitHub Release assets and configured `generic_artifacts` . The global `default` never applies; metadata/API responses are not artifact retention candidates. |
2026-10-12 14:04:16 +02:00
| `golang` | yes | `pkg:golang/github.com/stretchr/testify` , the module path as in `go.mod` ; case doesn't matter. A major version like `/v2` (or `.v3` on gopkg.in) is a module of its own and needs its own rule. The `!` -escaped form from proxy URLs and the module cache (`github.com/!azure/...` ) is refused. |
2026-10-11 01:55:16 +02:00
| `helm` | yes | No package rules: charts are cached per repository and digest, not under the chart name. The `helm` rule and `default` cover charts from HTTP repositories; charts pulled over OCI fall under `oci` . |
2026-10-12 19:57:57 +02:00
| `hex` | yes | `pkg:hex/phoenix` , the package name as published. |
2026-10-11 01:38:14 +02:00
| `julia` | yes | No package rules: a julia PURL needs a `uuid` qualifier, which the cache doesn't store. The `julia` rule and `default` cover package tarballs, artifacts and registry snapshots alike. |
2026-10-12 13:58:09 +02:00
| `maven` | yes | `pkg:maven/com.google.guava/guava` , group and artifact as published, case included. Gradle build cache entries are not covered; they use `gradle.build_cache.max_age` . |
2026-10-12 14:21:38 +02:00
| `npm` | yes | `pkg:npm/lodash` , `pkg:npm/@babel/core` ; the name has to match exactly, case included, so `pkg:npm/jsonstream` doesn't cover `JSONStream` . |
2026-10-12 19:56:38 +02:00
| `nuget` | yes | `pkg:nuget/Newtonsoft.Json` ; case doesn't matter. |
2026-10-12 14:00:57 +02:00
| `oci` | yes | `pkg:oci/library/nginx` , the repository name of the pull. A named upstream keeps its prefix, also when containerd reaches it through `ns` : `ghcr.io/owner/app` pulled that way is `pkg:oci/upstream/ghcr/owner/app` . Homebrew bottles use Homebrew's image name, with `@` written as `/` and `+` as `x` (`openssl@3` is `pkg:oci/homebrew/core/openssl/3` ). Helm charts pulled over OCI are stored like images. Only blobs are evicted; manifests stay in the metadata cache. |
2026-10-10 23:15:44 +02:00
| `pub` | yes | `pkg:pub/http` , the package name as published. |
2026-10-12 14:06:37 +02:00
| `pypi` | yes | `pkg:pypi/requests` ; case and single `_` /`-` don't matter. Dots and repeated separators are kept as written, so a project like `zope.interface` can be stored as `zope.interface` and `zope-interface` ; add a rule for each. |
2026-10-11 01:21:03 +02:00
| `rpm` | yes | `pkg:rpm/curl` , the name as packaged, case included (`pkg:rpm/NetworkManager` ). A vendor namespace such as `pkg:rpm/fedora/curl` is accepted and dropped. Fedora and EPEL update repositories keep only the latest build of a package, so an evicted older one may not be fetchable again. |
Accept Swift source repository PURLs in the denylist and retention rules (#641)
* Accept Swift source repository PURLs in the denylist and retention rules
Closes #613. pkg:swift/github.com/apple/swift-nio, as SBOM tools and GHSA
write Swift packages, is resolved to its registry identity through the
upstream's /identifiers:
- Denylist entries are resolved at startup, and the proxy refuses to start
when one can't be, since a skipped entry would let the version through.
- Retention keys are accepted at validation and kept pending. The
retention loop resolves them before each sweep, and Swift isn't swept
while any is pending, so a rule that keeps a package is never skipped.
The mirror's source URL helper moves to the handler as SwiftSourceURL.
* Resolve Swift source denylist entries in proxy mirror too
serve resolved source-form Swift denylist entries through the upstream,
but proxy mirror built its denylist directly, so such an entry never
matched a mirror download. Both now use server.ConfigureDenylist, and
neither runs when an entry can't be resolved.
* Apply Swift source policies to every listed identity
A Swift registry may list several identities for one source repository,
but a denylist entry or retention rule naming the repository only
covered the first. Denying pkg:swift/github.com/acme/widget@1.0.0 let
acme.widget through when the registry listed other.widget first, and a
never-evict rule cleared the cached apple.swift-nio archive when
other.swift-nio came first.
Both now cover every identity the registry lists. A retention rule that
conflicts with another rule for any of them stays pending, with none
applied. Mirroring a source PURL still downloads the first identity.
2026-10-31 12:34:05 +01:00
| `swift` | yes | `pkg:swift/apple/swift-nio` , with the registry scope as the namespace, or the stored form `pkg:generic/swift-registry/apple.swift-nio` ; case doesn't matter. A source repository key like `pkg:swift/github.com/apple/swift-nio` is looked up through `upstream.swift` 's `/identifiers` before each sweep and applies to every identity listed for it; until every such key resolves, Swift artifacts aren't swept, so a rule that keeps a package is never skipped. |
Add age-based retention for cached artifacts (#428)
* Add age-based retention for cached artifacts
Cached artifacts stayed forever unless storage.max_size pushed them out.
storage.retention adds a default, per-ecosystem and per-package duration
after which an artifact nobody downloaded gets evicted.
Ecosystems opt in one at a time: a handler registers with
retention.Default, and until it does, configuring that ecosystem (or one
of its packages) fails validation, and the default doesn't touch it. This
commit registers none, so nothing changes yet. Startup logs which
ecosystems the default covers and which it doesn't.
An artifact counts as expired when both its fetch time and its last
access are older than the cutoff. Checking only the last access would
evict a refetched artifact again right away, since clearing a record
keeps its old access time and a cache miss doesn't record a hit.
The sweep scans artifacts in id windows, so no single query holds the
SQLite connection for long, and checks each row against its own rule.
Expired records are cleared and their files go through pending_deletes
instead of being deleted directly, because a buffered hit can make an
artifact that's being served right now look expired. Buffered hits are
flushed before each sweep, and one sweep clears at most 10,000
artifacts.
Since a big first sweep can queue a lot, reclaim now keeps working
through due entries for up to 30 seconds per tick instead of stopping
after 100, and pending_deletes gets an index on queued_at.
Also:
- proxy_artifacts_evicted_total{reason, ecosystem} counts LRU and
retention evictions.
- gradle.build_cache.max_age accepts "7d", as its comment always said.
Refs #306
* Document storage retention
Describes the new storage.retention block, the order rules apply in,
that the default only covers ecosystems that support retention yet, and
what to expect from the first sweep over an old cache. Also lists the
new eviction counter and notes that gradle max_age takes days.
Refs #306
* Fix retention timing on Postgres and tighten the sweep
On Postgres, fetched_at and last_accessed_at are TIMESTAMP columns
without a zone. They hold the local time the proxy wrote, but lib/pq
reads them back labelled UTC. East of UTC that makes every artifact look
hours younger than it is, so the sweep evicted late: two hours in
Berlin, nine in Tokyo. The candidates' times are now read back as local
time, and the database tests run on both SQLite and Postgres, with one
that sets the local zone to Asia/Tokyo.
A few smaller things:
- When a delete fails, reclaim stops for the rest of the minute instead
of trying every queued path. Otherwise a storage outage turns into
thirty seconds of failed deletes and warnings every minute.
- If the buffered hits can't be written, the sweep skips that round.
Running on stale access times could clear something that is being
downloaded right now.
- DefaultCanonical only accepts a package key when rebuilding it gives
the same PURL. For alpine and deb it would otherwise produce keys
that never match.
- sweep_interval has to be at least a minute.
- New tests cover the eviction counter and an artifact that gets a hit
between the scan and the clear.
- The docs no longer say nothing changes without retention, and they
point out that the example's ecosystem and package rules only pass
validation once those ecosystems support retention.
Refs #306
* Reject qualifiers and subpaths in retention package keys
A rule names a whole package. Qualifiers like ?type=pom were dropped on
the way to the stored PURL and the rule quietly covered every file of the
package. Now such a key fails validation.
Refs #306
* Say when a package key doesn't match how packages are stored
A package rule for an ecosystem that supports retention, written in a
form its packages aren't stored under (pkg:npm/babel/core without the
@), failed with "retention is not supported for npm packages". Once the
ecosystem is enabled that is no longer true. Such a key now fails with
"does not match how npm packages are stored" and points to the package
rule forms in the docs. Keys for ecosystems without retention support
keep the old message.
Refs #306
* Note that re-running the mirror keeps packages cached
A mirror run over a package that is still cached goes through the normal
cache lookup and records a download, so the package's age starts over.
The docs only said that mirrored packages age from the time they were
mirrored.
Refs #306
* Say when an ecosystem takes no package rules
Some ecosystems will support retention without package rules, because nothing in their stored records names a package the way a PURL would (conan, helm). A package key for one of them used to fail with "does not match how conan packages are stored; see the package rule forms", which sends people looking for a form that doesn't exist. It now says package rules aren't supported there and points to the ecosystem rule.
Refs #306
* Use helm as the example of an ecosystem without package rules
Conan gets package rules once its handler stores archives under the recipe name, so the config test that shows the refusal now uses helm, whose cache records name the repository, not the chart.
Refs #306
* Say that each cached file ages on its own
The retention docs said each artifact, each version of a package, ages on its own. A version can have several cached files (a jar and its pom, several wheels, Conan sources and binaries), and each of them ages separately.
Refs #306
* Clear an expired artifact and queue its file in one transaction
The sweep cleared the record and queued the file for deletion as two separate writes. If queueing failed, or the process stopped in between, the file stayed in storage with no record pointing at it and no queue entry, and no later sweep would find it again. Both writes now share a transaction, so a failed queue rolls the clear back and the next sweep retries.
Refs #306
* Show artifact evictions on the analytics page
proxy_artifacts_evicted_total had no place on /ui/analytics. The Runtime card now lists evicted artifacts by reason and ecosystem, without the failure tint, and the coverage test records an eviction so it doesn't depend on test order.
Refs #306
* Store artifact times in UTC
fetched_at and last_accessed_at are now written in UTC, and the retention cutoffs are compared in UTC, the same convention #434 uses for the metadata cache. lib/pq sends a time with its offset and a Postgres TIMESTAMP keeps only the wall clock, so UTC on write is what reads back correctly. That replaces reinterpreting the times as local on read.
Rows written before this still hold local time. East of UTC they look younger by the offset until they are rewritten, so retention evicts them a few hours late once; west of UTC that much early.
Refs #306
---------
Co-authored-by: t <t@t>
2026-10-12 10:06:01 +02:00
An evicted artifact is fetched again from upstream on the next request. Packages preloaded with `proxy mirror` age from the time they were mirrored, so an unused mirrored package is evicted like any other. Running `proxy mirror` again over a package that is still cached counts as a download and restarts its clock, so packages a scheduled mirror job keeps listing never age out. A registry that removes old releases (Linux distribution archives in particular) cannot serve an evicted version again, so set those ecosystems' rules with that in mind.
Expired artifacts are not deleted right away. Their records are cleared and the files wait in the same queue a replaced artifact goes through, for at least an hour (or `storage.direct_serve_ttl` if longer), so downloads already in progress can finish. `storage.max_size` counts the space as free from the moment the record is cleared. Once that wait is over, each minute the queue is worked through for up to 30 seconds, pausing until the next minute after a delete fails. One sweep clears at most 10,000 artifacts, so the first sweep over a large, old cache spreads its evictions, and the downloads that bring popular artifacts back, over several intervals.
With several proxies sharing one database and `database.hit_flush_interval` set, a proxy's sweep cannot see downloads another proxy has not written yet. Keep retention durations well above the flush interval.
2026-01-29 16:33:09 +00:00
### Amazon S3
```yaml
storage:
url: "s3://my-bucket"
```
Configure credentials via environment variables:
```bash
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
export AWS_REGION=us-east-1
```
2026-10-25 17:22:23 +01:00
Objects up to 16 MiB are buffered in memory and sent in one request. Larger ones are uploaded in 5 MiB parts, two at a time, so a large artifact being cached holds about 30 MiB of upload buffers. Azure Blob Storage uploads use the same part size and concurrency.
2026-01-29 16:33:09 +00:00
### S3-Compatible (MinIO, etc.)
```yaml
storage:
url: "s3://my-bucket?endpoint=http://localhost:9000&disableSSL=true&s3ForcePathStyle=true"
```
## Database
The proxy supports SQLite (default) and PostgreSQL for storing package metadata.
### SQLite
```yaml
database:
driver: "sqlite"
path: "./cache/proxy.db"
```
| Config | Environment | Flag | Description |
|--------|-------------|------|-------------|
| `database.driver` | `PROXY_DATABASE_DRIVER` | `-database-driver` | `sqlite` or `postgres` |
| `database.path` | `PROXY_DATABASE_PATH` | `-database-path` | SQLite file path |
2026-10-25 17:35:49 +01:00
The database runs in WAL mode. Writes go through a single connection, since SQLite allows one writer at a time. Reads use a separate pool of read-only connections (one per CPU, between 2 and 8), so request lookups don't wait behind writes, stats queries or eviction scans. An in-memory database (`:memory:` ) uses one connection for both.
2026-01-29 16:33:09 +00:00
### PostgreSQL
```yaml
database:
driver: "postgres"
url: "postgres://user:password@localhost:5432/proxy ?sslmode=disable"
```
| Config | Environment | Flag | Description |
|--------|-------------|------|-------------|
| `database.url` | `PROXY_DATABASE_URL` | `-database-url` | PostgreSQL connection URL |
2026-10-23 16:15:24 +02:00
### Several instances
Several instances can serve one cache when they share a PostgreSQL database and storage. The storage can be S3 or a `file://` directory on a volume every instance mounts, such as NFS, CephFS or a Kubernetes `ReadWriteMany` volume. Each fetch writes its own file and renames it into place, and records are updated with compare-and-swap, so concurrent fetches and a crashed instance leave no partial files or records that don't match a file. SQLite only supports a single instance.
Instances coordinate through PostgreSQL advisory locks:
- Instances started together create the schema and run migrations one after the other.
- Eviction, reclaim, retention and the Gradle build cache sweep run on one instance at a time. An instance that finds a job running elsewhere skips that pass.
- A [scheduled mirror ](#scheduled-refresh ) runs on whichever instance reaches it first in each interval.
2026-10-25 17:34:08 +01:00
Cache misses are fetched from upstream once across instances. The instance that fetches holds a lease on the miss in the `fetch_leases` table and renews it while it downloads. Another instance missing the same file waits until it is stored and serves the stored copy; `proxy_fetch_lease_waits_total` counts those waits. If the fetch fails, the instances waiting on it stop waiting together: each serves its stale metadata copy if it has one, and otherwise fetches itself, so a failing upstream costs every instance one attempt at the same time rather than one after another. When the fetch failed because the upstream rate limited it, the waiters without a stale copy answer 503 with the upstream's `Retry-After` instead of fetching. A lease of an instance that stopped expires after 30 seconds. Metadata is shared this way only with [`cache_metadata` ](#metadata-caching ) and a `metadata_ttl` above zero, since without a stored copy there is nothing to wait for. A cache hit costs no extra query; a miss costs one.
Rate-limit holds (see `upstream_holds` under [Health Check ](../README.md#health-check )) are shared through the `upstream_holds` table. When one instance gets a 429, or a 503 with `Retry-After` , it records the host and the end of the hold, and the other instances stop sending to that host too instead of each waiting for its own 429. Each instance rereads the table at most every two seconds, so another instance can still send for up to two seconds after a hold starts. A hold is also recorded with SQLite, where it then outlasts a restart.
2026-10-23 16:54:54 +02:00
Each instance needs its own [`git.cache_dir` ](#git-repositories ).
2026-10-23 16:15:24 +02:00
2026-09-30 23:09:04 -07:00
### Hit counts
2026-10-25 17:35:49 +01:00
Every cache hit updates the artifact's hit count and last-access time, which the stats pages and LRU eviction use. Rather than writing each hit as its own transaction, the proxy counts hits in memory and writes them together every `hit_flush_interval` . SQLite allows one writer at a time, and the proxy writes through a single SQLite connection, so with a write per hit, concurrent downloads queue behind each other.
2026-09-30 23:09:04 -07:00
```yaml
database:
hit_flush_interval: "1s"
```
| Config | Environment | Flag | Description |
|--------|-------------|------|-------------|
| `database.hit_flush_interval` | `PROXY_DATABASE_HIT_FLUSH_INTERVAL` | - | How often batched hits are written (default `1s` ). `0` writes each hit as it happens. |
Hits not yet written are lost if the process is killed; a normal shutdown writes them.
2026-01-29 16:33:09 +00:00
## Logging
```yaml
log:
level: "info"
format: "text"
```
| Config | Environment | Flag | Values |
|--------|-------------|------|--------|
| `log.level` | `PROXY_LOG_LEVEL` | `-log-level` | `debug` , `info` , `warn` , `error` |
| `log.format` | `PROXY_LOG_FORMAT` | `-log-format` | `text` , `json` |
2026-08-16 18:07:39 +01:00
## Access Log
The optional access log records client requests and each HTTP exchange with an upstream registry. It is always written as JSONL, with one JSON object per line. Records for the same client request share a `request_id` .
```yaml
access_log:
path: "/var/log/proxy/access.jsonl"
```
| Config | Environment | Flag | Description |
|--------|-------------|------|-------------|
| `access_log.path` | `PROXY_ACCESS_LOG_PATH` | `-access-log` | File to append JSONL records to; empty disables the log |
2026-10-01 14:32:21 +02:00
Each client request record carries `remote_addr` (the TCP peer, host and port), `remote_ip` (the address the request is attributed to), `user_agent` , `client` (the tool name derived from the User-Agent), `ecosystem` , and `bytes` (the response body size written to the client). The structured log carries the same `client` , `remote` and `remote_ip` fields.
`remote_ip` follows the top-level `trust_forwarded_for` setting. Enable that only when the proxy sits behind a load balancer or ingress that sets the header: any client can send `X-Forwarded-For` , so behind such a hop it is the only way to see the real caller, but in front of one it lets a caller choose what address it is logged as. It can also choose a fresh one per request, which fills the bounded source table on `/ui/analytics` and evicts the genuine callers from it. `remote_addr` is unaffected and always records the TCP peer.
2026-08-16 18:07:39 +01:00
The parent directory must exist and be writable when the proxy starts. A newly created log file is readable and writable only by the proxy process owner.
A request that receives a rate limit response from an upstream can produce records like these:
```json
{"time":"2026-08-16T12:00:00Z","event":"upstream","request_id":"host/example-000001","method":"GET","url":"https://registry.example/packages/example","status_code":429,"duration_ms":42}
{"time":"2026-08-16T12:00:00Z","event":"request","request_id":"host/example-000001","method":"GET","path":"/npm/example","status_code":502,"duration_ms":43,"remote_addr":"192.0.2.10:41234"}
```
Upstream retries and OCI authentication calls are separate `upstream` records, so the log preserves every status returned over the wire. Network failures have an `error` field and no `status_code` . URL credentials, query strings, and fragments are omitted from both upstream URLs and client paths.
2026-01-29 16:33:09 +00:00
## Upstream Registries
2026-08-29 11:19:54 +01:00
Each upstream used by a built-in package route can be set in YAML or JSON under `upstream` , or with its matching environment variable. Existing installations keep the same public upstreams by default. Trailing slashes are ignored.
| Config | Environment | Default |
|--------|-------------|---------|
| `upstream.allow_private_hosts` | `PROXY_UPSTREAM_ALLOW_PRIVATE_HOSTS` | `[]` |
| `upstream.allow_loopback` | `PROXY_UPSTREAM_ALLOW_LOOPBACK` | `false` |
| `upstream.npm` | `PROXY_UPSTREAM_NPM` | `https://registry.npmjs.org` |
2026-09-02 07:13:12 -04:00
| `upstream.npm_full_metadata` | `PROXY_UPSTREAM_NPM_FULL_METADATA` | `false` |
2026-08-29 11:19:54 +01:00
| `upstream.cargo` | `PROXY_UPSTREAM_CARGO` | `https://index.crates.io` |
| `upstream.cargo_download` | `PROXY_UPSTREAM_CARGO_DOWNLOAD` | `https://static.crates.io/crates` |
| `upstream.gem` | `PROXY_UPSTREAM_GEM` | `https://rubygems.org` |
| `upstream.go` | `PROXY_UPSTREAM_GO` | `https://proxy.golang.org` |
2026-10-12 22:35:23 +02:00
| `upstream.go_sumdbs_offline` | `PROXY_UPSTREAM_GO_SUMDBS_OFFLINE` | `false` |
Serve the Go vulnerability database at /go/vulndb
govulncheck reads vuln.go.dev directly, so on networks that only reach
the proxy it cannot run. upstream.go_vulndb (default https://vuln.go.dev)
is now served at /go/vulndb for `govulncheck -db`.
Only the files the database API defines are passed on (index/db,
index/modules, index/vulns and ID/{id}, as .json and .json.gz), with IDs
limited to letters, digits, '-' and '_'. They are stored and served
byte-for-byte; .json.gz files are gzip data without Content-Encoding.
All files, entries included, follow the metadata TTL since entries change
when their modified date does, and fall back to the cached copy when the
database is unreachable. Non-JSON responses are refused so a block page
does not replace the copy kept for an outage. HEAD is allowed under
/vulndb/, because govulncheck checks a non-default database with HEAD on
index/modules.json.gz.
2026-10-27 16:16:04 +01:00
| `upstream.go_vulndb` | `PROXY_UPSTREAM_GO_VULNDB` | `https://vuln.go.dev` |
2026-08-29 11:19:54 +01:00
| `upstream.hex` | `PROXY_UPSTREAM_HEX` | `https://repo.hex.pm` |
| `upstream.hex_api` | `PROXY_UPSTREAM_HEX_API` | `https://hex.pm` |
| `upstream.pub` | `PROXY_UPSTREAM_PUB` | `https://pub.dev` |
| `upstream.pypi` | `PROXY_UPSTREAM_PYPI` | `https://pypi.org` |
| `upstream.pypi_download` | `PROXY_UPSTREAM_PYPI_DOWNLOAD` | `https://files.pythonhosted.org` |
| `upstream.maven` | `PROXY_UPSTREAM_MAVEN` | `https://repo1.maven.org/maven2` |
| `upstream.gradle_plugin_portal` | `PROXY_UPSTREAM_GRADLE_PLUGIN_PORTAL` | `https://plugins.gradle.org/m2` |
| `upstream.nuget` | `PROXY_UPSTREAM_NUGET` | `https://api.nuget.org` |
| `upstream.nuget_search` | `PROXY_UPSTREAM_NUGET_SEARCH` | `https://azuresearch-usnc.nuget.org` |
| `upstream.composer` | `PROXY_UPSTREAM_COMPOSER` | `https://packagist.org` |
| `upstream.composer_repository` | `PROXY_UPSTREAM_COMPOSER_REPOSITORY` | `https://repo.packagist.org` |
| `upstream.conan` | `PROXY_UPSTREAM_CONAN` | `https://center.conan.io` |
2026-10-19 12:29:39 +05:30
| `upstream.chef` | `PROXY_UPSTREAM_CHEF` | `https://supermarket.chef.io` |
2026-08-29 11:19:54 +01:00
| `upstream.conda` | `PROXY_UPSTREAM_CONDA` | `https://conda.anaconda.org` |
| `upstream.cran` | `PROXY_UPSTREAM_CRAN` | `https://cloud.r-project.org` |
| `upstream.julia` | `PROXY_UPSTREAM_JULIA` | `https://pkg.julialang.org` |
2026-09-02 13:06:34 +01:00
| `upstream.swift` | `PROXY_UPSTREAM_SWIFT` | `https://tuist.dev/api/registry/swift` |
2026-08-29 11:19:54 +01:00
| `upstream.oci_default` | `PROXY_UPSTREAM_OCI_DEFAULT` | `https://registry-1.docker.io` |
| `upstream.debian` | `PROXY_UPSTREAM_DEBIAN` | `http://deb.debian.org/debian` |
| `upstream.rpm` | `PROXY_UPSTREAM_RPM` | `https://dl.fedoraproject.org/pub/fedora/linux` |
2026-09-03 16:59:12 +01:00
| `upstream.homebrew_api` | `PROXY_UPSTREAM_HOMEBREW_API` | `https://formulae.brew.sh/api` |
| `upstream.homebrew_artifact` | `PROXY_UPSTREAM_HOMEBREW_ARTIFACT` | `https://ghcr.io` |
2026-08-29 11:19:54 +01:00
Private, ULA, CGNAT, and loopback addresses are rejected by default. Add each private upstream hostname or IP address to `upstream.allow_private_hosts` . The matching environment variable accepts a comma-separated list. Loopback upstreams also require `upstream.allow_loopback: true` . That setting permits upstream requests and redirects to reach any loopback address.
2026-01-29 16:33:09 +00:00
Check upstream destinations sent through a forward proxy
The upstream client refuses loopback, private and link-local addresses
when it dials, but with HTTP_PROXY or HTTPS_PROXY set it dials the
forward proxy, so only the forward proxy's address was checked. A URL
taken from upstream metadata, such as a Helm chart URL, a Composer dist
URL, a token realm or a redirect, could then reach 169.254.169.254 or
an internal host through the forward proxy.
The transport's proxy function now resolves the destination of every
request it sends to a forward proxy and applies the same rules,
including upstream.allow_loopback and upstream.allow_private_hosts,
before handing it over. A name that cannot be resolved locally is still
passed on, since some networks only resolve public names at the
forward proxy. Direct requests are unchanged.
Git fetches and token requests use the same transport, so they are
covered too.
2026-10-24 08:07:13 +02:00
Upstream requests honour `HTTP_PROXY` , `HTTPS_PROXY` and `NO_PROXY` . When a request goes through a forward proxy, these rules apply twice: to the forward proxy's own address, which therefore has to be listed in `upstream.allow_private_hosts` if it is private, and to the request's destination, which the proxy resolves before handing the request over. Link-local addresses such as `169.254.169.254` are always refused. A destination the proxy cannot resolve itself is passed to the forward proxy, since some networks only resolve public names there; the forward proxy's own policy decides what such a name may reach.
2026-01-29 16:33:09 +00:00
```yaml
upstream:
2026-08-29 11:19:54 +01:00
allow_private_hosts:
- "upstream-proxy.internal"
pypi: "http://upstream-proxy.internal/pypi"
pypi_download: "http://upstream-proxy.internal/pypi"
```
For protocols that use separate metadata and download services, configure both values. They may point to the same endpoint when chaining proxies:
```yaml
upstream:
pypi: "https://upstream-proxy.example.com/pypi"
pypi_download: "https://upstream-proxy.example.com/pypi"
nuget: "https://upstream-proxy.example.com/nuget"
nuget_search: "https://upstream-proxy.example.com/nuget"
composer: "https://upstream-proxy.example.com/composer"
composer_repository: "https://upstream-proxy.example.com/composer"
```
`upstream.hex_api` is used for cooldown timestamps and must expose Hex's `/api/packages/{name}` JSON endpoint.
2026-10-19 12:29:39 +05:30
`upstream.chef` is the Supermarket served at `/chef/` , for Berkshelf (`source "https://proxy.example.com/chef"` ) and Policyfiles (`default_source :supermarket, "https://proxy.example.com/chef"` ). The upstream has to serve the Supermarket API: `/universe` , `/api/v1/cookbooks/{name}` , its versions and their downloads. A private Supermarket that needs credentials takes them from `upstream.auth` , scoped to its URL. A download redirect to another host, such as the storage bucket behind Supermarket, does not receive them:
```yaml
upstream:
chef: "https://supermarket.example.com"
auth:
"https://supermarket.example.com":
type: bearer
token: "${SUPERMARKET_TOKEN}"
```
Add package routes for private Composer and npm registries (#426)
* Add Composer package routes for private repositories
- Add upstream.composer_routes mapping package name patterns to repositories
- Serve matching packages only from their route, with no fallback to the default repository
- Qualify metadata and artifact cache keys with the route's upstream
- Give routed versions an empty notification-url so installs are not reported to notify-batch
- Document routes in configuration docs, example config and README
* Add npm package routes for private registries
- Add upstream.npm_routes mapping package name patterns to registries
- Serve matching packages only from their route, with no fallback to upstream.npm
- Resolve and validate tarball URLs against the route's registry
- Qualify metadata and artifact cache keys with the route's upstream
- Document npm routes and the audit endpoint limitation
* Share routed artifact filename qualification between handlers
* Match Composer routes without the ~dev suffix
Packagist serves a package's dev versions from {name}~dev.json, and the
handler matched routes against the name with ~dev still attached. An exact
route for example/library therefore missed /p2/example/library~dev.json, and
the private package's dev metadata came from the default repository.
- Match routes against the package name without ~dev, keeping it in the
upstream URL and the cache key so the two files stay apart
- Add a regression test with an exact route
* Keep routed npm publish times out of the shared versions row
The download cooldown check stores publish times under the plain version
PURL, which does not say which registry a version came from. A time read
from the default registry was trusted for a private version of the same
name, so a private release published an hour ago passed a seven-day
cooldown when the public one was a month old.
- Skip the stored row for routed packages, both reading and writing, and
take the publish time from the route's metadata, which is cached under
the route
- This also stops a private time from holding back the public version, and
a route's previous upstream from lending its times to the new one
- Cover adding a route after a public download, the reverse order, and
moving a route to another upstream
2026-10-11 19:06:18 +02:00
### Private package routes
Clients that use the proxy as their only registry can still install private packages. A package route sends every package whose name matches a pattern to a private registry, and never to the default upstream:
```yaml
upstream:
composer_routes:
"example/*": "https://packages.example.com/composer"
npm_routes:
"@example/ *": "https://packages.example.com/npm"
allow_private_hosts:
- "packages.example.com"
auth:
"https://packages.example.com":
type: bearer
token: "${PRIVATE_REGISTRY_TOKEN}"
```
- Patterns use Go's [`path.Match` ](https://pkg.go.dev/path#Match ) syntax and are matched case-insensitively. `*` does not cross `/` , so `example/*` matches `example/library` but not `example-fork/library` . The longest matching pattern wins.
- A Composer route URL is the repository base that serves `/p2/{vendor}/{package}.json` . A route also covers the package's dev versions in `/p2/{vendor}/{package}~dev.json` ; patterns are matched against the package name without `~dev` .
- An npm route URL is the registry base that serves `/{package}` . Tarball URLs in its packuments must be on the same host and below the same path.
- A routed package is not looked up anywhere else. If the private registry does not have it, the client gets a 404. If the registry fails or rejects the credentials, the client gets a 502. A package with the same name on the public registry is never served in its place (dependency confusion).
- Metadata and artifacts of routed packages are cached under keys that include the route's URL. Entries cached from the public registry before a route was added are never served for a routed package.
Serve cached Composer releases without a metadata lookup
Every Composer archive download resolved its dist from the package
metadata before checking the artifact cache. With cache_metadata off,
the default, that meant an uncoalesced upstream GET of the p2 document
for every download of an archive the proxy already had: three downloads
of a cached archive made three metadata requests. With the cache on,
each download after the TTL revalidated and rescanned the document.
The lookup is only needed for the publish time cooldown checks
downloads against, since lock files name archive URLs directly. Like
the npm handler, record that time in the versions table when the dist
is first resolved, and on later downloads of a tagged release read it
from there and serve the cached archive straight away. Without a stored
time, as for archives cached by an older proxy, the download takes the
metadata path, which then records it. The time is stored even with
cooldown off, so enabling cooldown later keeps the fast path.
The cache still enforces the denylist and verifies the stored digest,
and an archive is only ever cached under a name that was checked
against the metadata, so those guarantees hold. Branches, whose
archive and time change with every push, routed packages, whose
versions row would not say which repository the time came from, and
requests with a query string such as ?sha= keep resolving from the
metadata.
2026-10-23 13:22:56 +02:00
- With cooldown enabled, the publish time of a routed npm or Composer version always comes from the route's metadata. Times recorded for the public package of the same name, or for a route's previous URL, are not used, and routed downloads do not record times of their own.
Add package routes for private Composer and npm registries (#426)
* Add Composer package routes for private repositories
- Add upstream.composer_routes mapping package name patterns to repositories
- Serve matching packages only from their route, with no fallback to the default repository
- Qualify metadata and artifact cache keys with the route's upstream
- Give routed versions an empty notification-url so installs are not reported to notify-batch
- Document routes in configuration docs, example config and README
* Add npm package routes for private registries
- Add upstream.npm_routes mapping package name patterns to registries
- Serve matching packages only from their route, with no fallback to upstream.npm
- Resolve and validate tarball URLs against the route's registry
- Qualify metadata and artifact cache keys with the route's upstream
- Document npm routes and the audit endpoint limitation
* Share routed artifact filename qualification between handlers
* Match Composer routes without the ~dev suffix
Packagist serves a package's dev versions from {name}~dev.json, and the
handler matched routes against the name with ~dev still attached. An exact
route for example/library therefore missed /p2/example/library~dev.json, and
the private package's dev metadata came from the default repository.
- Match routes against the package name without ~dev, keeping it in the
upstream URL and the cache key so the two files stay apart
- Add a regression test with an exact route
* Keep routed npm publish times out of the shared versions row
The download cooldown check stores publish times under the plain version
PURL, which does not say which registry a version came from. A time read
from the default registry was trusted for a private version of the same
name, so a private release published an hour ago passed a seven-day
cooldown when the public one was a month old.
- Skip the stored row for routed packages, both reading and writing, and
take the publish time from the route's metadata, which is cached under
the route
- This also stops a private time from holding back the public version, and
a route's previous upstream from lending its times to the new one
- Cover adding a route after a public download, the reverse order, and
moving a route to another upstream
2026-10-11 19:06:18 +02:00
- Composer metadata of routed packages carries an empty `notification-url` , so Composer does not report their installs to the default upstream's `notify-batch` endpoint.
- Use `upstream.auth` for the registry's credentials. The proxy has no authentication of its own, so every client that can reach it can download routed packages.
- Search (`/composer/search.json` ) and the package list (`/composer/packages/list.json` ) still come from the default upstream only.
2026-10-27 16:16:11 +01:00
- Attestations of a routed npm version (`/npm/-/npm/v1/attestations/...` ) come from the route, like its tarballs.
Fix npm search routing and abbreviated packuments under cooldown
`npm search` requests /-/v1/search?text=... . The path has no "/-/"
after the leading slash is trimmed, so it fell through to the package
metadata handler: upstream got /-/v1%2Fsearch without the query
string, and the answer was cached under "-/v1/search", so every later
search was served the first one's results. Search and /-/ping (used
by `npm ping`, which lost its query string the same way) are now
relayed to upstream uncached with their query strings, like the
signing-keys endpoint.
Packuments share one metadata cache row whatever format they were
fetched in. Without cooldown the proxy asks for the abbreviated
format, which has no "time" map. After cooldown was turned on, a
still-fresh abbreviated packument kept being served from cache, and
used as the stale copy when upstream failed, so cooldown filtered
nothing and the tarball check let recent versions through. When full
metadata is required (cooldown or npm_full_metadata on), packument
lookups now pass a validator that rejects an abbreviated document
(top-level "modified", no "time"), so it counts as a cache miss, is
refetched in full and is never used as a stale fallback. Behaviour
without cooldown is unchanged.
fetchMetadataDocument takes the validate callback that
fetchOrCacheMetadata already has; other callers pass nil.
2026-10-23 13:22:49 +02:00
- npm audit, signing-key, search and ping requests still go to `upstream.npm` . An audit request lists every package in the dependency tree, routed ones included.
Add package routes for private Composer and npm registries (#426)
* Add Composer package routes for private repositories
- Add upstream.composer_routes mapping package name patterns to repositories
- Serve matching packages only from their route, with no fallback to the default repository
- Qualify metadata and artifact cache keys with the route's upstream
- Give routed versions an empty notification-url so installs are not reported to notify-batch
- Document routes in configuration docs, example config and README
* Add npm package routes for private registries
- Add upstream.npm_routes mapping package name patterns to registries
- Serve matching packages only from their route, with no fallback to upstream.npm
- Resolve and validate tarball URLs against the route's registry
- Qualify metadata and artifact cache keys with the route's upstream
- Document npm routes and the audit endpoint limitation
* Share routed artifact filename qualification between handlers
* Match Composer routes without the ~dev suffix
Packagist serves a package's dev versions from {name}~dev.json, and the
handler matched routes against the name with ~dev still attached. An exact
route for example/library therefore missed /p2/example/library~dev.json, and
the private package's dev metadata came from the default repository.
- Match routes against the package name without ~dev, keeping it in the
upstream URL and the cache key so the two files stay apart
- Add a regression test with an exact route
* Keep routed npm publish times out of the shared versions row
The download cooldown check stores publish times under the plain version
PURL, which does not say which registry a version came from. A time read
from the default registry was trusted for a private version of the same
name, so a private release published an hour ago passed a seven-day
cooldown when the public one was a month old.
- Skip the stored row for routed packages, both reading and writing, and
take the publish time from the route's metadata, which is cached under
the route
- This also stops a private time from holding back the public version, and
a route's previous upstream from lending its times to the new one
- Cover adding a route after a public download, the reverse order, and
moving a route to another upstream
2026-10-11 19:06:18 +02:00
- Routes can only be set in the configuration file, not with environment variables.
2026-08-29 11:19:54 +01:00
Helm HTTP repositories and additional OCI registries are configured as named maps:
2026-08-16 22:42:55 +05:30
2026-08-29 11:19:54 +01:00
```yaml
upstream:
2026-08-16 22:42:55 +05:30
# Named HTTP Helm chart repositories, served at /helm/{name}/.
helm:
bitnami: "https://charts.bitnami.com/bitnami"
# Named OCI registries. Select one with the repository prefix
# upstream/{name}/, e.g. oci://proxy.example.com/upstream/ghcr/owner/chart.
oci:
ghcr: "https://ghcr.io"
2026-01-29 16:33:09 +00:00
```
2026-08-16 22:42:55 +05:30
Helm HTTP repositories are read-only. The proxy fetches and rewrites each
repository's `index.yaml` so chart archives are downloaded through the proxy.
Chart archives are retained only when their SHA-256 digest matches the digest
listed in the index. Relative and absolute chart URLs are both supported.
2026-09-19 14:39:29 +05:30
HTTP indexes can also contain `oci://` chart references. References matching the
host and port of a configured `upstream.oci` registry or `upstream.oci_default`
are rewritten through the OCI proxy, with named registries preferred. Unmatched
references are left unchanged, so Helm contacts those registries directly.
Automatic rewriting requires both the OCI upstream URL and `base_url` to be
root URLs without a path prefix. Repository paths, tags, and digests are preserved.
Add generic HTTP download proxy for GitHub release assets (mise/aqua) (#302)
* Add generic HTTP download proxy for GitHub release assets
Adds a /generic/{name}/ route backed by a new upstream.generic named-upstream
map, so tools that download from fixed URL shapes (mise's aqua backend
fetching GitHub release assets, and its tag lookups on api.github.com) can be
pointed at the proxy with client-side URL rewriting. Only configured
upstreams are reachable, so this is not an open HTTP proxy.
Paths shaped like {owner}/{repo}/releases/download/{tag}/{asset} are
version-pinned and go through the artifact cache: fetched once, hashed,
served without revalidation, and still served when the upstream is down.
Every other path goes through the metadata cache with the client's Accept
header and query string replayed, so API responses are fresh within
metadata_ttl, revalidated after that, and served stale when the upstream
fails or rate-limits the request.
Tests cover path classification, unknown upstreams and traversal, cache
hits with the upstream down, HEAD, 404 pass-through, Accept and query
forwarding, stale-on-429, cache isolation between upstreams, and that an
upstream token scoped to the release host is not sent to the object store
it redirects to.
Closes #183.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDjDeq27CKzEP2o3GWBY7F
* Use fixed Accept header for generic metadata
---------
Co-authored-by: Giles Westwood <giles@gileswestwood.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-03 16:44:54 +01:00
Generic HTTP upstreams proxy plain downloads from fixed base URLs:
```yaml
upstream:
# Named HTTP upstreams, served at /generic/{name}/. The rest of the
# request path and the query string are appended to the upstream URL.
generic:
github: "https://github.com"
github-api: "https://api.github.com"
Require generic_paths for generic upstreams with credentials
A generic upstream forwards any path below /generic/{name}/. The
documented mise setup puts a GITHUB_TOKEN on api.github.com to raise
the rate limit, so anyone who could reach the proxy could make any GET
request with that token, including for private repositories it can
read, and the answers went into the shared metadata cache. upstream.git
already refuses an upstream with credentials unless it lists its repos.
upstream.generic_paths now lists, per generic upstream, the paths it
may serve, with the same pattern syntax as git repos. Other paths
answer 404 without reaching the upstream, and so does a path holding
'%', '?', '#' or a backslash, which the upstream could read as a
different path than the one that matched. Config validation refuses a
generic upstream that upstream.auth has credentials for unless it has
generic_paths, which also covers a github.com token reaching the
generic github upstream. Upstreams without credentials keep serving
every path.
The docs and example config show the paths mise needs and recommend a
fine-grained token limited to public repositories.
2026-10-24 08:02:22 +02:00
# Paths each upstream may serve. Required for an upstream that
# upstream.auth has credentials for.
generic_paths:
github-api:
- "repos/*/*/releases"
- "repos/*/*/releases/**"
- "repos/*/*/tags"
Add generic HTTP download proxy for GitHub release assets (mise/aqua) (#302)
* Add generic HTTP download proxy for GitHub release assets
Adds a /generic/{name}/ route backed by a new upstream.generic named-upstream
map, so tools that download from fixed URL shapes (mise's aqua backend
fetching GitHub release assets, and its tag lookups on api.github.com) can be
pointed at the proxy with client-side URL rewriting. Only configured
upstreams are reachable, so this is not an open HTTP proxy.
Paths shaped like {owner}/{repo}/releases/download/{tag}/{asset} are
version-pinned and go through the artifact cache: fetched once, hashed,
served without revalidation, and still served when the upstream is down.
Every other path goes through the metadata cache with the client's Accept
header and query string replayed, so API responses are fresh within
metadata_ttl, revalidated after that, and served stale when the upstream
fails or rate-limits the request.
Tests cover path classification, unknown upstreams and traversal, cache
hits with the upstream down, HEAD, 404 pass-through, Accept and query
forwarding, stale-on-429, cache isolation between upstreams, and that an
upstream token scoped to the release host is not sent to the object store
it redirects to.
Closes #183.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDjDeq27CKzEP2o3GWBY7F
* Use fixed Accept header for generic metadata
---------
Co-authored-by: Giles Westwood <giles@gileswestwood.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-03 16:44:54 +01:00
auth:
# Optional: raise the GitHub API rate limit. Scoped to this host only,
# so the token is never sent to the object store GitHub redirects to.
"https://api.github.com":
type: bearer
token: "${GITHUB_TOKEN}"
```
Only configured upstreams are reachable, so this is not an open HTTP proxy.
Require generic_paths for generic upstreams with credentials
A generic upstream forwards any path below /generic/{name}/. The
documented mise setup puts a GITHUB_TOKEN on api.github.com to raise
the rate limit, so anyone who could reach the proxy could make any GET
request with that token, including for private repositories it can
read, and the answers went into the shared metadata cache. upstream.git
already refuses an upstream with credentials unless it lists its repos.
upstream.generic_paths now lists, per generic upstream, the paths it
may serve, with the same pattern syntax as git repos. Other paths
answer 404 without reaching the upstream, and so does a path holding
'%', '?', '#' or a backslash, which the upstream could read as a
different path than the one that matched. Config validation refuses a
generic upstream that upstream.auth has credentials for unless it has
generic_paths, which also covers a github.com token reaching the
generic github upstream. Upstreams without credentials keep serving
every path.
The docs and example config show the paths mise needs and recommend a
fine-grained token limited to public repositories.
2026-10-24 08:02:22 +02:00
`generic_paths` limits an upstream to the paths after `/generic/{name}/` that
its patterns match; any other path answers 404. The patterns use the syntax of
`upstream.git` `repos` : `path.Match` , where `*` does not cross a `/` , and a
pattern ending in `/**` also matches every path nested below its prefix. A
path containing `%` , `?` , `#` or a backslash is refused for an upstream with
`generic_paths` . An upstream without `generic_paths` serves every path.
When `upstream.auth` has credentials for a generic upstream, `generic_paths`
is required and the proxy refuses to start without it: everyone who can reach
the proxy can send requests with those credentials, so list only the paths
they need. For GitHub, use a fine-grained personal access token limited to
public repositories with read-only access; it raises the rate limit like any
other token but cannot read anything private, even through an allowed path.
Add generic HTTP download proxy for GitHub release assets (mise/aqua) (#302)
* Add generic HTTP download proxy for GitHub release assets
Adds a /generic/{name}/ route backed by a new upstream.generic named-upstream
map, so tools that download from fixed URL shapes (mise's aqua backend
fetching GitHub release assets, and its tag lookups on api.github.com) can be
pointed at the proxy with client-side URL rewriting. Only configured
upstreams are reachable, so this is not an open HTTP proxy.
Paths shaped like {owner}/{repo}/releases/download/{tag}/{asset} are
version-pinned and go through the artifact cache: fetched once, hashed,
served without revalidation, and still served when the upstream is down.
Every other path goes through the metadata cache with the client's Accept
header and query string replayed, so API responses are fresh within
metadata_ttl, revalidated after that, and served stale when the upstream
fails or rate-limits the request.
Tests cover path classification, unknown upstreams and traversal, cache
hits with the upstream down, HEAD, 404 pass-through, Accept and query
forwarding, stale-on-429, cache isolation between upstreams, and that an
upstream token scoped to the release host is not sent to the object store
it redirects to.
Closes #183.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDjDeq27CKzEP2o3GWBY7F
* Use fixed Accept header for generic metadata
---------
Co-authored-by: Giles Westwood <giles@gileswestwood.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-03 16:44:54 +01:00
Paths shaped like `{owner}/{repo}/releases/download/{tag}/{asset}` are
version-pinned GitHub release assets: they are stored in the artifact cache
and served from it without revalidation, including while the upstream is
2026-10-12 09:12:16 +01:00
down.
2026-10-12 15:39:23 +01:00
Upstreams pointed at these hosts get version-pinned paths built in, whatever
the upstream is named:
| Upstream URL | Cached as artifacts | Package name |
|---|---|---|
| `https://archive.apache.org` | `dist/{project dirs}/{version}/[binaries/]{file}` , where `{version}` starts with a digit | the project dirs, e.g. `maven/maven-3` , `maven/mvnd` |
| `https://nodejs.org` | `dist/v{version}/{file}` | the upstream name |
| `https://dl.google.com/go` | `go{version}.{os}-{arch}.tar.gz` , `.zip` and their `.sha256` | the upstream name |
2026-10-12 09:12:16 +01:00
Other upstreams can declare their own version-pinned paths in
2026-10-12 19:22:42 +01:00
`generic_artifacts` . These are tried before any built-in ones, so a configured
pattern that overlaps a built-in decides the package name. Each pattern is a regular expression that must match the
2026-10-12 09:12:16 +01:00
whole path after `/generic/{name}/` , with `version` and `file` named groups
(single path segments) and an optional `name` group for the package name,
which otherwise is the upstream name. Matching downloads are cached like
release assets and listed under the `generic` ecosystem:
```yaml
upstream:
generic:
2026-10-12 15:39:23 +01:00
eclipse: "https://download.eclipse.org"
2026-10-12 09:12:16 +01:00
generic_artifacts:
2026-10-12 15:39:23 +01:00
eclipse:
- 'jdtls/milestones/(?P<version>[^/]+)/(?P<file>[^/]+)'
2026-10-12 09:12:16 +01:00
```
Only list paths whose content never changes for a given version: a match is
never revalidated.
Every other path is served through the metadata cache (`cache_metadata`
Add generic HTTP download proxy for GitHub release assets (mise/aqua) (#302)
* Add generic HTTP download proxy for GitHub release assets
Adds a /generic/{name}/ route backed by a new upstream.generic named-upstream
map, so tools that download from fixed URL shapes (mise's aqua backend
fetching GitHub release assets, and its tag lookups on api.github.com) can be
pointed at the proxy with client-side URL rewriting. Only configured
upstreams are reachable, so this is not an open HTTP proxy.
Paths shaped like {owner}/{repo}/releases/download/{tag}/{asset} are
version-pinned and go through the artifact cache: fetched once, hashed,
served without revalidation, and still served when the upstream is down.
Every other path goes through the metadata cache with the client's Accept
header and query string replayed, so API responses are fresh within
metadata_ttl, revalidated after that, and served stale when the upstream
fails or rate-limits the request.
Tests cover path classification, unknown upstreams and traversal, cache
hits with the upstream down, HEAD, 404 pass-through, Accept and query
forwarding, stale-on-429, cache isolation between upstreams, and that an
upstream token scoped to the release host is not sent to the object store
it redirects to.
Closes #183.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDjDeq27CKzEP2o3GWBY7F
* Use fixed Accept header for generic metadata
---------
Co-authored-by: Giles Westwood <giles@gileswestwood.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-03 16:44:54 +01:00
must be enabled for offline fallback): fresh within `metadata_ttl` , then
revalidated with the upstream's `ETag` /`Last-Modified` , and served stale with
a `Warning: 110` header when the upstream fails, refuses or rate-limits the
request. Metadata responses are buffered up to `metadata_max_size` , so keep
large mutable downloads (`releases/latest/download/...` ) off this route.
This is the cache behind [mise ](https://mise.jdx.dev )'s aqua backend; see the
mise section in the README for the client-side `url_replacements` .
2026-08-29 11:19:54 +01:00
`upstream.oci_default` sets the registry used by unprefixed `/v2` requests,
while `upstream.oci` selects named registries through the `upstream/{name}/`
repository prefix. For example, `oci://proxy.example.com/upstream/ghcr/owner/chart`
uses the `ghcr` registry with `owner/chart` as its repository.
Route containerd ns mirror requests to configured registries (#405)
* Route containerd ns mirror requests to configured registries
containerd's hosts.toml mirrors append ?ns=<registry-host> to every
request. The container handler ignored it, so a single _default mirror
entry could not serve more than one registry.
ns is a closed-world lookup key and is never dialed. Docker Hub aliases
and the host of upstream.oci_default select the default route, hosts of
upstream.oci entries select their named upstream. Unknown hosts return
NAME_UNKNOWN so containerd falls back to its next host. Registry URLs
with a path are not indexed, because ns names the registry at the root
of a host. Host collisions log a warning and resolve deterministically.
With ns the path is the verbatim upstream repository, the reserved
upstream/ prefix is rejected, and repository prefix routes such as
Homebrew's do not apply. Cache names match the unprefixed and
upstream/{name}/ routes, so all routes share cached blobs, manifests and
tag lists. ns is no longer forwarded on tag-list requests, and the
pagination Link is rewritten per request instead of at store time so
clients on different routes get links for their own route.
Refs #303
* Document containerd ns mirroring
Refs #303
* Version the tag-list cache identity for the raw-link format
Tag-list rows now store the upstream Link verbatim and rewrite it per
request. Rows written by earlier versions hold a Link already rewritten
for one route. Under the unchanged key an older binary running next to
this one (rolling update on shared Postgres, or a rollback) would serve
the raw Link unrewritten, sending clients to the wrong route for the
next page. A format marker in the key keeps the two apart at the cost of
one cache miss per tag list after the upgrade.
Adversarial review finding F1: tag-list cache rows hold raw upstream
Links under unchanged keys, so an older binary serves them unrewritten.
* Qualify the README claim about ns mirror coverage
Registries whose URL has a path and hosts shared by two entries are not
reachable through ns; point to the configuration guide for both rules.
Adversarial review finding F3: README says ns covers every upstream.oci
registry.
* Redact registry URLs in ns index warnings
Configured registry URLs may carry userinfo credentials, which the new
startup warnings wrote to the log verbatim. Log them with the password
masked, and say precisely what a path-prefixed default registry means:
its host is not indexed, while the Docker Hub aliases still select it.
Adversarial review finding F2: startup warnings print registry URLs
unredacted and the default-registry message is misleading.
* Keep non-default ports apart in the ns index
Dropping 80 and 443 regardless of scheme turned https://host:80 and
https://host into the same key, so a request for one of them could land
on the other registry. Only the default port of the URL's scheme is
optional now. A configured host is indexed with and without that port,
because image references spell it either way; any other port has to
match exactly, and the ns value is used as containerd sends it, apart
from case and IPv6 brackets.
Since a URL now yields two keys that can belong to different routes,
the collision warning lists the owner per key.
The handler test runs two registries on one host that differ only in
the port. Tests can't bind 80 or 443, so a dialer maps those addresses
to the fake servers.
* Allow upstream/{name}/ together with ns when they agree
containerd sends ns on every request to a mirror host, override_path or
not. Per-registry mirrors pointing at /v2/upstream/{name} therefore
arrive as upstream/{name}/...?ns=<registry>, and refusing every prefixed
name under ns broke them, including the only way to mirror an upstream
whose URL has a path.
Such requests are now handled like the prefix route without ns, as long
as ns names that upstream's own host under the same port rules as the
index. Any other ns on a prefixed name is still NAME_UNKNOWN, so ns
can't be used to create cache entries under another registry's name.
* README: containerd config_path and how to check it
A _default/hosts.toml does nothing while CRI has no hosts directory
configured, so add the config.toml snippet for containerd 2.x and 1.x.
To check the setup, look at containerd config dump and pull with
crictl, then find the request in the proxy log; ctr --hosts-dir reads
the directory on its own and would succeed even without config_path.
Also mention that existing per-registry override_path entries keep
working and that k3s generates the directory itself.
* Let the prefix route take ns for aliases and mirrored registries
With upstream.oci.hub pointing at registry-1.docker.io and a docker.io
hosts.toml mirror on /v2/upstream/hub, containerd sends ns=docker.io,
and the prefix check only accepted the upstream's own host. The same
happens whenever the upstream is a mirror of the registry the nodes
pull from, say an Artifactory remote for ghcr.io: ns carries ghcr.io,
the upstream host is something else, and the pull got a 404.
The prefix already decides where the content comes from and which
cache entries are used, so ns can't redirect anything there. The check
now only refuses an ns that contradicts the prefix, meaning a host the
proxy knows that belongs to another route. The Docker Hub aliases count
as one host, and a host the proxy doesn't know at all is accepted,
since a client can only arrive at the prefix through its own mirror
entry.
* Take docker.io on every prefix route
A Docker Hub mirror configured as a named upstream (mirror.gcr.io, an
Artifactory remote) with a docker.io hosts.toml pointing at
/v2/upstream/hub got a 404 again: docker.io is always in the index for
the default route, so the check saw a known host of another route. That
setup worked before this branch. Docker Hub repository names have
exactly two path components, so docker.io/upstream/... can never be a
real image, and accepting the aliases on every prefix route shadows
nothing.
While here: the collision warning claimed ns requests go to the other
route, which is only true without the prefix, and the README said pulls
for unknown registries always get a 404, which isn't the case for paths
under upstream/{name}/.
* Only take a foreign ns on a prefix route when it is configured
A containerd _default mirror sends the same request for a pull of
unconfigured.example/upstream/mirror/owner/app as a ghcr.io hosts.toml
with override_path does for a mirror upstream. Accepting any host the
proxy didn't know therefore let such a pull be answered with owner/app
from the mirror upstream, a different image, instead of a 404 that lets
containerd go to the registry the image actually names.
The prefix route now accepts ns only for the upstream's own host, for
Docker Hub, and for hosts listed in the new upstream.oci_mirrors setting.
That setting is how an Artifactory remote of ghcr.io keeps working:
upstream:
oci_mirrors:
ghcr: ["ghcr.io"]
The listed hosts also select the upstream for unprefixed ns requests,
which makes a mirror with a path in its URL reachable through the
_default entry as well. Hosts from configured URLs still win over
mirror entries; the proxy warns about such overlaps at startup.
* Fix the docs sentence on which ns a prefix route takes
configuration.md still said a prefix request with ns is only refused
when ns names another route's host. Since the last commit it is the
other way round: only the upstream's own host, its oci_mirrors entries
and Docker Hub get through.
* Note what a host accepted on a prefix route gives up
Listing ghcr.io for an upstream (or the upstream's own host) still has
the ambiguity from the wildcard case: ghcr.io/upstream/ghcr/owner/app
lands on that upstream. That is the price of an explicit entry and
fine in practice, but it should be written down.
* Test that oci_mirrors entries stay with their own upstream
Two gaps in the coverage of the last change: nothing failed if a host
listed for one upstream was also accepted on another upstream's prefix,
and nothing checked that the server actually hands upstream.oci_mirrors
to the container handler. The handler test now has a second upstream
that must refuse ghcr.io, and a small server test pulls through a
mirror entry on both the prefix and the unprefixed route.
* Log when a prefix route refuses an ns
Setups with a ghcr.io hosts.toml pointing at a mirror upstream worked
before this branch and now need an oci_mirrors entry. Without one,
containerd gets a 404, quietly pulls from ghcr.io directly, and nobody
notices the proxy is being skipped. The refusal is now logged as a
warning that names the upstream and the ns and points at
upstream.oci_mirrors.
* Drop a check in validateOCIMirrors that could never fire
user@ghcr.io parses with ghcr.io as its host, so the parsed.Host != host
comparison already rejects it and the extra parsed.User test was dead.
A test case for a host with credentials keeps that path covered.
* Reject an oci_mirrors host that is listed for two upstreams
With ghcr.io under both art and nexus, both prefix routes took ns=ghcr.io
while unprefixed requests went to whichever name sorted first, with
nothing but a startup warning. Config validation now refuses the second
entry, comparing hosts without case and without the default :443.
* Log each refused ns once per upstream and refuse overlong values
The refusal warning fired on every request, and a node with a missing
oci_mirrors entry asks for the manifest and each layer, so the same line
piled up. Each upstream/ns pair is now logged once, for up to 1024
pairs; after that every refusal is logged again.
ns comes straight from the client and can be as long as the request
line allows, so remembering it as is would let anyone pin a lot of
memory with a handful of requests. No registry host is longer than 259
characters (a 253-character name plus a port), so anything longer is
refused right away, before it is logged or remembered.
* Compare oci_mirrors hosts the way the handler does
The duplicate check lowercased the host and trimmed :443, while the
handler splits host and port, brackets IPv6 literals and drops an empty
port. So ghcr.io and ghcr.io:, or [::1] and [::1]:, passed as different
hosts in two upstreams and then collided in the handler. The check now
builds the same key the handler uses.
* Rewrap the oci_mirrors paragraphs in the configuration docs
Two sentences had been dropped into the middle of wrapped lines, and the
prefix-route paragraph named Docker Hub twice in a row. Same content,
shorter lines, one mention.
* Stop logging ns refusals once the set is full
Past 1024 different upstream/ns pairs every refusal was logged again,
so anyone could fill the set with made-up hosts and then flood the log
at warning level. When the set is full the proxy now says so once and
keeps quiet after that. A real misconfiguration shows up long before
that, with the first pull.
* Test the limit on remembered ns refusals
Nothing covered what happens once 1024 pairs are remembered. The new
test fills the set, then sends a known pair and two new ones: the proxy
must log the limit notice exactly once, log none of the refusals and
keep the set at its size.
2026-10-10 08:35:50 +02:00
2026-10-23 17:34:15 +01:00
The OCI handler only serves pulls (`GET` and `HEAD` ). Pushes, such as a
BuildKit `--cache-to type=registry` export, need a separate registry.
Route containerd ns mirror requests to configured registries (#405)
* Route containerd ns mirror requests to configured registries
containerd's hosts.toml mirrors append ?ns=<registry-host> to every
request. The container handler ignored it, so a single _default mirror
entry could not serve more than one registry.
ns is a closed-world lookup key and is never dialed. Docker Hub aliases
and the host of upstream.oci_default select the default route, hosts of
upstream.oci entries select their named upstream. Unknown hosts return
NAME_UNKNOWN so containerd falls back to its next host. Registry URLs
with a path are not indexed, because ns names the registry at the root
of a host. Host collisions log a warning and resolve deterministically.
With ns the path is the verbatim upstream repository, the reserved
upstream/ prefix is rejected, and repository prefix routes such as
Homebrew's do not apply. Cache names match the unprefixed and
upstream/{name}/ routes, so all routes share cached blobs, manifests and
tag lists. ns is no longer forwarded on tag-list requests, and the
pagination Link is rewritten per request instead of at store time so
clients on different routes get links for their own route.
Refs #303
* Document containerd ns mirroring
Refs #303
* Version the tag-list cache identity for the raw-link format
Tag-list rows now store the upstream Link verbatim and rewrite it per
request. Rows written by earlier versions hold a Link already rewritten
for one route. Under the unchanged key an older binary running next to
this one (rolling update on shared Postgres, or a rollback) would serve
the raw Link unrewritten, sending clients to the wrong route for the
next page. A format marker in the key keeps the two apart at the cost of
one cache miss per tag list after the upgrade.
Adversarial review finding F1: tag-list cache rows hold raw upstream
Links under unchanged keys, so an older binary serves them unrewritten.
* Qualify the README claim about ns mirror coverage
Registries whose URL has a path and hosts shared by two entries are not
reachable through ns; point to the configuration guide for both rules.
Adversarial review finding F3: README says ns covers every upstream.oci
registry.
* Redact registry URLs in ns index warnings
Configured registry URLs may carry userinfo credentials, which the new
startup warnings wrote to the log verbatim. Log them with the password
masked, and say precisely what a path-prefixed default registry means:
its host is not indexed, while the Docker Hub aliases still select it.
Adversarial review finding F2: startup warnings print registry URLs
unredacted and the default-registry message is misleading.
* Keep non-default ports apart in the ns index
Dropping 80 and 443 regardless of scheme turned https://host:80 and
https://host into the same key, so a request for one of them could land
on the other registry. Only the default port of the URL's scheme is
optional now. A configured host is indexed with and without that port,
because image references spell it either way; any other port has to
match exactly, and the ns value is used as containerd sends it, apart
from case and IPv6 brackets.
Since a URL now yields two keys that can belong to different routes,
the collision warning lists the owner per key.
The handler test runs two registries on one host that differ only in
the port. Tests can't bind 80 or 443, so a dialer maps those addresses
to the fake servers.
* Allow upstream/{name}/ together with ns when they agree
containerd sends ns on every request to a mirror host, override_path or
not. Per-registry mirrors pointing at /v2/upstream/{name} therefore
arrive as upstream/{name}/...?ns=<registry>, and refusing every prefixed
name under ns broke them, including the only way to mirror an upstream
whose URL has a path.
Such requests are now handled like the prefix route without ns, as long
as ns names that upstream's own host under the same port rules as the
index. Any other ns on a prefixed name is still NAME_UNKNOWN, so ns
can't be used to create cache entries under another registry's name.
* README: containerd config_path and how to check it
A _default/hosts.toml does nothing while CRI has no hosts directory
configured, so add the config.toml snippet for containerd 2.x and 1.x.
To check the setup, look at containerd config dump and pull with
crictl, then find the request in the proxy log; ctr --hosts-dir reads
the directory on its own and would succeed even without config_path.
Also mention that existing per-registry override_path entries keep
working and that k3s generates the directory itself.
* Let the prefix route take ns for aliases and mirrored registries
With upstream.oci.hub pointing at registry-1.docker.io and a docker.io
hosts.toml mirror on /v2/upstream/hub, containerd sends ns=docker.io,
and the prefix check only accepted the upstream's own host. The same
happens whenever the upstream is a mirror of the registry the nodes
pull from, say an Artifactory remote for ghcr.io: ns carries ghcr.io,
the upstream host is something else, and the pull got a 404.
The prefix already decides where the content comes from and which
cache entries are used, so ns can't redirect anything there. The check
now only refuses an ns that contradicts the prefix, meaning a host the
proxy knows that belongs to another route. The Docker Hub aliases count
as one host, and a host the proxy doesn't know at all is accepted,
since a client can only arrive at the prefix through its own mirror
entry.
* Take docker.io on every prefix route
A Docker Hub mirror configured as a named upstream (mirror.gcr.io, an
Artifactory remote) with a docker.io hosts.toml pointing at
/v2/upstream/hub got a 404 again: docker.io is always in the index for
the default route, so the check saw a known host of another route. That
setup worked before this branch. Docker Hub repository names have
exactly two path components, so docker.io/upstream/... can never be a
real image, and accepting the aliases on every prefix route shadows
nothing.
While here: the collision warning claimed ns requests go to the other
route, which is only true without the prefix, and the README said pulls
for unknown registries always get a 404, which isn't the case for paths
under upstream/{name}/.
* Only take a foreign ns on a prefix route when it is configured
A containerd _default mirror sends the same request for a pull of
unconfigured.example/upstream/mirror/owner/app as a ghcr.io hosts.toml
with override_path does for a mirror upstream. Accepting any host the
proxy didn't know therefore let such a pull be answered with owner/app
from the mirror upstream, a different image, instead of a 404 that lets
containerd go to the registry the image actually names.
The prefix route now accepts ns only for the upstream's own host, for
Docker Hub, and for hosts listed in the new upstream.oci_mirrors setting.
That setting is how an Artifactory remote of ghcr.io keeps working:
upstream:
oci_mirrors:
ghcr: ["ghcr.io"]
The listed hosts also select the upstream for unprefixed ns requests,
which makes a mirror with a path in its URL reachable through the
_default entry as well. Hosts from configured URLs still win over
mirror entries; the proxy warns about such overlaps at startup.
* Fix the docs sentence on which ns a prefix route takes
configuration.md still said a prefix request with ns is only refused
when ns names another route's host. Since the last commit it is the
other way round: only the upstream's own host, its oci_mirrors entries
and Docker Hub get through.
* Note what a host accepted on a prefix route gives up
Listing ghcr.io for an upstream (or the upstream's own host) still has
the ambiguity from the wildcard case: ghcr.io/upstream/ghcr/owner/app
lands on that upstream. That is the price of an explicit entry and
fine in practice, but it should be written down.
* Test that oci_mirrors entries stay with their own upstream
Two gaps in the coverage of the last change: nothing failed if a host
listed for one upstream was also accepted on another upstream's prefix,
and nothing checked that the server actually hands upstream.oci_mirrors
to the container handler. The handler test now has a second upstream
that must refuse ghcr.io, and a small server test pulls through a
mirror entry on both the prefix and the unprefixed route.
* Log when a prefix route refuses an ns
Setups with a ghcr.io hosts.toml pointing at a mirror upstream worked
before this branch and now need an oci_mirrors entry. Without one,
containerd gets a 404, quietly pulls from ghcr.io directly, and nobody
notices the proxy is being skipped. The refusal is now logged as a
warning that names the upstream and the ns and points at
upstream.oci_mirrors.
* Drop a check in validateOCIMirrors that could never fire
user@ghcr.io parses with ghcr.io as its host, so the parsed.Host != host
comparison already rejects it and the extra parsed.User test was dead.
A test case for a host with credentials keeps that path covered.
* Reject an oci_mirrors host that is listed for two upstreams
With ghcr.io under both art and nexus, both prefix routes took ns=ghcr.io
while unprefixed requests went to whichever name sorted first, with
nothing but a startup warning. Config validation now refuses the second
entry, comparing hosts without case and without the default :443.
* Log each refused ns once per upstream and refuse overlong values
The refusal warning fired on every request, and a node with a missing
oci_mirrors entry asks for the manifest and each layer, so the same line
piled up. Each upstream/ns pair is now logged once, for up to 1024
pairs; after that every refusal is logged again.
ns comes straight from the client and can be as long as the request
line allows, so remembering it as is would let anyone pin a lot of
memory with a handful of requests. No registry host is longer than 259
characters (a 253-character name plus a port), so anything longer is
refused right away, before it is logged or remembered.
* Compare oci_mirrors hosts the way the handler does
The duplicate check lowercased the host and trimmed :443, while the
handler splits host and port, brackets IPv6 literals and drops an empty
port. So ghcr.io and ghcr.io:, or [::1] and [::1]:, passed as different
hosts in two upstreams and then collided in the handler. The check now
builds the same key the handler uses.
* Rewrap the oci_mirrors paragraphs in the configuration docs
Two sentences had been dropped into the middle of wrapped lines, and the
prefix-route paragraph named Docker Hub twice in a row. Same content,
shorter lines, one mention.
* Stop logging ns refusals once the set is full
Past 1024 different upstream/ns pairs every refusal was logged again,
so anyone could fill the set with made-up hosts and then flood the log
at warning level. When the set is full the proxy now says so once and
keeps quiet after that. A real misconfiguration shows up long before
that, with the first pull.
* Test the limit on remembered ns refusals
Nothing covered what happens once 1024 pairs are remembered. The new
test fills the set, then sends a known pair and two new ones: the proxy
must log the limit notice exactly once, log none of the refusals and
keep the set at its size.
2026-10-10 08:35:50 +02:00
containerd mirror requests carry the original registry host in an `ns` query
parameter (see the containerd section in the README). The proxy only looks the
host up and never connects to it: `docker.io` , `index.docker.io` ,
`registry-1.docker.io` and the host of `upstream.oci_default` select the default
registry, and the host of each `upstream.oci` URL selects that named registry.
Hosts are compared case-insensitively. The scheme's default port (443 for
`https` , 80 for `http` ) may be spelled out or left out in the image reference;
any other port must match exactly, so `https://registry.example:80` and
`https://registry.example` are two different registries. An
unknown host returns `404 NAME_UNKNOWN` , so containerd falls back to its next
host. Registry URLs with a path (for example an Artifactory repository path)
are not reachable through their own host in `ns` , only through
`upstream/{name}/` or through `upstream.oci_mirrors` . When two
entries share a host, the proxy logs a warning at startup; the default registry
wins, otherwise the alphabetically first name. Pulls through `ns` ,
`upstream/{name}/` and unprefixed requests share the same cache entries.
Requests that combine the `upstream/{name}/` prefix with `ns` , as per-registry
containerd mirrors with `override_path = true` send them, are routed by the
prefix. They are accepted only when `ns` is the upstream's own host, a host
listed for it in `upstream.oci_mirrors` , or Docker Hub (`docker.io` and its
aliases). Docker Hub repository names have two path components and can never
start with `upstream/` , so a Docker Hub mirror behind any prefix stays
reachable. Any other host gets `404 NAME_UNKNOWN` : a containerd `_default`
mirror sends the same request for an image such as
`unconfigured.example/upstream/ghcr/owner/app` , and that pull must not be
answered from the `ghcr` upstream.
When a named upstream mirrors another registry, for example an Artifactory
remote of `ghcr.io` , say so in `upstream.oci_mirrors` :
```yaml
upstream:
oci:
ghcr: "https://artifactory.example.com/artifactory/api/docker/ghcr-remote"
oci_mirrors:
ghcr: ["ghcr.io"]
```
Each entry lists bare registry hosts as containerd sends them, optionally with
a port. A host can be listed for one upstream only. Those hosts select the
upstream for unprefixed `ns` requests and are accepted as `ns` on its
`upstream/{name}/` prefix. A host that is also the host of a configured
registry URL stays with that registry for unprefixed requests; the proxy logs a
warning at startup.
A host that is accepted on a prefix route gives up image paths of the form
`<host>/upstream/{name}/...` under a `_default` mirror: with `ghcr: ["ghcr.io"]` ,
a pull of `ghcr.io/upstream/ghcr/owner/app` is answered with `owner/app` from
the `ghcr` upstream, because containerd sends the same request for it as a
per-registry mirror does. The same applies to the upstream's own host.
2026-08-16 22:42:55 +05:30
When the proxy uses plain HTTP (for example `localhost:8080` ), pass
`--plain-http` to Helm OCI commands.
Add Alpine APK repository proxy support (#293)
* Add Alpine APK repository proxy support
- Serve named APK repositories at /apk/{repository}/ with the official
Alpine mirror as the default repository
- Cache v2 APKINDEX.tar.gz and v3 Packages.adb indexes and detached
signatures via the metadata cache, serving stored bytes unchanged so
apk signature verification keeps working
- Cache .apk packages in the shared artifact cache keyed by the full
repository path, since APK filenames do not include the architecture
- Add configurable upstream repositories via upstream.apk with
validation, plus dashboard registry instructions
- Add tests for index/signature byte fidelity, per-arch caching, cache
hits, offline reads, upstream authentication, and 404 handling
- Document apk usage in README, config example, and configuration docs
* Serve APK package HEAD requests without a body
Use the method-aware serveArtifact helper (as container.go does) so HEAD
responses carry Content-Length but omit the body; add a regression test.
* Drop doubled blank line in docs/configuration.md
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
2026-09-02 13:36:00 +02:00
```yaml
upstream:
# Named Alpine APK repositories, served at /apk/{name}/.
apk:
alpine: "https://dl-cdn.alpinelinux.org/alpine"
```
Alpine APK repositories are read-only. Requests to `/apk/{name}/…` mirror the
upstream layout, e.g. `/apk/alpine/v3.22/main/x86_64/APKINDEX.tar.gz` . Indexes
(v2 `APKINDEX.tar.gz` , v3 `Packages.adb` ) and detached signatures are cached
with the metadata TTL and served byte-for-byte unchanged so apk signature
verification keeps working; `.apk` packages use the shared artifact cache.
When `upstream.apk` is empty, a single repository named `alpine` pointing at
the official mirror is available; configuring any entry replaces that default.
2026-10-11 23:41:16 +02:00
```yaml
upstream:
# Named Arch Linux pacman mirrors, served at /arch/{name}/.
arch:
archlinux: "https://geo.mirror.pkgbuild.com"
```
Arch Linux repositories are read-only. Requests to `/arch/{name}/…` are passed
to the mirror unchanged, so both the Arch Linux layout
(`/arch/archlinux/core/os/x86_64/core.db` ) and the Arch Linux ARM layout
(`/arch/archlinuxarm/aarch64/core/core.db` ) work. Databases (`{repo}.db` ,
`{repo}.files` and their `.tar.*` forms) are cached with the metadata TTL and
served byte-for-byte unchanged; a database's `.sig` is refreshed together with
the database. Packages and package signatures use the shared artifact cache.
When `upstream.arch` is empty, a single repository named `archlinux` pointing at
`https://geo.mirror.pkgbuild.com` is available; configuring any entry replaces
that default. Arch Linux ARM has no default: pick a regional HTTPS mirror, not
the `mirror.archlinuxarm.org` redirector.
2026-10-22 12:13:20 +02:00
```yaml
upstream:
# Named FreeBSD pkg mirrors, served at /freebsd/{name}/.
freebsd:
freebsd: "https://pkg.freebsd.org"
```
FreeBSD pkg repositories are read-only. Requests to `/freebsd/{name}/…` are
passed to the mirror unchanged, e.g.
`/freebsd/freebsd/FreeBSD:14:amd64/quarterly/packagesite.pkg` , so ports and
pkgbase repositories and repositories built with poudriere work. Catalogues
(`meta` , `meta.conf` , `packagesite` and `data` as `.pkg` , `.txz` or `.tzst` )
are cached with the metadata TTL and served byte-for-byte unchanged, which
keeps their embedded signatures valid. Packages under `Hashed/` use the shared
2026-10-23 09:50:22 +02:00
artifact cache. Other package files (`.pkg` , `.txz` , `.tzst` ), unhashed ones
and the bootstrap package under `Latest/` , can be rebuilt under the same name,
so they are passed through uncached after the [denylist ](#version-denylist )
check on their `{name}-{version}` . Only cached artifacts are scanned, so while
a scanner covers the `freebsd` ecosystem these files are refused with
`403 Forbidden` : serve repositories built with `pkg repo --hash` (as the
official mirror is) and bootstrap pkg without the proxy. Everything else, such
as signatures and keys, is passed through uncached. When `upstream.freebsd` is empty, a single
2026-10-22 12:13:20 +02:00
repository named `freebsd` pointing at `https://pkg.freebsd.org` is available;
configuring any entry replaces that default.
2026-10-10 09:57:43 +02:00
### Go checksum database
```yaml
upstream:
# Named Go checksum databases, served at /go/sumdb/{name}/.
go_sumdbs:
sum.golang.org: "https://sum.golang.org"
Add upstream.go_sumdbs_offline for networks where only the proxy has egress
On a build network where clients cannot reach sum.golang.org at all, the
proxy should keep answering for the database during an outage, since a 404
on supported would only send the go command somewhere it cannot go. With
the switch set, a cached tree of any age is enough; without it, the tree
has to be fresh, as before.
That does not make every cached module verify. The go command checks a
lookup against tiles of the tree that came with it, and sum.golang.org
attaches a different tree to almost every lookup, so it can ask for a tile
nobody fetched through the proxy before. configuration.md says so.
While the database is known to be unreachable, offline mode also serves
cached lookups and tiles straight away. Otherwise each of them would sit
through the upstream timeout before the fetch falls back to the cache,
and a build would take minutes for as long as the outage lasts. A lookup
or tile that falls back to the cache renews what the proxy knows about
the database, unless a probe has reached it since the request started.
The go command asks supported only once per process, so a long build would
otherwise be back to waiting a minute into the outage. recordTimeout is
now recordFailure, since lookups use it too.
The option needs cache_metadata, since without it there is no cached tree
to fall back on, and the config refuses the combination.
2026-10-10 17:55:04 +02:00
# Keep answering for a database from its cached tree, however old, when
# the database cannot be reached. For networks where clients have no
# route to the database of their own. Requires cache_metadata.
go_sumdbs_offline: false
2026-10-10 09:57:43 +02:00
```
The `go` command asks `GOPROXY` for the checksum database before contacting it
directly. The proxy answers for every name listed here and fetches lookups and
tiles from the database's URL, not from `upstream.go` . Unknown names get 404,
so the `go` command falls back to contacting that database itself. When
`upstream.go_sumdbs` is empty, `sum.golang.org` is served; configuring any
entry replaces that default.
2026-10-10 10:18:40 +02:00
Once the proxy says it serves a database, the `go` command no longer contacts
that database directly. The proxy therefore only says so when it can fetch the
Don't let the supported check hang, and don't take a block page for an answer
Two problems with the check behind supported. The fetch it waits on is
shared with other callers and only bounded by the upstream timeout, which
defaults to 30 seconds. A firewall that silently drops packets to
sum.golang.org would stall the first go command every minute for that long,
and the go command itself has no timeout on this request. Now we wait five
seconds at most; the fetch carries on in the background, records how it
went and fills the cache for the next check. A timeout only counts as a
failure if the fetch hasn't succeeded in the meantime.
The fetch reports no error when it falls back to the cached tree after the
database failed, so the metadata result now says when it did, and the
check treats that as the database not answering. A fresh cached tree
(within metadata_ttl) still lets a timed-out check say yes, since the
fetch only falls back once upstream has failed, which a silent database
delays past our timeout. An older tree does not count: once the proxy has
lost the database, clients go there directly, as they would without the
proxy.
And the check believed any 200. Egress proxies tend to answer blocked hosts
with a 200 HTML page, which we would have cached as the signed tree and taken
as proof that the database is reachable. latest now has to start with
"go.sum database tree", or it is refused like a bad tile.
Lookups had the same gap. A block page cached as a lookup would also
replace the good copy that is served during an outage, so a lookup now has
to be a record followed by a signed tree, the way the go command reads it.
2026-10-10 11:31:01 +02:00
database's latest signed tree, or has a fresh one cached (fetched within
`metadata_ttl` ). If it cannot, it answers 404 as for an unknown name, and
clients go to the database themselves, as they would without the proxy. The
check waits five seconds at most, and after a failed check the proxy waits a
minute before it tries the database again.
2026-10-10 10:18:40 +02:00
Add upstream.go_sumdbs_offline for networks where only the proxy has egress
On a build network where clients cannot reach sum.golang.org at all, the
proxy should keep answering for the database during an outage, since a 404
on supported would only send the go command somewhere it cannot go. With
the switch set, a cached tree of any age is enough; without it, the tree
has to be fresh, as before.
That does not make every cached module verify. The go command checks a
lookup against tiles of the tree that came with it, and sum.golang.org
attaches a different tree to almost every lookup, so it can ask for a tile
nobody fetched through the proxy before. configuration.md says so.
While the database is known to be unreachable, offline mode also serves
cached lookups and tiles straight away. Otherwise each of them would sit
through the upstream timeout before the fetch falls back to the cache,
and a build would take minutes for as long as the outage lasts. A lookup
or tile that falls back to the cache renews what the proxy knows about
the database, unless a probe has reached it since the request started.
The go command asks supported only once per process, so a long build would
otherwise be back to waiting a minute into the outage. recordTimeout is
now recordFailure, since lookups use it too.
The option needs cache_metadata, since without it there is no cached tree
to fall back on, and the config refuses the combination.
2026-10-10 17:55:04 +02:00
Set `upstream.go_sumdbs_offline: true` when clients have no route to the
database of their own. The proxy then keeps answering for it from the cached
tree for as long as the outage lasts and serves cached lookups and tiles.
That does not mean every cached module still verifies: the `go` command
checks each lookup against tiles of the signed tree that came with it, the
database attaches a different tree to almost every lookup, and a tile no
client fetched through the proxy before cannot be served during the outage.
Lookups for modules that were never cached fail as well. Builds whose
`go.sum` already lists every module do not ask the database at all and are
not affected. This requires `cache_metadata` .
2026-10-13 16:23:12 +02:00
With `cache_metadata` enabled, lookups, partial hash tiles and the latest
signed tree follow `metadata_ttl` and are served from cache when the database
is unreachable. Complete hash tiles never change and are not revalidated once
cached; a tile of the wrong size is refused and not stored. Data tiles are
always forwarded and never stored: the `go` command does not fetch them, only
mirrors and auditors do.
2026-10-10 10:19:46 +02:00
Without `cache_metadata` the requests are forwarded and nothing is stored.
A 404 from the database keeps its text, which the `go` command prints to
explain why a version is missing. With `cache_metadata` enabled the text is
cut at 4 KiB; without it the response is passed on unchanged.
2026-10-10 09:57:43 +02:00
If `upstream.go` points at a module proxy that serves the checksum database
itself, such as Athens, point the entry there instead, for example
`sum.golang.org: "https://athens.example/sumdb/sum.golang.org"` .
Serve the Go vulnerability database at /go/vulndb
govulncheck reads vuln.go.dev directly, so on networks that only reach
the proxy it cannot run. upstream.go_vulndb (default https://vuln.go.dev)
is now served at /go/vulndb for `govulncheck -db`.
Only the files the database API defines are passed on (index/db,
index/modules, index/vulns and ID/{id}, as .json and .json.gz), with IDs
limited to letters, digits, '-' and '_'. They are stored and served
byte-for-byte; .json.gz files are gzip data without Content-Encoding.
All files, entries included, follow the metadata TTL since entries change
when their modified date does, and fall back to the cached copy when the
database is unreachable. Non-JSON responses are refused so a block page
does not replace the copy kept for an outage. HEAD is allowed under
/vulndb/, because govulncheck checks a non-default database with HEAD on
index/modules.json.gz.
2026-10-27 16:16:04 +01:00
### Go vulnerability database
```yaml
upstream:
# The Go vulnerability database, served at /go/vulndb.
go_vulndb: "https://vuln.go.dev"
```
Point govulncheck at the proxy with its `-db` flag:
```bash
govulncheck -db https://proxy.example.com/go/vulndb ./...
```
The proxy passes on the files the
[database API ](https://go.dev/security/vuln/database#api ) defines:
`index/db.json` , `index/modules.json` , `index/vulns.json` and `ID/{id}.json` ,
each also as `.json.gz` , which is what govulncheck fetches. Other paths get
404 without reaching the database. The files are stored and served
byte-for-byte: a `.json.gz` file is gzip data, not a response with
`Content-Encoding: gzip` .
With `cache_metadata` enabled, every file follows `metadata_ttl` , entries
included, since the database updates an entry when its `modified` date
changes. A response that is not JSON (after decompressing a `.json.gz` file)
is refused and not stored. When the database cannot be reached, cached files
are served however old, with a `Warning` header. Without `cache_metadata`
the requests are forwarded and nothing is stored.
On a network where clients cannot reach `vuln.go.dev` , only files someone
fetched through the proxy before an outage are available during it. To have
every entry cached, fetch the indexes and all entries through the proxy while
it is online, for example:
```bash
base=https://proxy.example.com/go/vulndb
for f in db modules vulns; do curl -sfo /dev/null "$base/index/$f.json.gz"; done
curl -sf "$base/index/vulns.json" | jq -r '.[].id' |
xargs -I{} curl -sfo /dev/null "$base/ID/{}.json.gz"
```
feat(debian): serve additional APT archives at /debian/{name}/
A Debian release is served by more than one archive. Security updates live on
a different host than the main archive, so a single upstream.debian URL cannot
serve a complete suite set and -security suites are unreachable.
Add upstream.debian_repositories, a name-to-URL map served at /debian/{name}/,
modelled on upstream.apk. The field is additive: upstream.debian keeps serving
/debian/pool/... and /debian/dists/... with unchanged cache identities, so
existing deployments and their warm caches are unaffected, and the scalar
field and PROXY_UPSTREAM_DEBIAN are untouched.
Named repositories scope both caches by name, since the same filename can hold
different bytes in different archives. Metadata keys are hashed over name,
upstream URL, and path, as APKHandler does. Names are validated through
validateNamedUpstreams, and "pool" and "dists" are refused because they would
shadow the main archive's own paths.
An unconfigured first path segment stays a main-archive path rather than
returning 404 as the APK handler does: the main archive is unnamed and serves
paths of its own at the root.
2026-09-14 15:41:13 -07:00
### Debian archives
```yaml
upstream:
# The main APT archive, served at /debian/.
debian: "http://deb.debian.org/debian"
# Additional APT archives, served at /debian/{name}/.
debian_repositories:
security: "https://security.debian.org/debian-security"
```
A Debian release is served by more than one archive: security updates live on
a separate host from the main archive, so `upstream.debian` alone cannot serve
a complete suite set. Requests to `/debian/{name}/…` mirror the upstream
layout, e.g. `/debian/security/dists/trixie-security/InRelease` .
`upstream.debian_repositories` is additive. `/debian/pool/…` and
`/debian/dists/…` continue to address `upstream.debian` with unchanged cache
identities, so existing deployments and their warm caches are unaffected. A
repository name shadows the main archive's root path of the same name; `pool`
and `dists` are refused at config load for that reason, and other root paths
an archive may serve (`indices` , `project` , `doc` , `tools` ) are not, so avoid
those names unless the shadowing is intended.
This field has no environment variable, as with the other named upstream maps.
`PROXY_UPSTREAM_DEBIAN` still sets `upstream.debian` .
Serve Git repositories from a local cache (#492)
* Serve Git repositories from a local cache
Named upstream.git hosts are served read-only over Git's smart HTTP
protocol at /git/{name}/{repo}, so git clone and git fetch work against
the proxy through an insteadOf rewrite. Each repository is a bare clone
at git.cache_dir/{name}-{hash of the upstream URL}/{repo}.git, served
by git upload-pack.
Every request for refs checks the upstream's branches, tags and HEAD
with a protocol v2 ls-refs limited to those prefixes, and fetches only
when they changed, atomically and with --prune. Concurrent requests
share one refresh. Only branches and tags are cached, and HEAD follows
the upstream's default branch.
A commit a client asks for by ID that the cache lacks, such as a pull
request head, is fetched and kept under a hidden ref until a branch or
tag includes it or for 7 days. One the upstream cannot provide is not
asked for again for a minute, or an hour when it is larger than
git.max_repo_size leaves. This is off for upstreams upstream.auth has
credentials for, since GitHub serves commits from any repository in a
fork network.
git never reaches an upstream itself. Its fetches go through a loopback
relay that forwards with the proxy's upstream client, so address
checks, upstream.auth and the access log apply, and git never sees a
credential. Redirects are not followed, and git runs with an
environment of its own.
The cached copy is served while the upstream fails or is slower than
git.refresh_wait, until git.stale_max_age has passed since the last
successful check. Refusals and missing repositories are never answered
from the cache.
git.max_repo_size cuts a transfer off as it streams, and every fetched
pack is kept as a pack, so a cut-off leaves nothing behind. A
repository that can no longer be refreshed within the limit is cloned
again with only its branches and tags. git.max_size evicts the least
recently used repositories, never one in use. At startup only the
directory of a repointed upstream's old URL is removed, and Close waits
for refreshes in progress.
Hosts with upstream.auth credentials need a repos allowlist, and a
repository outside it answers 404.
Refs #403
* Document setting the Git rewrite through environment variables
The Git section of the configuration docs now explains what insteadOf
does, with the URL a clone fetches from, and shows the same rewrites set
through GIT_CONFIG_COUNT, GIT_CONFIG_KEY_n and GIT_CONFIG_VALUE_n for
CI. Package managers that run git for Git dependencies inherit those
variables, so their fetches are cached too; npm's GitHub tarball
downloads do not go through git and are not.
Refs #403
* Note that upstream.auth credentials reach every route to a host
Credentials are matched by URL, so a token configured for a Git upstream
is also sent by a generic upstream at the same URL, which forwards any
path and checks no repos allowlist. With a token for a private GitHub
org, /generic/github/acme/private.git/info/refs would list that
repository's refs to anyone who can reach the proxy.
* Cap concurrent upload-pack processes and request body size
Every POST git-upload-pack started a pack-objects with git's defaults,
a delta-search thread per CPU and unbounded window memory, and nothing
limited how many ran at once, so a burst of CI clones of one large
repository could run the proxy out of memory. At most two upload-pack
processes per CPU (GOMAXPROCS, so a container's limit) now build packs
at once, and further requests wait for a slot while the client is
connected. ls-refs builds no pack and does not wait. Serving runs with
pack.threads=1 and pack.windowMemory=64m, and every git process with
core.bigFileThreshold=8m, so index-pack streams large files instead of
holding them in memory.
A request body was streamed to upload-pack unbounded, so a small gzip
body could feed it an arbitrarily long negotiation for as long as the
request may run. Bodies are now limited to 100MB, both as received and
decoded, and one past the limit fails with an error instead of being
cut short quietly.
* Apply refresh_wait to the ref check only
refresh_wait cut off the whole refresh, fetch included, so after a push
with a large pack a client got the old refs once it passed, though the
upstream was fine and the docs say a clone right after a push sees the
new commit. It now bounds only the check of the upstream's branches and
tags. Once the check finds they changed, requests wait for the fetch;
one that fails, or stalls and is abandoned by http.lowSpeedTime, still
falls back to the cached copy within stale_max_age.
* Lock cache_dir against a second process
The cache clears its temporary directory at startup, so a second
process given the same cache_dir, such as a second pod on a shared
volume, would delete the first one's clones in progress, and each
process's eviction knows only its own readers. New now locks
{cache_dir}/.gitcache-lock before touching anything and holds it until
Close: flock on Unix, a handle opened without sharing on Windows. The
OS releases it when the process exits, so a crashed process can
restart. A second process fails to start with an error naming the
directory.
* Evict again once repositories in use are released
Eviction skips repositories a request or refresh is using, and only the
release of one too large to refresh ran it again, so a cache filled past
max_size while its repositories were in use stayed over it once they
went idle, until the next fetch. Eviction now records when it is left
over max_size, and the next release runs it again.
* Cap git.max_repo_size at git.max_size
A repository larger than max_size could still be cloned, up to
max_repo_size, which defaults to 5GB whatever max_size is. Once released
it was evicted to bring the cache back within max_size, and the next
request cloned it again. max_size now caps max_repo_size, so such a
repository is refused for an hour like any other too large one, with an
error that names both limits.
* Count fetches in progress toward git.max_size
Fetches enforced only max_repo_size, and eviction ran after each one,
so up to 16 concurrent fetches could take the cache past max_size by as
much as each may download. Each fetch now reserves space within
max_size as the relay receives its bytes, a chunk at a time, evicting
least recently used idle repositories to make room. When the fetch
ends, the repository's measured size replaces the reservation under the
cache lock, so its bytes are neither counted twice nor missed.
When evicting every idle repository would not make room, none is
evicted for the fetch, and it is cut off with ErrCacheFull rather than
ErrTooLarge, so the repository is not refused for an hour: a clone
answers 507, and a refresh serves the cached copy as when the upstream
fails. A fetch that runs out of room while another has just been cut
off waits, up to a minute, for that one's space, so two fetches that do
not fit together do not both fail. A fetch that was cut off never
waits, so no two wait on each other.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
2026-10-21 03:47:16 -07:00
### Git repositories
Git servers configured under `upstream.git` are served read-only at
`/git/{name}/{repo}` over Git's smart HTTP protocol, from bare repositories
cached on local disk:
```yaml
upstream:
git:
github: "https://github.com"
wikimedia:
url: "https://gerrit.wikimedia.org/r"
repos: ["mediawiki/core", "mediawiki/extensions/**"] # optional allowlist
git:
cache_dir: /data/git # local disk, used by this process only
max_size: 50GB # evict least recently used repositories past this
max_repo_size: 5GB # refuse a repository that grows past this
refresh_wait: 5s # then serve the cached copy if the upstream check is slow
stale_max_age: 24h # never serve a copy not checked for this long
```
Clients reach the proxy through Git's `insteadOf` setting, which rewrites the
start of a URL before Git connects. After
```bash
git config --global url."http://localhost:8080/git/github/".insteadOf https://github.com/
```
`git clone https://github.com/git-pkgs/purl` fetches from
`http://localhost:8080/git/github/git-pkgs/purl` , while the clone's remote
stays `https://github.com/git-pkgs/purl` .
In CI, the same setting can come from environment variables instead of a
config file (Git 2.31 or newer), next to the variables that point package
managers at the proxy. Each `GIT_CONFIG_KEY_n` and `GIT_CONFIG_VALUE_n` pair is
one `git config` setting, and `GIT_CONFIG_COUNT` says how many pairs there are.
This sends both upstreams from the example above through the proxy:
```bash
export GIT_CONFIG_COUNT=2
export GIT_CONFIG_KEY_0=url.http://localhost:8080/git/github/.insteadOf
export GIT_CONFIG_VALUE_0=https://github.com/
export GIT_CONFIG_KEY_1=url.http://localhost:8080/git/wikimedia/.insteadOf
export GIT_CONFIG_VALUE_1=https://gerrit.wikimedia.org/r/
```
Every process the job starts inherits these variables, so the rewrite also
applies when a package manager runs `git` for a Git dependency, as pip,
Bundler, Composer, SwiftPM, and Go for modules it fetches directly do. Those
fetches are cached too. npm downloads GitHub dependencies as tarballs over
HTTPS where it can, without `git` , so the rewrite does not reach them.
`repos` patterns use `path.Match` syntax, where `*` does not cross a `/` ; a
pattern ending in `/**` also matches every repository nested below its
prefix, so `mediawiki/**` matches `mediawiki/skins/Vector` . An upstream
without `repos` serves every repository on it, and a repository outside
`repos` answers 404. When `upstream.auth` has credentials for an upstream,
`repos` is required: everyone who can reach the proxy can read every
repository it serves, so list only those they may read. Per-client
permissions are not supported. `upstream.auth` matches credentials by URL, so
Require generic_paths for generic upstreams with credentials
A generic upstream forwards any path below /generic/{name}/. The
documented mise setup puts a GITHUB_TOKEN on api.github.com to raise
the rate limit, so anyone who could reach the proxy could make any GET
request with that token, including for private repositories it can
read, and the answers went into the shared metadata cache. upstream.git
already refuses an upstream with credentials unless it lists its repos.
upstream.generic_paths now lists, per generic upstream, the paths it
may serve, with the same pattern syntax as git repos. Other paths
answer 404 without reaching the upstream, and so does a path holding
'%', '?', '#' or a backslash, which the upstream could read as a
different path than the one that matched. Config validation refuses a
generic upstream that upstream.auth has credentials for unless it has
generic_paths, which also covers a github.com token reaching the
generic github upstream. Upstreams without credentials keep serving
every path.
The docs and example config show the paths mise needs and recommend a
fine-grained token limited to public repositories.
2026-10-24 08:02:22 +02:00
they are sent by every route to that host, not only `/git` . A `generic`
upstream at the same URL, such as the one the mise setup in the README uses,
checks `generic_paths` instead of `repos` , and needs them too; keep Git paths
like `{owner}/{repo}.git/info/refs` out of its patterns.
Serve Git repositories from a local cache (#492)
* Serve Git repositories from a local cache
Named upstream.git hosts are served read-only over Git's smart HTTP
protocol at /git/{name}/{repo}, so git clone and git fetch work against
the proxy through an insteadOf rewrite. Each repository is a bare clone
at git.cache_dir/{name}-{hash of the upstream URL}/{repo}.git, served
by git upload-pack.
Every request for refs checks the upstream's branches, tags and HEAD
with a protocol v2 ls-refs limited to those prefixes, and fetches only
when they changed, atomically and with --prune. Concurrent requests
share one refresh. Only branches and tags are cached, and HEAD follows
the upstream's default branch.
A commit a client asks for by ID that the cache lacks, such as a pull
request head, is fetched and kept under a hidden ref until a branch or
tag includes it or for 7 days. One the upstream cannot provide is not
asked for again for a minute, or an hour when it is larger than
git.max_repo_size leaves. This is off for upstreams upstream.auth has
credentials for, since GitHub serves commits from any repository in a
fork network.
git never reaches an upstream itself. Its fetches go through a loopback
relay that forwards with the proxy's upstream client, so address
checks, upstream.auth and the access log apply, and git never sees a
credential. Redirects are not followed, and git runs with an
environment of its own.
The cached copy is served while the upstream fails or is slower than
git.refresh_wait, until git.stale_max_age has passed since the last
successful check. Refusals and missing repositories are never answered
from the cache.
git.max_repo_size cuts a transfer off as it streams, and every fetched
pack is kept as a pack, so a cut-off leaves nothing behind. A
repository that can no longer be refreshed within the limit is cloned
again with only its branches and tags. git.max_size evicts the least
recently used repositories, never one in use. At startup only the
directory of a repointed upstream's old URL is removed, and Close waits
for refreshes in progress.
Hosts with upstream.auth credentials need a repos allowlist, and a
repository outside it answers 404.
Refs #403
* Document setting the Git rewrite through environment variables
The Git section of the configuration docs now explains what insteadOf
does, with the URL a clone fetches from, and shows the same rewrites set
through GIT_CONFIG_COUNT, GIT_CONFIG_KEY_n and GIT_CONFIG_VALUE_n for
CI. Package managers that run git for Git dependencies inherit those
variables, so their fetches are cached too; npm's GitHub tarball
downloads do not go through git and are not.
Refs #403
* Note that upstream.auth credentials reach every route to a host
Credentials are matched by URL, so a token configured for a Git upstream
is also sent by a generic upstream at the same URL, which forwards any
path and checks no repos allowlist. With a token for a private GitHub
org, /generic/github/acme/private.git/info/refs would list that
repository's refs to anyone who can reach the proxy.
* Cap concurrent upload-pack processes and request body size
Every POST git-upload-pack started a pack-objects with git's defaults,
a delta-search thread per CPU and unbounded window memory, and nothing
limited how many ran at once, so a burst of CI clones of one large
repository could run the proxy out of memory. At most two upload-pack
processes per CPU (GOMAXPROCS, so a container's limit) now build packs
at once, and further requests wait for a slot while the client is
connected. ls-refs builds no pack and does not wait. Serving runs with
pack.threads=1 and pack.windowMemory=64m, and every git process with
core.bigFileThreshold=8m, so index-pack streams large files instead of
holding them in memory.
A request body was streamed to upload-pack unbounded, so a small gzip
body could feed it an arbitrarily long negotiation for as long as the
request may run. Bodies are now limited to 100MB, both as received and
decoded, and one past the limit fails with an error instead of being
cut short quietly.
* Apply refresh_wait to the ref check only
refresh_wait cut off the whole refresh, fetch included, so after a push
with a large pack a client got the old refs once it passed, though the
upstream was fine and the docs say a clone right after a push sees the
new commit. It now bounds only the check of the upstream's branches and
tags. Once the check finds they changed, requests wait for the fetch;
one that fails, or stalls and is abandoned by http.lowSpeedTime, still
falls back to the cached copy within stale_max_age.
* Lock cache_dir against a second process
The cache clears its temporary directory at startup, so a second
process given the same cache_dir, such as a second pod on a shared
volume, would delete the first one's clones in progress, and each
process's eviction knows only its own readers. New now locks
{cache_dir}/.gitcache-lock before touching anything and holds it until
Close: flock on Unix, a handle opened without sharing on Windows. The
OS releases it when the process exits, so a crashed process can
restart. A second process fails to start with an error naming the
directory.
* Evict again once repositories in use are released
Eviction skips repositories a request or refresh is using, and only the
release of one too large to refresh ran it again, so a cache filled past
max_size while its repositories were in use stayed over it once they
went idle, until the next fetch. Eviction now records when it is left
over max_size, and the next release runs it again.
* Cap git.max_repo_size at git.max_size
A repository larger than max_size could still be cloned, up to
max_repo_size, which defaults to 5GB whatever max_size is. Once released
it was evicted to bring the cache back within max_size, and the next
request cloned it again. max_size now caps max_repo_size, so such a
repository is refused for an hour like any other too large one, with an
error that names both limits.
* Count fetches in progress toward git.max_size
Fetches enforced only max_repo_size, and eviction ran after each one,
so up to 16 concurrent fetches could take the cache past max_size by as
much as each may download. Each fetch now reserves space within
max_size as the relay receives its bytes, a chunk at a time, evicting
least recently used idle repositories to make room. When the fetch
ends, the repository's measured size replaces the reservation under the
cache lock, so its bytes are neither counted twice nor missed.
When evicting every idle repository would not make room, none is
evicted for the fetch, and it is cut off with ErrCacheFull rather than
ErrTooLarge, so the repository is not refused for an hour: a clone
answers 507, and a refresh serves the cached copy as when the upstream
fails. A fetch that runs out of room while another has just been cut
off waits, up to a minute, for that one's space, so two fetches that do
not fit together do not both fail. A fetch that was cut off never
waits, so no two wait on each other.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
2026-10-21 03:47:16 -07:00
Every request for a repository's refs checks the upstream's branches and tags,
and fetches only when they changed, so a clone right after a push sees the new
commit. Requests that arrive while a check or fetch is running share it. Only
branches and tags are cached and served; other refs, such as Gerrit's
`refs/changes/*` , are left out, and protocol v2 lets the server filter them
before sending.
A commit a client asks for by object ID that is on no cached branch or tag,
such as a pull request head fetched by `actions/checkout` , is fetched from the
upstream when the client asks for it, and kept until a cached branch or tag
includes it or for 7 days. A commit the upstream could not provide, or could
not be reached for, is not asked for again for a minute, or for an hour when
it is larger than what `max_repo_size` leaves. This is off for upstreams that `upstream.auth` has
credentials for: GitHub serves a commit from any repository in a fork network
through any other, so a fetch by ID could reach repositories outside `repos` .
Any commit the cache holds can be fetched by ID, including one a force-push
left behind, as from the upstream.
If the upstream fails, or its check of branches and tags takes longer than
`refresh_wait` , a repository checked successfully within `stale_max_age` is
served from the cache while the refresh continues. Once the check finds new
commits, the request waits for their fetch, so a clone right after a push gets
them; if the fetch fails or stalls, the cached copy is served. A
`stale_max_age` of `0` never serves a copy the upstream has not just
confirmed. A repository the upstream reports missing (404), refuses (401, 403)
or redirects is not served from the cache. Redirects are never followed, so
point the upstream at a repository's current location.
The proxy runs `git` 2.31 or newer, which must be installed (the Docker image includes it).
The `git` processes never reach the network themselves: their fetches go
through a loopback endpoint in the proxy that forwards them with the same
upstream client as every other request, so the address checks
(`allow_loopback` , `allow_private_hosts` ), `upstream.auth` and the access log
apply, and credentials are never passed to `git` .
`max_repo_size` defaults to 5GB, and `max_size` to no limit. `max_size` also
caps `max_repo_size` , since a repository larger than the whole cache would be
evicted as soon as no request was reading it. A clone or fetch that would
pass `max_repo_size` is stopped, and nothing of it is kept. A new repository
is then refused for an hour. When a refresh of one already cached does not
fit, it is served as it is and dropped once no request is reading it, so the
next request clones it again with only its branches and tags; if that is
still too large, it is refused for an hour. A commit fetched by ID that does
not fit is refused, and the repository kept.
`max_size` counts fetches in progress as well as cached repositories, so the
cache stays within it however many fetches run at once. As a fetch receives
data, the least recently used repositories are evicted to make room, never
one a request is reading; those are evicted once released if the cache is
still over. When evicting every idle repository would not make room, none is
evicted for the fetch, and it is stopped and nothing of it is kept, but the
repository is not refused: a clone answers 507, and a refresh serves the
cached copy, as when the upstream fails. The count is of bytes received, so
index files and garbage collection can take the cache a little past
`max_size` until the fetch ends and the repository is measured.
A transfer that receives nothing for 10 minutes is abandoned. Git's automatic
garbage collection runs as part of fetches, and files over 8MB are kept
without delta compression, so one large file does not take a fetch's memory.
Packs for clients are built by at most two `upload-pack` processes per CPU
the proxy may use, each with one delta-search thread and 64MB of delta
window, and further clones wait for one to finish; listing refs does not
wait. An `upload-pack` request is limited to 100MB, before and after gzip.
Repositories are kept under `{cache_dir}/{name}-{hash of the URL}/` , so
repointing an upstream starts a fresh cache, and the old URL's directory is
removed at startup. Nothing else in `cache_dir` is touched, and the
directories of upstreams no longer configured are only logged, so remove
those yourself. The proxy locks `cache_dir` while it runs, through a
`.gitcache-lock` file in it, so a second process given the same directory,
such as another replica on a shared volume, fails to start; give each its own.
2026-10-25 14:57:49 +01:00
Git LFS objects are cached in the artifact cache, so they use the configured
storage backend and retention rather than `cache_dir` . git-lfs finds the LFS
API under the rewritten remote, at `/git/{name}/{repo}.git/info/lfs` , with no
further configuration. A batch request is passed to the upstream, which
decides which objects the repository has, and each download is pointed at
the proxy. An object the cache doesn't have is fetched from the download URL
2026-10-25 16:46:40 +01:00
the batch returned, which the proxy keeps in memory until shortly before it
expires, or else from one a fresh batch request returns. That URL is often
on another host than the Git server, such as GitHub's or Hugging Face's
object storage; the proxy fetches from whatever host the upstream names,
with the usual address checks, and `upstream.auth` credentials only go to
the hosts they are configured for. A download that isn't the object's size
is cut off. If the object's content is already stored under another
repository, such as a fork, it is copied from there instead of downloaded,
once the upstream has confirmed this repository has it; the copy takes its
own space.
2026-10-25 16:53:11 +01:00
2026-10-25 20:29:03 +01:00
Large LFS objects may need longer timeouts. An object the proxy doesn't
have is sent to git-lfs as it is downloaded (see [Cache misses ](#cache-misses )),
but the download must finish within `artifact_timeout` and the whole response
within `write_timeout` ; both default to 5 minutes.
2026-10-25 16:08:15 +01:00
Download URLs are often signed; only the URL without its query is logged
and recorded with the object. An object is only cached once its content
matches its sha256 object ID. Objects are cached per repository, so one
repository's objects are never served for another. When the upstream can't
be reached or fails, a batch is answered with the objects cached for the
repository, and the others are reported as unavailable; a batch the upstream
refuses is refused. A cached object stays available after it is removed
upstream, until retention removes it or you delete it through the
[cache delete API ](#cache-delete-api ).
2026-10-25 14:57:49 +01:00
Pushing, LFS uploads and locking, and the dumb HTTP protocol are not
supported. The settings
Serve Git repositories from a local cache (#492)
* Serve Git repositories from a local cache
Named upstream.git hosts are served read-only over Git's smart HTTP
protocol at /git/{name}/{repo}, so git clone and git fetch work against
the proxy through an insteadOf rewrite. Each repository is a bare clone
at git.cache_dir/{name}-{hash of the upstream URL}/{repo}.git, served
by git upload-pack.
Every request for refs checks the upstream's branches, tags and HEAD
with a protocol v2 ls-refs limited to those prefixes, and fetches only
when they changed, atomically and with --prune. Concurrent requests
share one refresh. Only branches and tags are cached, and HEAD follows
the upstream's default branch.
A commit a client asks for by ID that the cache lacks, such as a pull
request head, is fetched and kept under a hidden ref until a branch or
tag includes it or for 7 days. One the upstream cannot provide is not
asked for again for a minute, or an hour when it is larger than
git.max_repo_size leaves. This is off for upstreams upstream.auth has
credentials for, since GitHub serves commits from any repository in a
fork network.
git never reaches an upstream itself. Its fetches go through a loopback
relay that forwards with the proxy's upstream client, so address
checks, upstream.auth and the access log apply, and git never sees a
credential. Redirects are not followed, and git runs with an
environment of its own.
The cached copy is served while the upstream fails or is slower than
git.refresh_wait, until git.stale_max_age has passed since the last
successful check. Refusals and missing repositories are never answered
from the cache.
git.max_repo_size cuts a transfer off as it streams, and every fetched
pack is kept as a pack, so a cut-off leaves nothing behind. A
repository that can no longer be refreshed within the limit is cloned
again with only its branches and tags. git.max_size evicts the least
recently used repositories, never one in use. At startup only the
directory of a repointed upstream's old URL is removed, and Close waits
for refreshes in progress.
Hosts with upstream.auth credentials need a repos allowlist, and a
repository outside it answers 404.
Refs #403
* Document setting the Git rewrite through environment variables
The Git section of the configuration docs now explains what insteadOf
does, with the URL a clone fetches from, and shows the same rewrites set
through GIT_CONFIG_COUNT, GIT_CONFIG_KEY_n and GIT_CONFIG_VALUE_n for
CI. Package managers that run git for Git dependencies inherit those
variables, so their fetches are cached too; npm's GitHub tarball
downloads do not go through git and are not.
Refs #403
* Note that upstream.auth credentials reach every route to a host
Credentials are matched by URL, so a token configured for a Git upstream
is also sent by a generic upstream at the same URL, which forwards any
path and checks no repos allowlist. With a token for a private GitHub
org, /generic/github/acme/private.git/info/refs would list that
repository's refs to anyone who can reach the proxy.
* Cap concurrent upload-pack processes and request body size
Every POST git-upload-pack started a pack-objects with git's defaults,
a delta-search thread per CPU and unbounded window memory, and nothing
limited how many ran at once, so a burst of CI clones of one large
repository could run the proxy out of memory. At most two upload-pack
processes per CPU (GOMAXPROCS, so a container's limit) now build packs
at once, and further requests wait for a slot while the client is
connected. ls-refs builds no pack and does not wait. Serving runs with
pack.threads=1 and pack.windowMemory=64m, and every git process with
core.bigFileThreshold=8m, so index-pack streams large files instead of
holding them in memory.
A request body was streamed to upload-pack unbounded, so a small gzip
body could feed it an arbitrarily long negotiation for as long as the
request may run. Bodies are now limited to 100MB, both as received and
decoded, and one past the limit fails with an error instead of being
cut short quietly.
* Apply refresh_wait to the ref check only
refresh_wait cut off the whole refresh, fetch included, so after a push
with a large pack a client got the old refs once it passed, though the
upstream was fine and the docs say a clone right after a push sees the
new commit. It now bounds only the check of the upstream's branches and
tags. Once the check finds they changed, requests wait for the fetch;
one that fails, or stalls and is abandoned by http.lowSpeedTime, still
falls back to the cached copy within stale_max_age.
* Lock cache_dir against a second process
The cache clears its temporary directory at startup, so a second
process given the same cache_dir, such as a second pod on a shared
volume, would delete the first one's clones in progress, and each
process's eviction knows only its own readers. New now locks
{cache_dir}/.gitcache-lock before touching anything and holds it until
Close: flock on Unix, a handle opened without sharing on Windows. The
OS releases it when the process exits, so a crashed process can
restart. A second process fails to start with an error naming the
directory.
* Evict again once repositories in use are released
Eviction skips repositories a request or refresh is using, and only the
release of one too large to refresh ran it again, so a cache filled past
max_size while its repositories were in use stayed over it once they
went idle, until the next fetch. Eviction now records when it is left
over max_size, and the next release runs it again.
* Cap git.max_repo_size at git.max_size
A repository larger than max_size could still be cloned, up to
max_repo_size, which defaults to 5GB whatever max_size is. Once released
it was evicted to bring the cache back within max_size, and the next
request cloned it again. max_size now caps max_repo_size, so such a
repository is refused for an hour like any other too large one, with an
error that names both limits.
* Count fetches in progress toward git.max_size
Fetches enforced only max_repo_size, and eviction ran after each one,
so up to 16 concurrent fetches could take the cache past max_size by as
much as each may download. Each fetch now reserves space within
max_size as the relay receives its bytes, a chunk at a time, evicting
least recently used idle repositories to make room. When the fetch
ends, the repository's measured size replaces the reservation under the
cache lock, so its bytes are neither counted twice nor missed.
When evicting every idle repository would not make room, none is
evicted for the fetch, and it is cut off with ErrCacheFull rather than
ErrTooLarge, so the repository is not refused for an hour: a clone
answers 507, and a refresh serves the cached copy as when the upstream
fails. A fetch that runs out of room while another has just been cut
off waits, up to a minute, for that one's space, so two fetches that do
not fit together do not both fail. A fetch that was cut off never
waits, so no two wait on each other.
---------
Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com>
2026-10-21 03:47:16 -07:00
under `git` can also be set with `PROXY_GIT_CACHE_DIR` , `PROXY_GIT_MAX_SIZE` ,
`PROXY_GIT_MAX_REPO_SIZE` , `PROXY_GIT_REFRESH_WAIT` and
`PROXY_GIT_STALE_MAX_AGE` .
Serve additional Maven repositories at /maven/{name}/ (#539)
* Serve additional Maven repositories at /maven/{name}/
upstream.maven_repositories maps names to further Maven repositories,
each served at /maven/{name}/ from its own upstream, like
debian_repositories does for APT. upstream.maven stays the default at
/maven/ with unchanged cache keys and its Gradle Plugin Portal fallback;
named repositories don't fall back to the portal.
Named repositories cache their files under {name}/{filename} and hash
their metadata keys from name, upstream URL and path, so the same
coordinates in two repositories, which may hold different bytes, never
share a cache entry.
A repository name is matched before the default repository and would
hide group IDs starting with it, so names of two letters or less and
common top-level domains (com, org, net, io, ...) are refused at config
load. The documentation asks for names that can't start a group ID.
Fixes #295
* Refuse every Maven Central group root as a repository name
The hand-written list missed real roots such as tools (Jackson 3),
software (AWS SDK), jakarta, javax and junit, so a repository named after
one would have hidden those groups in the default repository. The name
check now uses the first path segments of all group IDs on Maven Central,
plus Google's androidx and android, embedded from the repository's
directory listing.
Names are compared case-sensitively, since routing is, so "ORG" is no
longer refused for hiding "org".
A named repository's files are now cached under its name and a hash of
its upstream URL, so pointing it at a different URL stops serving what
the old one sent.
* Refuse Gradle Plugin Portal roots and unroutable repository names
- The default repository falls back to the Gradle Plugin Portal, so its
group roots (gradle, nebula, ...) would be hidden by a repository of
the same name too. The embedded list now holds the roots of Maven
Central and the Plugin Portal.
- Repository names are limited to letters, digits, '.', '_' and '-', so
a name can't contain characters such as '?', '#' or spaces that never
reach the handler as configured.
- Named repositories are built once in WithRepositories, with their
cache prefix, instead of on every request.
2026-10-21 12:10:49 +02:00
### Maven repositories
```yaml
upstream:
# The default repository, served at /maven/.
maven: "https://repo1.maven.org/maven2"
# Additional Maven repositories, served at /maven/{name}/.
maven_repositories:
grails: "https://repo.grails.org/grails/restricted"
```
Builds often need more than one Maven repository. Each entry in
`upstream.maven_repositories` is served at `/maven/{name}/` with the upstream's
layout, e.g. `/maven/grails/org/grails/grails-core/6.0.0/grails-core-6.0.0.pom` ,
and is fetched from that upstream alone: unlike the default repository, a named
one doesn't fall back to the Gradle Plugin Portal. Files are cached per
repository, so the same coordinates in two repositories never share a cache
entry.
`upstream.maven_repositories` is additive. `/maven/…` keeps serving
`upstream.maven` with unchanged cache identities. A repository name is matched
before the default repository, so it hides every group ID whose first segment
is the same name there: a repository named `tools` would take
`/maven/tools/jackson/…` away from Maven Central. At config load the proxy
refuses names of two characters or less, common top-level domains such as `com`
and `io` , and every first segment of a group ID on Maven Central, the Gradle
Plugin Portal and Google's Maven repository (about 1,200 names, such as `javax` ,
`junit` , `software` or `gradle` ). Names are compared case-sensitively, as paths
are, and may only use letters, digits, `.` , `_` and `-` . Other repositories can
have group IDs the list doesn't know, so pick names that can't be the start of
a group ID, such as `grails` or `company-releases` .
A repository's files are cached under its name and a hash of its upstream URL,
so pointing a repository at a different URL stops serving what the old one
sent.
This field has no environment variable, as with the other named upstream maps.
`PROXY_UPSTREAM_MAVEN` still sets `upstream.maven` .
2026-01-29 16:33:09 +00:00
## Authentication
2026-08-13 07:08:23 +01:00
Configure authentication for private upstream registries. The same authentication-aware client is used for metadata and artifact downloads, and credentials can reference environment variables using `${VAR_NAME}` syntax.
Send registry tokens only to the token service's own registry
After a 401 the transport fetches a token from the challenge's realm,
using credentials configured for that realm, and sends it to the host
that issued the challenge. It trusted that host if it had any configured
credentials, even unrelated ones, or shared the realm's registrable
domain. The second rule makes every tenant on a shared domain such as
jfrog.io, azurecr.io or pkg.dev the same site, and the first hands a
token minted from one registry's credentials to another registry the
proxy happens to have a key for.
A credentialed token now goes only to a registry on the realm's own
host and port, to one configured with the same credentials as the
realm, or to Docker Hub and GitLab.com, whose registries use a token
service on another host. Anonymous tokens are unaffected.
The scope in the token request also came from the challenge, so a
registry could ask for push access or for other repositories. The proxy
only reads, so it now always asks for pull access to the repository in
the request path.
2026-10-24 08:00:48 +02:00
OCI registries that return a Bearer challenge from a `/v2/{repository}/…` endpoint are handled automatically. The proxy discovers the token realm from `WWW-Authenticate` , applies any configured credentials for the token URL, and reuses the token until shortly before it expires. The token is always requested for pull access to the repository being fetched (`repository:{repository}:pull` ), whatever scope the challenge names.
2026-01-29 16:33:09 +00:00
Send registry tokens only to the token service's own registry
After a 401 the transport fetches a token from the challenge's realm,
using credentials configured for that realm, and sends it to the host
that issued the challenge. It trusted that host if it had any configured
credentials, even unrelated ones, or shared the realm's registrable
domain. The second rule makes every tenant on a shared domain such as
jfrog.io, azurecr.io or pkg.dev the same site, and the first hands a
token minted from one registry's credentials to another registry the
proxy happens to have a key for.
A credentialed token now goes only to a registry on the realm's own
host and port, to one configured with the same credentials as the
realm, or to Docker Hub and GitLab.com, whose registries use a token
service on another host. Anonymous tokens are unaffected.
The scope in the token request also came from the challenge, so a
registry could ask for push access or for other repositories. The proxy
only reads, so it now always asks for pull access to the repository in
the request path.
2026-10-24 08:00:48 +02:00
A token fetched with credentials configured for the token URL is only fetched for, and sent to, a registry that is trusted with it: one on the same host and port as the token service, one configured with the same credentials as the token URL, or Docker Hub (`registry-1.docker.io` and `index.docker.io` with `auth.docker.io` ) and GitLab.com (`registry.gitlab.com` with `gitlab.com` ). Sharing a parent domain is not enough, so `team-a.jfrog.io` is not trusted with tokens from `team-b.jfrog.io` . Any other host that answers with a challenge naming that token service gets its 401 back, so it cannot obtain a token minted from your credentials. Tokens from a token service with no configured credentials are anonymous and are fetched for any registry, as before. If your registry and its token service are on different hosts, such as a self-hosted GitLab with its registry on its own hostname, configure the same credentials for both URLs:
```yaml
upstream:
auth:
"https://registry.gitlab.example.com":
type: basic
username: "deploy-token"
password: "${GITLAB_DEPLOY_TOKEN}"
"https://gitlab.example.com/jwt/auth":
type: basic
username: "deploy-token"
password: "${GITLAB_DEPLOY_TOKEN}"
```
2026-10-14 17:56:18 +02:00
2026-01-29 16:33:09 +00:00
### Bearer Token
Used by npm, GitHub Package Registry, and many other registries:
```yaml
upstream:
auth:
"https://registry.npmjs.org":
type: bearer
token: "${NPM_TOKEN}"
"https://npm.pkg.github.com":
type: bearer
token: "${GITHUB_TOKEN}"
```
### Basic Authentication
Used by PyPI, Artifactory, and others:
```yaml
upstream:
auth:
"https://pypi.org":
type: basic
username: "__token__"
password: "${PYPI_TOKEN}"
"https://artifactory.mycompany.com":
type: basic
username: "deploy"
password: "${ARTIFACTORY_PASSWORD}"
```
### Custom Header
For registries that use non-standard authentication headers:
```yaml
upstream:
auth:
"https://maven.mycompany.com":
type: header
header_name: "X-Auth-Token"
header_value: "${MAVEN_TOKEN}"
```
2026-09-02 12:43:43 +01:00
### AWS ECR
Private ECR registries issue authorization tokens that expire after 12 hours. The `ecr` auth type calls `ecr:GetAuthorizationToken` on demand, caches the result, and refreshes it shortly before expiry, so no static credential appears in the config file:
```yaml
upstream:
oci:
ecr: "https://123456789012.dkr.ecr.eu-west-1.amazonaws.com"
auth:
"https://123456789012.dkr.ecr.eu-west-1.amazonaws.com":
type: ecr
```
AWS credentials are resolved by the SDK's default chain, which covers EKS IAM Roles for Service Accounts (IRSA), EC2/ECS instance profiles, `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` environment variables, and `~/.aws/credentials` . The IAM identity needs the `ecr:GetAuthorizationToken` action plus the usual `ecr:BatchGetImage` / `ecr:GetDownloadUrlForLayer` permissions on the target repositories. The region is inferred from private ECR IPv4, dual-stack, and FIPS hostnames. For other endpoint formats, set `region` explicitly or configure a default region for the SDK.
2026-01-29 16:33:09 +00:00
### URL Matching
2026-08-13 07:08:23 +01:00
Auth keys must be absolute URLs. Matching compares the scheme, host, effective port, and path-segment prefix, preventing credentials for `registry.example.com` from being sent to a lookalike host such as `registry.example.com.evil.test` . The longest matching scope wins, so you can configure different credentials for different paths:
2026-01-29 16:33:09 +00:00
```yaml
upstream:
auth:
# All requests to this registry
"https://registry.mycompany.com":
type: bearer
token: "${REGISTRY_TOKEN}"
# Override for a specific scope
"https://registry.mycompany.com/@private ":
type: bearer
token: "${PRIVATE_TOKEN}"
```
2026-05-04 12:15:16 +02:00
## Gradle Build Cache
The `/gradle` endpoint supports optional safeguards for upload control and cache retention.
```yaml
gradle:
build_cache:
2026-10-18 19:31:46 +02:00
read_only: true
2026-05-04 12:15:16 +02:00
max_upload_size: "100MB"
max_age: "168h"
max_size: "20GB"
sweep_interval: "10m"
```
| Config | Environment | Description |
|--------|-------------|-------------|
2026-10-18 19:31:46 +02:00
| `gradle.build_cache.read_only` | `PROXY_GRADLE_BUILD_CACHE_READ_ONLY` | Reject PUT uploads and serve GET/HEAD only (default `true` ; the variable takes `true` /`false` or `1` /`0` , any other value keeps uploads off and logs a warning) |
2026-05-04 12:15:16 +02:00
| `gradle.build_cache.max_upload_size` | `PROXY_GRADLE_BUILD_CACHE_MAX_UPLOAD_SIZE` | Maximum accepted PUT body size (must be > 0) |
Add age-based retention for cached artifacts (#428)
* Add age-based retention for cached artifacts
Cached artifacts stayed forever unless storage.max_size pushed them out.
storage.retention adds a default, per-ecosystem and per-package duration
after which an artifact nobody downloaded gets evicted.
Ecosystems opt in one at a time: a handler registers with
retention.Default, and until it does, configuring that ecosystem (or one
of its packages) fails validation, and the default doesn't touch it. This
commit registers none, so nothing changes yet. Startup logs which
ecosystems the default covers and which it doesn't.
An artifact counts as expired when both its fetch time and its last
access are older than the cutoff. Checking only the last access would
evict a refetched artifact again right away, since clearing a record
keeps its old access time and a cache miss doesn't record a hit.
The sweep scans artifacts in id windows, so no single query holds the
SQLite connection for long, and checks each row against its own rule.
Expired records are cleared and their files go through pending_deletes
instead of being deleted directly, because a buffered hit can make an
artifact that's being served right now look expired. Buffered hits are
flushed before each sweep, and one sweep clears at most 10,000
artifacts.
Since a big first sweep can queue a lot, reclaim now keeps working
through due entries for up to 30 seconds per tick instead of stopping
after 100, and pending_deletes gets an index on queued_at.
Also:
- proxy_artifacts_evicted_total{reason, ecosystem} counts LRU and
retention evictions.
- gradle.build_cache.max_age accepts "7d", as its comment always said.
Refs #306
* Document storage retention
Describes the new storage.retention block, the order rules apply in,
that the default only covers ecosystems that support retention yet, and
what to expect from the first sweep over an old cache. Also lists the
new eviction counter and notes that gradle max_age takes days.
Refs #306
* Fix retention timing on Postgres and tighten the sweep
On Postgres, fetched_at and last_accessed_at are TIMESTAMP columns
without a zone. They hold the local time the proxy wrote, but lib/pq
reads them back labelled UTC. East of UTC that makes every artifact look
hours younger than it is, so the sweep evicted late: two hours in
Berlin, nine in Tokyo. The candidates' times are now read back as local
time, and the database tests run on both SQLite and Postgres, with one
that sets the local zone to Asia/Tokyo.
A few smaller things:
- When a delete fails, reclaim stops for the rest of the minute instead
of trying every queued path. Otherwise a storage outage turns into
thirty seconds of failed deletes and warnings every minute.
- If the buffered hits can't be written, the sweep skips that round.
Running on stale access times could clear something that is being
downloaded right now.
- DefaultCanonical only accepts a package key when rebuilding it gives
the same PURL. For alpine and deb it would otherwise produce keys
that never match.
- sweep_interval has to be at least a minute.
- New tests cover the eviction counter and an artifact that gets a hit
between the scan and the clear.
- The docs no longer say nothing changes without retention, and they
point out that the example's ecosystem and package rules only pass
validation once those ecosystems support retention.
Refs #306
* Reject qualifiers and subpaths in retention package keys
A rule names a whole package. Qualifiers like ?type=pom were dropped on
the way to the stored PURL and the rule quietly covered every file of the
package. Now such a key fails validation.
Refs #306
* Say when a package key doesn't match how packages are stored
A package rule for an ecosystem that supports retention, written in a
form its packages aren't stored under (pkg:npm/babel/core without the
@), failed with "retention is not supported for npm packages". Once the
ecosystem is enabled that is no longer true. Such a key now fails with
"does not match how npm packages are stored" and points to the package
rule forms in the docs. Keys for ecosystems without retention support
keep the old message.
Refs #306
* Note that re-running the mirror keeps packages cached
A mirror run over a package that is still cached goes through the normal
cache lookup and records a download, so the package's age starts over.
The docs only said that mirrored packages age from the time they were
mirrored.
Refs #306
* Say when an ecosystem takes no package rules
Some ecosystems will support retention without package rules, because nothing in their stored records names a package the way a PURL would (conan, helm). A package key for one of them used to fail with "does not match how conan packages are stored; see the package rule forms", which sends people looking for a form that doesn't exist. It now says package rules aren't supported there and points to the ecosystem rule.
Refs #306
* Use helm as the example of an ecosystem without package rules
Conan gets package rules once its handler stores archives under the recipe name, so the config test that shows the refusal now uses helm, whose cache records name the repository, not the chart.
Refs #306
* Say that each cached file ages on its own
The retention docs said each artifact, each version of a package, ages on its own. A version can have several cached files (a jar and its pom, several wheels, Conan sources and binaries), and each of them ages separately.
Refs #306
* Clear an expired artifact and queue its file in one transaction
The sweep cleared the record and queued the file for deletion as two separate writes. If queueing failed, or the process stopped in between, the file stayed in storage with no record pointing at it and no queue entry, and no later sweep would find it again. Both writes now share a transaction, so a failed queue rolls the clear back and the next sweep retries.
Refs #306
* Show artifact evictions on the analytics page
proxy_artifacts_evicted_total had no place on /ui/analytics. The Runtime card now lists evicted artifacts by reason and ecosystem, without the failure tint, and the coverage test records an eviction so it doesn't depend on test order.
Refs #306
* Store artifact times in UTC
fetched_at and last_accessed_at are now written in UTC, and the retention cutoffs are compared in UTC, the same convention #434 uses for the metadata cache. lib/pq sends a time with its offset and a Postgres TIMESTAMP keeps only the wall clock, so UTC on write is what reads back correctly. That replaces reinterpreting the times as local on read.
Rows written before this still hold local time. East of UTC they look younger by the offset until they are rewritten, so retention evicts them a few hours late once; west of UTC that much early.
Refs #306
---------
Co-authored-by: t <t@t>
2026-10-12 10:06:01 +02:00
| `gradle.build_cache.max_age` | `PROXY_GRADLE_BUILD_CACHE_MAX_AGE` | Delete entries older than this duration (default `168h` , also accepts days such as `7d` ; set `0` to disable) |
2026-05-04 12:15:16 +02:00
| `gradle.build_cache.max_size` | `PROXY_GRADLE_BUILD_CACHE_MAX_SIZE` | Total size cap for `_gradle/http-build-cache` , deleting oldest first (`0` disables) |
| `gradle.build_cache.sweep_interval` | `PROXY_GRADLE_BUILD_CACHE_SWEEP_INTERVAL` | Frequency for background eviction sweeps |
2026-10-18 19:31:46 +02:00
Uploads are off by default, so a client that can reach the proxy cannot overwrite cache entries that end up in other people's builds. Set `read_only: false` (or `PROXY_GRADLE_BUILD_CACHE_READ_ONLY=false` ) on the instance your CI jobs upload to, and let only CI push (`isPush = System.getenv("CI") != null` ). Before this default changed, uploads were accepted without configuration; setups that upload need the setting now.
2026-05-04 12:15:16 +02:00
`max_age` and `max_size` are independent and can be combined. When both are set, age-based eviction runs first, then size-based eviction trims remaining entries oldest-first.
2026-03-04 19:00:31 +00:00
## Cooldown
The cooldown feature hides package versions published too recently, giving the community time to spot malicious releases before they reach your projects. When a version is within its cooldown period, it's stripped from metadata responses so package managers won't install it.
```yaml
cooldown:
default: "3d"
ecosystems:
npm: "7d"
cargo: "0"
packages:
"pkg:npm/lodash": "0"
"pkg:npm/@babel/core ": "14d"
2026-10-04 20:48:41 +01:00
package_patterns:
"pkg:npm/@example/ *": "0"
2026-03-04 19:00:31 +00:00
```
| Config | Environment | Description |
|--------|-------------|-------------|
| `cooldown.default` | `PROXY_COOLDOWN_DEFAULT` | Global default cooldown |
| `cooldown.ecosystems` | - | Per-ecosystem overrides |
| `cooldown.packages` | - | Per-package overrides (keyed by PURL) |
2026-10-04 20:48:41 +01:00
| `cooldown.package_patterns` | - | Per-package glob overrides (keyed by PURL glob) |
2026-03-04 19:00:31 +00:00
Durations support days (`7d` ), hours (`48h` ), and minutes (`30m` ). Set to `0` to disable.
2026-07-17 18:11:47 -07:00
Package PURL keys are normalized to canonical form before matching, so `pkg:npm/@babel/core` and `pkg:npm/%40babel/core` are equivalent, as are `pkg:pypi/Django` and `pkg:pypi/django` . If both forms configure the same package, the canonical entry wins.
2026-07-13 17:11:04 -07:00
2026-10-31 09:49:35 +00:00
Go module names in exact cooldown overrides are matched case-insensitively. If both lowercase and mixed-case keys are configured, the lowercase entry wins. Go package patterns match lowercase module names.
2026-10-04 20:48:41 +01:00
`package_patterns` matches canonical, versionless PURLs using `*` for zero or more characters and `?` for one character. Neither wildcard crosses `/` separators. Character classes (`[...]` ) and backslash escapes are rejected at startup. For example, `"pkg:npm/@example/*"` matches packages under the `@example` npm scope. Patterns accept `@` as an alias for `%40` ; other characters must use their canonical PURL form. Equivalent patterns with different durations are rejected at startup. Equal durations, such as `1d` and `24h` , are accepted.
Exact `packages` entries take precedence over patterns. When several patterns match, longer patterns win after excluding `*` and `?` from the length. Ties use lexical order of the normalized patterns.
Resolution order: exact package override, then package pattern, then ecosystem override, then global default. This lets you set a conservative default while exempting trusted package families.
2026-03-04 19:00:31 +00:00
2026-10-28 16:25:08 +01:00
The web UI's dashboard shows the default and the ecosystem overrides, and how many package overrides and patterns there are; their names aren't shown, since the UI has no authentication. A package's page shows the cooldown that applies to it and which kind of rule it comes from; it shows none for ecosystems without cooldown support, and the dashboard marks an ecosystem override for one as not supported.
2026-10-27 13:03:42 +01:00
2026-10-28 16:25:08 +01:00
Currently supported for npm, PyPI, pub.dev, Composer, Cargo, NuGet, Conda, RubyGems, Hex, Go, Maven and Helm. Most of these ecosystems include publish timestamps in their metadata; Go and Maven are described below.
Apply cooldown to Go modules
With cooldown on, @v/list leaves out versions whose .info time is
inside the window, and @latest falls back to the newest release out of
cooldown when upstream's answer is too new. Each listed version's
.info is read once and its time is stored in the versions table, as
npm does, except for modules whose path has uppercase letters, since
golang PURLs are lowercased.
A .info, .mod or .zip for a version in cooldown, and a query such as a
branch name that resolves to one, gets 403. The go command moves on to
the next GOPROXY entry after 404 or 410, so a 404 would let
GOPROXY=proxy,direct fetch the version directly.
Go has no publish time: .info carries the commit time of the revision,
so a version tagged on an old commit counts as old. The docs say so.
Part of #132
2026-10-23 12:40:19 +02:00
2026-10-23 12:46:39 +02:00
Go has no publish time. The module proxy's `.info` gives the commit time of the revision a version points at, so a version tagged on an old commit counts as old. The proxy reads each listed version's `.info` once, stores the time, and leaves versions in cooldown out of `@v/list` and `@latest` . When upstream's `@latest` is in cooldown, it serves the newest release out of cooldown instead. Requests for a version in cooldown, including its `.info` , `.mod` and `.zip` , and a branch or commit query that resolves to one, get 403 rather than 404, because the go command moves on to the next `GOPROXY` entry, such as `direct` , after a 404. If a version's `.info` can't be fetched, the list or download fails with 502 instead of letting the version through.
2026-04-06 12:55:15 +01:00
2026-10-23 20:53:06 +02:00
Maven metadata has no publish time per version. The proxy uses the `Last-Modified` header of each version's `.pom` , which on Maven Central is the time the version was deployed. It reads the header once per version and stores the time, for each repository under `upstream.maven_repositories` separately: from the response when the `.pom` or Gradle `.module` itself is downloaded, which clients do before a version's other files, and otherwise with a `HEAD` request. A `.pom` or `.module` refused this way is still cached, so it counts toward `max_size` before the version leaves cooldown. To keep requests to the upstream down, it only checks versions that can still be in cooldown: none when the metadata's `<lastUpdated>` is older than the cooldown, and otherwise the versions from the end of the list back to the first one that is old enough. Versions in cooldown are left out of `maven-metadata.xml` , `<latest>` and `<release>` are set to the last versions left, and the metadata's `.md5` , `.sha1` , `.sha256` and `.sha512` files are computed from what is served. Requests for any file of a version in cooldown get 403, as for a version on the denylist. If a `.pom` can't be checked, the metadata or download fails with 502 instead of letting the version through, or with 503 and `Retry-After` when the upstream rate-limits the proxy; a version without a `.pom` is let through.
2026-10-23 19:38:28 +02:00
Maven caveats:
- A Nexus or Artifactory upstream may report when it cached the `.pom` rather than when it was published, so a version can stay in cooldown longer than it should.
- On Google's Maven Central mirror, `Last-Modified` is when the file reached the mirror's bucket, a few hours after it was published, so versions stay in cooldown a little longer. The upload time is in `x-goog-meta-last-modified` , which the proxy does not read. It is missing for files from the mirror's 2019 bulk import.
- Maven Central lists versions in version order, not in the order they were published. A patch to an older release line published after a newer release that is out of cooldown stays listed while it is in cooldown, but its files are still refused.
- SNAPSHOT versions are not held back.
2026-10-28 16:25:08 +01:00
Helm uses each release's `created` time from the repository's `index.yaml` . Releases in cooldown are left out of the index the proxy serves, and their archives aren't served; a release without `created` is let through. Charts pulled over OCI have no publish time and aren't held back.
2026-10-25 17:29:22 +01:00
RubyGems, Hex and PyPI keep their publish times outside the metadata clients read: the RubyGems versions API, the Hex HTTP API (`upstream.hex_api` ) and the PyPI JSON API. The proxy fetches them alongside each compact index, package or simple page it filters, and for PyPI downloads. Cooldown fails closed: when it applies to a package and those times can't be fetched or parsed, or the metadata can't be rewritten, the request fails with 502, or with 503 and `Retry-After` while the upstream is rate limiting the proxy, instead of serving the metadata unfiltered. The same goes for npm, Composer and pub.dev metadata that can't be rewritten, and for npm tarball downloads whose publish time can't be read. A PyPI file or npm tarball that is already cached is still served when its publish time can't be read; only a download that would have to be fetched fails. A PyPI release's time is the upload time of its newest file. The proxy stores it when a download looks it up, and serves files of that release that are already cached against the stored time without reading the JSON API again. A file that isn't cached yet is always looked up, since it may have been added to the release since, and the newer time is stored. Packages exempted with a duration of `0` are served as before and don't need the times. A PyPI package the JSON API returns 404 for has no times to go by and is served unfiltered.
Fail closed in gem, Hex and PyPI cooldown
RubyGems, Hex and PyPI keep publish times outside the metadata clients
read, so cooldown fetches them from a second API. When that fetch
failed, was rate limited, or returned something that didn't parse, the
compact index, Hex package or simple page was served unfiltered, and a
Hex package that couldn't be re-encoded was passed on as upstream sent
it. A flaky or rate-limited timestamp API let every version in
cooldown through.
These now fail with 502, or with 503 and Retry-After when the rate
limit gate held the request, as the Go handler already does. PyPI file
downloads check the same way. A PyPI package the JSON API answers 404
for still has no times and is served unfiltered, as before.
npm, Composer, pub.dev and PyPI JSON metadata that couldn't be
rewritten was also passed on unfiltered unless the package had denied
versions. It now fails with 502 whenever cooldown applies too. npm
tarball downloads whose packument couldn't be fetched skipped the
cooldown check and downloaded straight from upstream; they now fail.
All of this only applies when cooldown covers the package: packages
exempted with a duration of 0 skip the timestamp lookups, and the gem
and Hex handlers serve them through the metadata cache as when cooldown
is off.
2026-10-24 08:11:25 +02:00
2026-10-30 15:06:42 +01:00
For RubyGems and Hex, both documents go through the metadata cache. With [`cache_metadata` ](#metadata-caching ) they are served from it within `metadata_ttl` without asking the upstream, revalidated with their ETags after it, and filtered from the last cached copies, marked stale, while the upstream is down; without it they are still fetched once for concurrent requests. Each document has its own TTL, so the compact index or Hex package can be newer than the cached times and list a version they don't have. The times are then fetched again before filtering, and if that fails, such versions are left out until the times can be read. The filtered index or package carries an ETag of its own, so Bundler and Mix revalidate it with `304 Not Modified` . Publish times are compared with the current time on each request, and the filtered result is kept until the first version it held back leaves cooldown (see [rewritten metadata cache ](#rewritten-metadata-cache )); a newly published version shows up within `metadata_ttl` , as with other cached metadata. Upstream errors are answered as for other cached metadata (404, or 502) rather than relayed.
Read gem, Hex and conda cooldown metadata through the metadata cache
With cooldown on, the RubyGems compact index and versions API, the Hex
package and Hex API entry, and conda's repodata.json were fetched with
plain HTTP requests on every client request: no TTL, no revalidation,
no coalescing and no stale fallback, even with cache_metadata.
Gem and Hex now read both documents through the metadata cache and
filter the cached copies. The filtered result goes through the rewrite
cache and is served with an ETag. When the times document is older than
the index and misses a version, it is fetched again; if that fails the
version is left out. Conda filters a fresh stored repodata.json as it
reads it from storage, uses the metadata cache otherwise, and remembers
the ETag of the filtered copy until a withheld entry leaves cooldown.
2026-10-25 17:41:40 +01:00
2026-10-27 16:19:14 +01:00
With cooldown on, the proxy asks npm for full packuments, since the abbreviated ones it uses otherwise carry no publish times. Abbreviated packuments cached before cooldown was turned on are fetched again in full rather than served, including as a stale copy when the upstream is down. When `latest` is in cooldown, the packument tags the newest version out of cooldown instead, and `npm dist-tag ls` shows the same tags.
Fix npm search routing and abbreviated packuments under cooldown
`npm search` requests /-/v1/search?text=... . The path has no "/-/"
after the leading slash is trimmed, so it fell through to the package
metadata handler: upstream got /-/v1%2Fsearch without the query
string, and the answer was cached under "-/v1/search", so every later
search was served the first one's results. Search and /-/ping (used
by `npm ping`, which lost its query string the same way) are now
relayed to upstream uncached with their query strings, like the
signing-keys endpoint.
Packuments share one metadata cache row whatever format they were
fetched in. Without cooldown the proxy asks for the abbreviated
format, which has no "time" map. After cooldown was turned on, a
still-fresh abbreviated packument kept being served from cache, and
used as the stale copy when upstream failed, so cooldown filtered
nothing and the tarball check let recent versions through. When full
metadata is required (cooldown or npm_full_metadata on), packument
lookups now pass a validator that rejects an abbreviated document
(top-level "modified", no "time"), so it counts as a cache miss, is
refetched in full and is never used as a stale fallback. Behaviour
without cooldown is unchanged.
fetchMetadataDocument takes the validate callback that
fetchOrCacheMetadata already has; other callers pass nil.
2026-10-23 13:22:49 +02:00
2026-10-25 19:17:18 +01:00
Conda cooldown covers `repodata.json` , `current_repodata.json` , `repodata_from_packages.json` and sharded repodata (CEP 16). Other forms of the repodata, such as `repodata.json.zst` , `repodata.json.bz2` and `repodata.jlap` patches, would list versions still in cooldown, so with cooldown on they are answered 404 and conda and mamba fall back to `repodata.json` . With `cache_metadata` , `repodata.json` is read through the metadata cache as it is without cooldown: within `metadata_ttl` the stored copy is filtered as it is read from storage, after it the upstream is asked with its ETag, and while the upstream is down the stale copy is filtered. Once a document has been filtered, the proxy remembers the ETag of the result until the first entry it left out leaves cooldown, so a client revalidating with `If-None-Match` gets `304 Not Modified` without the document being filtered or sent again. The first response for each document has no ETag, as it is sent while it is filtered. Without `cache_metadata` , `repodata.json` is fetched and filtered on every request. Clients cache shards by the hash in the shard index, so with cooldown on the proxy serves a rewritten index whose shard hashes change every hour and filters each shard as it is fetched. Versions leaving cooldown show up within the hour, and clients fetch the shards they need again once an hour. If a channel's shard index points packages or shards somewhere the proxy doesn't route, the proxy answers 404 for it and clients fall back to `repodata.json` .
Serve conda sharded repodata through cooldown and the cache
The shard index (repodata_shards.msgpack.zst) and the shards of CEP 16
fell through to the uncached passthrough, so clients that use them, such
as pixi, got no cooldown filtering and no offline copy.
Without cooldown, the index and shards are now served through the
metadata cache. With cooldown, shards have to be filtered, but clients
cache them by the hash in the index and never fetch a hash twice. The
proxy rewrites the index instead: each hash is XORed with a mask derived
from channel, subdir and the current hour, and shards_base_url points at
cooldown-shards/{hour}/. That route undoes the mask, fetches the
upstream shard and drops records still in cooldown. The unmasked shard
paths answer 404 while cooldown is on.
When the index puts packages or shards somewhere the proxy doesn't
route, or has an unknown version, the proxy answers 404 and clients fall
back to repodata.json, which is filtered as before.
Fixes #529
2026-10-23 12:07:01 +02:00
2026-04-06 12:55:15 +01:00
Note: Hex cooldown requires disabling registry signature verification since the proxy re-encodes the protobuf payload without the original signature. Set `HEX_NO_VERIFY_REPO_ORIGIN=1` or configure your repo with `no_verify: true` .
2026-03-04 19:00:31 +00:00
2026-09-20 14:42:30 +05:30
## Version denylist
Block exact package versions independently of cooldown and artifact scanning:
```yaml
denylist:
packages:
- "pkg:pypi/requests@2 .31.0"
- "pkg:cargo/some-crate@1 .2.3"
- "pkg:npm/%40scope/example@4 .5.6"
```
2026-10-22 20:44:02 +02:00
Entries must be versioned PURLs without subpaths or qualifiers (except a conda
`channel` , below). Package names are canonicalized (including npm scopes);
2026-09-20 14:42:30 +05:30
versions match exactly, not by range or glob. Invalid entries prevent startup.
An empty list disables the policy. Restart the proxy after changing the list.
2026-10-22 20:44:02 +02:00
Names and versions are compared the way each ecosystem treats them:
- PyPI names ignore case, and runs of `-` , `_` and `.` are equivalent (PEP 503),
so `pkg:pypi/zope.interface@6.0` also covers `zope_interface-6.0-*.whl` .
- NuGet ids and versions ignore case.
- A RubyGems entry also covers that version's platform gems:
`pkg:gem/nokogiri@1.16.0` denies `nokogiri-1.16.0-x86_64-linux.gem` .
2026-10-23 09:55:06 +02:00
- Composer versions ignore a leading `v` , which Packagist keeps from the tag:
`pkg:composer/symfony/console@6.1.7` also covers `v6.1.7` .
- Go versions are matched as written, `pkg:golang/example.com/mod@v1.0.0-RC1` ,
though download URLs escape uppercase letters (`v1.0.0-!r!c1` ).
2026-10-31 09:49:35 +00:00
Go module names in denylist entries are matched case-insensitively.
2026-10-23 09:55:06 +02:00
- A Swift entry names the registry scope and package, ignoring case:
`pkg:swift/apple/swift-nio@2.62.0` . The stored form,
Accept Swift source repository PURLs in the denylist and retention rules (#641)
* Accept Swift source repository PURLs in the denylist and retention rules
Closes #613. pkg:swift/github.com/apple/swift-nio, as SBOM tools and GHSA
write Swift packages, is resolved to its registry identity through the
upstream's /identifiers:
- Denylist entries are resolved at startup, and the proxy refuses to start
when one can't be, since a skipped entry would let the version through.
- Retention keys are accepted at validation and kept pending. The
retention loop resolves them before each sweep, and Swift isn't swept
while any is pending, so a rule that keeps a package is never skipped.
The mirror's source URL helper moves to the handler as SwiftSourceURL.
* Resolve Swift source denylist entries in proxy mirror too
serve resolved source-form Swift denylist entries through the upstream,
but proxy mirror built its denylist directly, so such an entry never
matched a mirror download. Both now use server.ConfigureDenylist, and
neither runs when an entry can't be resolved.
* Apply Swift source policies to every listed identity
A Swift registry may list several identities for one source repository,
but a denylist entry or retention rule naming the repository only
covered the first. Denying pkg:swift/github.com/acme/widget@1.0.0 let
acme.widget through when the registry listed other.widget first, and a
never-evict rule cleared the cached apple.swift-nio archive when
other.swift-nio came first.
Both now cover every identity the registry lists. A retention rule that
conflicts with another rule for any of them stays pending, with none
applied. Mirroring a source PURL still downloads the first identity.
2026-10-31 12:34:05 +01:00
`pkg:generic/swift-registry/apple.swift-nio@2.62.0` , works too, and so does
the source repository, as SBOM tools and GHSA advisories write it:
`pkg:swift/github.com/apple/swift-nio@2.62.0` . The proxy looks a source
repository up through `upstream.swift` 's `/identifiers` when `serve` or
`proxy mirror` starts and denies every identity listed for it; neither runs
when one can't be resolved.
2026-10-23 09:55:06 +02:00
- An OCI entry names the repository as it is pulled through the proxy and a
digest or tag: `pkg:oci/library/alpine@sha256:<digest>` refuses the blob or
manifest with that digest, including a manifest a tag resolves to, and
`pkg:oci/library/alpine@3.19` refuses pulling that tag. Repositories on a
named upstream keep their prefix, as in `pkg:oci/upstream/ghcr/org/image@...` .
An entry covers only the digest it names: denying an image index does not
deny the platform manifests and layers it lists to a client that requests
them by digest directly.
2026-10-22 20:44:02 +02:00
- A conda entry covers the package in every channel. Add a `channel` qualifier,
such as `pkg:conda/numpy@1.26.4?channel=conda-forge` , to deny it in one
channel only.
2026-09-20 14:42:30 +05:30
Denied versions are removed from npm packuments, Cargo sparse indexes, and PyPI
simple HTML/JSON and release metadata, including responses served from cached
metadata during an upstream outage. No publication timestamp or enabled cooldown
is required. npm tags pointing at denied versions are removed; `latest` is moved
to the highest remaining stable version when available. Other tags are not
retargeted. PyPI version-specific JSON endpoints return 404 for denied versions.
The shared artifact download pipeline returns 403 for denied package/version
identities before reading storage, issuing a signed redirect, or fetching bytes.
This also applies to mirror downloads and recognized Debian package downloads.
Existing cached artifacts are retained, so removing a denial restores access.
Already-issued storage URLs and clients' local caches cannot be revoked by this
configuration.
Metadata filtering currently covers npm, PyPI, and Cargo. Other ecosystems'
metadata is unchanged; in particular, APT indexes cannot be rewritten without
regenerating their checksums and signatures. Pass-through resources without a
resolved package/version identity are not covered. This is an explicit operator
policy, not automatic CVE detection or cache re-scanning. Exact pins or dependency
constraints may still make an install fail when no allowed version can satisfy it.
2026-09-22 12:47:37 +05:30
### APT version pinning
For Debian and Ubuntu clients, combine the proxy's download denylist with
client-side APT preferences. The proxy blocks recognized denied `.deb` downloads
with 403, but does not remove versions from APT indexes. Without a client-side
pin, APT can select a denied version and then fail when downloading it.
On each client, add a record to `/etc/apt/preferences.d/proxy-denylist.pref` :
```text
Package: somepkg
Pin: version 1.2.3-1
Pin-Priority: -1
```
Replace `somepkg` with the binary package name and `1.2.3-1` with its full APT
version, including the epoch if present. A negative priority excludes that
version from normal candidate selection. For multi-architecture clients, use
`Package: somepkg:any` to cover all architectures. Separate additional records
with blank lines. See [apt_preferences(5) ](https://manpages.debian.org/stable/apt/apt_preferences.5.en.html ).
Check the effective priorities and simulate installation before deploying the
preferences broadly:
```bash
apt-cache policy somepkg
apt-get --simulate install somepkg
```
Confirm that the denied version has priority `-1` and is not selected. Existing
specific pins can take precedence, so check for conflicts. APT can select
another candidate only if one is available and satisfies dependency constraints;
a Debian suite commonly offers only one version. Pinning does not remove an
already-installed package or automatically downgrade it.
These preferences are managed on clients, not distributed by the proxy. Keep
the corresponding proxy denylist entries as a separate download guard.
APT indexes are left unchanged to preserve upstream authentication: `Release`
metadata contains index checksums and is signed by the repository. Filtering
would require regenerating metadata and signing it with a key clients trust,
making the proxy a repository authority rather than a transparent cache. Do not
disable signature verification to work around this limitation. See
[apt-secure(8) ](https://manpages.debian.org/stable/apt/apt-secure.8.en.html ).
If centrally curated APT indexes are required, use a repository manager such as
[aptly ](https://www.aptly.info/ ) or [reprepro ](https://salsa.debian.org/debian/reprepro )
as the upstream behind the proxy. Filtered APT repository generation and signing
are outside the proxy's current scope.
2026-09-03 05:06:44 -04:00
## Artifact Scanning
Cooldown only ever looks at a version's * publish timestamp * — it never inspects the actual bytes of an artifact. Artifact scanning runs after a fetched artifact is staged into storage but before it becomes visible from cache, so an external scanner (trivy, ClamAV, Wiz, or any custom service) can block a bad verdict from ever reaching a client.
```yaml
scanning:
enabled: true
fail_open: false
timeout: 30s
signing_key: ${PROXY_SCANNING_SIGNING_KEY}
fetch_base_url: http://proxy.internal:8080
scanners:
- name: clamav
url: http://clamav-adapter:8080/scan
mode: block
- name: trivy
url: http://trivy-adapter:8081/scan
mode: monitor
ecosystems: [npm, pypi]
```
| Config | Environment | Description |
|--------|-------------|-------------|
| `scanning.enabled` | `PROXY_SCANNING_ENABLED` | Turn on the scan gate. When false (default), artifacts are cached exactly as if scanning didn't exist |
| `scanning.fail_open` | `PROXY_SCANNING_FAIL_OPEN` | Treat scanner errors/timeouts as allow instead of block. Default is fail-closed |
| `scanning.timeout` | `PROXY_SCANNING_TIMEOUT` | Per-scan-call timeout, Go duration syntax (default `30s` ) |
| `scanning.signing_key` | `PROXY_SCANNING_SIGNING_KEY` | Signs pull requests to the internal scan-fetch route. Required whenever `enabled` is true |
| `scanning.fetch_base_url` | `PROXY_SCANNING_FETCH_BASE_URL` | Address scanners use to reach this proxy to pull staged artifacts. Defaults to `base_url` |
| `scanning.scanners` | - | List of external scanning services (YAML only) |
| `scanning.scanners[].name` | - | Identifies this scanner in logs and metrics |
| `scanning.scanners[].url` | - | Endpoint the proxy POSTs scan notifications to |
| `scanning.scanners[].mode` | - | `block` (default) or `monitor` |
| `scanning.scanners[].ecosystems` | - | Restricts this scanner to specific ecosystems (e.g. `npm` , `pypi` ). Empty means all ecosystems |
| `scanning.scanners[].headers` | - | Extra HTTP headers sent with every scan request (e.g. for authenticating to the scanner service). Values support `${VAR_NAME}` expansion |
### How caching defers to a scan verdict
2026-10-25 20:29:03 +01:00
The proxy never uploads artifact bytes to a scanner. When an artifact is fetched from upstream, it's stored to the configured storage backend first, exactly as without scanning. If scanning is enabled for the artifact's ecosystem, the proxy then notifies each applicable scanner with package metadata and a short-lived, HMAC-signed URL pointing at the internal `/_internal/scan-fetch` route; each scanner GETs that URL itself to pull the exact bytes staged in storage and runs its own scan against them. With a `block` -mode scanner for the ecosystem, the client gets nothing before the verdict; with only `monitor` -mode scanners, a cache miss is sent to the client while it is stored, and the scan runs after (see [Cache misses ](#cache-misses )).
2026-09-03 05:06:44 -04:00
Scanners configured for the same ecosystem all run concurrently, never sequentially. The moment any `block` -mode scanner reports a not-allowed verdict (or errors, unless `fail_open` is set), the proxy cancels the in-flight calls to the other scanners and deletes the staged artifact — it's never committed to the cache database, so it was never visible to a client. If nothing blocks, the proxy waits for every `block` -mode scanner to finish before caching the artifact and serving it. A `monitor` -mode scanner's findings are logged and never gate the wait or the caching decision, even when it reports not-allowed.
A blocked download surfaces to the client as `403 Forbidden` with the scanner's reason, across every ecosystem handler.
2026-10-19 12:32:05 +05:30
Because scanners pull the staged bytes from the running server, the `proxy mirror` command cannot scan what it fetches. While scanning is enabled it refuses every package and container image instead of caching them unscanned. Mirror through the `/api/mirror` endpoints of the running server, whose jobs are scanned like any other download.
2026-09-03 05:06:44 -04:00
### Scanner HTTP contract
Any external service that implements this contract can act as a scanner — a trivy wrapper, a clamav-rest bridge, a Wiz connector, or an in-house service. The proxy POSTs a notify request to `scanning.scanners[].url` and waits for a JSON verdict.
**Request**
| Field | Type | Description |
|-------|------|-------------|
| `ecosystem` | string | e.g. `npm` , `pypi` , `cargo` |
| `name` | string | Package name |
| `version` | string | Package version |
| `filename` | string | Artifact filename |
| `purl` | string | Package URL (PURL) identifying this exact version |
| `content_type` | string | Artifact content type |
| `size` | integer | Artifact size in bytes |
| `fetch_url` | string | Short-lived signed URL; GET this to retrieve the exact staged bytes |
```json
{
"ecosystem": "npm", "name": "left-pad", "version": "1.0.0",
"filename": "left-pad-1.0.0.tgz", "purl": "pkg:npm/left-pad@1 .0.0",
"content_type": "application/octet-stream", "size": 1234,
"fetch_url": "https://proxy.internal/_internal/scan-fetch?path=...&exp=...&sig=..."
}
```
**Response**
| Field | Type | Description |
|-------|------|-------------|
| `allowed` | boolean | Whether the artifact may be cached and served |
| `reason` | string | Human-readable reason, surfaced to the client when `allowed` is false |
| `findings` | array | Optional list of `{"severity", "title", "description"}` objects |
```json
{
"allowed": false,
"reason": "malware detected",
"findings": [
{"severity": "critical", "title": "Trojan.GenericKD", "description": "..."}
]
}
```
The scanner must respond within `scanning.timeout` (default `30s` ); a timeout is treated the same as a `block` verdict unless `fail_open` is set.
### The `/_internal/scan-fetch` route
`fetch_url` points at an internal route, `/_internal/scan-fetch` , that streams a staged object straight from the proxy's storage backend via a short-lived HMAC-signed token (`path` , `exp` , `sig` query parameters). This works identically across every storage backend — local filesystem, S3, GCS, Azure — since it never depends on a backend-specific presigned URL, only on the one storage operation every backend already implements.
This route is not part of the public API. It's meant only for scanners to pull artifacts they've been notified about, and should be restricted to internal-network access at the ingress/network-policy layer — the HMAC scoping (one object, a short TTL) limits what a leaked token can do, but isn't a substitute for network restriction. Its query parameters are also documented in the generated [OpenAPI spec ](../README.md#openapi-swagger ).
The route only exists when scanning is actually configured: it's not mounted at all unless at least one scanner is enabled and `scanning.signing_key` is set, and it also refuses every request with `404` if either condition somehow isn't met at request time. There is no way to reach it, even with a forged token, when scanning is disabled.
2026-03-19 21:06:02 +00:00
## Metadata Caching
By default the proxy fetches metadata fresh from upstream on every request. Enable `cache_metadata` to store metadata responses in the database and storage backend for offline fallback. When upstream is unreachable, the proxy serves the last cached copy. ETag-based revalidation avoids re-downloading unchanged metadata.
2026-09-02 17:10:35 +05:30
OCI manifests and tag lists are always cached because cached image blobs cannot be pulled without their manifests and offline clients may need tag resolution. Digest-addressed manifests are immutable and served directly from cache. Tag-addressed manifests and tag lists follow `metadata_ttl` , revalidate when stale, and fall back to the last cached response when the registry is unavailable.
2026-08-13 07:35:07 +01:00
2026-10-25 17:32:22 +01:00
A stale tag is revalidated with a HEAD request carrying the client's `Accept` header, and the `Docker-Content-Digest` it returns is compared with the cached one. If the tag did not move, or moved to a manifest already cached by digest, nothing else is fetched. Otherwise the new manifest is fetched once by digest and stored under the tag and the digest. Docker Hub counts manifest GETs, but not HEADs, against its pull rate limit, so an unchanged tag costs no pulls however often its TTL runs out. A registry whose HEAD response has no digest gets a conditional GET for the tag instead. A client's HEAD request for a stale or uncached tag is passed on as a HEAD; when the digest it returns is cached, the tag is pointed at it, so the GET that follows is served from cache. Concurrent requests for the same manifest share one upstream request, and instances sharing a Postgres database take turns through the same [fetch lease ](#several-instances ) as other metadata.
2026-03-19 21:06:02 +00:00
```yaml
cache_metadata: true
```
Or via environment variable: `PROXY_CACHE_METADATA=true` .
2026-10-25 17:29:22 +01:00
Some metadata never changes once published, and is cached like an artifact, with or without `cache_metadata` , and served without revalidation whatever `metadata_ttl` says: a Go module's `.mod` and `.info` for a canonical version (a release, pre-release, pseudo-version or `+incompatible` version; the checksum database fixes their content), and RPM repodata files named after their own checksum (`repodata/<sha256>-primary.xml.gz` ). A stored copy is checked against the digest recorded for it and fetched again if it doesn't match. Go branch and commit queries, `@v/list` and `@latest` , and RPM `repomd.xml` and plainly named repodata files follow `metadata_ttl` as before. A checksum-named repodata file over `metadata_max_size` is streamed from upstream uncached.
Serve cached Composer releases without a metadata lookup
Every Composer archive download resolved its dist from the package
metadata before checking the artifact cache. With cache_metadata off,
the default, that meant an uncoalesced upstream GET of the p2 document
for every download of an archive the proxy already had: three downloads
of a cached archive made three metadata requests. With the cache on,
each download after the TTL revalidated and rescanned the document.
The lookup is only needed for the publish time cooldown checks
downloads against, since lock files name archive URLs directly. Like
the npm handler, record that time in the versions table when the dist
is first resolved, and on later downloads of a tagged release read it
from there and serve the cached archive straight away. Without a stored
time, as for archives cached by an older proxy, the download takes the
metadata path, which then records it. The time is stored even with
cooldown off, so enabling cooldown later keeps the fast path.
The cache still enforces the denylist and verifies the stored digest,
and an archive is only ever cached under a name that was checked
against the metadata, so those guarantees hold. Branches, whose
archive and time change with every push, routed packages, whose
versions row would not say which repository the time came from, and
requests with a query string such as ?sha= keep resolving from the
metadata.
2026-10-23 13:22:56 +02:00
Composer archive downloads look up their upstream URL in the same cached metadata, so an archive is resolved without asking Packagist again and stays installable while it is down. Only a version or file the cached metadata does not list yet is looked up upstream. With or without `cache_metadata` , a cached archive of a tagged release from the default repository is served without any metadata lookup; with cooldown enabled it is checked against the publish time recorded when the archive was first resolved. Branch (dev) versions, routed packages and archives picked by a query string such as `?sha=` are always resolved from the metadata.
2026-10-18 21:32:02 +02:00
2026-10-25 21:48:59 +01:00
Swift release lists and manifests are cached with the upstream's `Link` header, which carries the release list's pages and the version-specific manifests a client chooses from.
2026-03-19 21:06:02 +00:00
The `proxy mirror` command always enables metadata caching regardless of this setting.
2026-04-06 19:30:59 +01:00
### Metadata TTL
When metadata caching is enabled, `metadata_ttl` controls how long a cached response is considered fresh before revalidating with upstream. During the TTL window, cached metadata is served directly without contacting upstream, reducing latency and upstream load.
```yaml
metadata_ttl: "5m" # default
```
Or via environment variable: `PROXY_METADATA_TTL=10m` .
Set to `"0"` to always revalidate with upstream (ETag-based conditional requests still avoid re-downloading unchanged content).
When upstream is unreachable and the cached entry is past its TTL, the proxy serves the stale cached copy with a `Warning: 110 - "Response is Stale"` header so clients can tell the data may be outdated.
2026-06-02 07:59:00 +01:00
### Metadata size limit
Read gem, Hex and conda cooldown metadata through the metadata cache
With cooldown on, the RubyGems compact index and versions API, the Hex
package and Hex API entry, and conda's repodata.json were fetched with
plain HTTP requests on every client request: no TTL, no revalidation,
no coalescing and no stale fallback, even with cache_metadata.
Gem and Hex now read both documents through the metadata cache and
filter the cached copies. The filtered result goes through the rewrite
cache and is served with an ETag. When the times document is older than
the index and misses a version, it is fetched again; if that fails the
version is left out. Conda filters a fresh stored repodata.json as it
reads it from storage, uses the metadata cache otherwise, and remembers
the ETag of the filtered copy until a withheld entry leaves cooldown.
2026-10-25 17:41:40 +01:00
Upstream metadata responses are buffered in memory before being rewritten and served. `metadata_max_size` caps that buffer to protect against OOM from a misbehaving upstream. Some npm packages with thousands of versions (for example `renovate` ) exceed the 100 MB default, so raise this if you see `metadata response exceeds size limit` in the logs. Conda `repodata.json` with cooldown enabled is filtered as it streams and isn't held in memory decompressed. With `cache_metadata` the limit applies to the compressed document as it does without cooldown; one over it is filtered as it streams from the upstream and not cached.
2026-06-02 07:59:00 +01:00
```yaml
metadata_max_size: "100MB" # default
```
Or via environment variable: `PROXY_METADATA_MAX_SIZE=250MB` .
2026-10-14 08:39:12 +02:00
### Artifact size limit
2026-10-14 12:02:54 +01:00
Artifacts have no size limit by default. `artifact_max_size` refuses any artifact larger than the limit, so a client cannot fill storage or tie up the proxy by requesting huge downloads. An artifact whose upstream `Content-Length` is over the limit is refused before its body is read; one without a declared size fails once the download passes the limit. Either way nothing is cached and the client gets `502 Bad Gateway` , except when artifacts are streamed: one without a declared size has already started sending when it passes the limit, so the client sees the download cut off. The limit applies to `serve` , `mirror` and streamed artifacts.
2026-10-14 08:39:12 +02:00
```yaml
artifact_max_size: "2GB" # default: unlimited ("" or "0")
```
Or via environment variable: `PROXY_ARTIFACT_MAX_SIZE=2GB` .
Cache rewritten npm and Composer metadata (#402)
The npm and Composer handlers rewrite every metadata document they serve
so that download URLs point at the proxy: decode the whole document into
generic maps, change the URLs, encode it again, and for Composer expand
the minified format first. That ran on every request, cached metadata
included. With metadata caching on and upstream out of the picture, a
cached request still cost 7 ms for @babel/core, 17 ms for
symfony/console and 155 ms and 134 MB of allocations for typescript, and
an 8 vCPU VM serving cached npm packuments to 20 clients ran the proxy
at 450% CPU for about 1,100 requests a second.
Rewritten documents are now kept in memory, keyed by the ecosystem,
proxy URL, package and a SHA-256 of the raw document, so new bytes from
upstream are rewritten again and nothing is served stale. Requests that
arrive while a document is being rewritten wait for that rewrite rather
than running their own; a waiter leaves when its client does, and the
rewrite still completes and is cached. The denylist is fixed at startup,
so it needs no place in the key. Cooldown filtering depends on the
current time, so with cooldown on the cache is bypassed.
metadata_rewrite_cache_size bounds the cache (default "256MB", least
recently used out first, "0" to rewrite on every request). NewProxy
callers keep the old behaviour unless they set it.
Whole cached requests, measured locally:
@babel/core 7.1 ms -> 0.38 ms, 54,489 -> 119 allocations
typescript 155 ms -> 10.5 ms, 1.37M -> 133 allocations
symfony/console 17.4 ms -> 0.48 ms, 165,909 -> 123 allocations
What remains is reading the raw document from storage and hashing it.
2026-10-03 04:09:15 -07:00
### Rewritten metadata cache
2026-10-30 15:06:42 +01:00
The npm, Composer, PyPI and Helm handlers rewrite every metadata document they serve, so that download URLs point at the proxy. That means decoding the whole document and encoding it again, and Composer's minified documents are also expanded, which together cost milliseconds per typical package and far more for very large ones; a large Helm `index.yaml` takes over a hundred milliseconds. The proxy keeps rewritten documents in memory, so each distinct upstream document is rewritten once and requests arriving while it is being rewritten wait for that rewrite. New bytes from upstream are rewritten again. A PyPI simple page is cached per representation (HTML or JSON) and per set of versions filtered out of it. With version cooldown enabled, a cached rewrite expires when the first version it held back leaves cooldown, since versions it let through stay allowed and a new upstream version changes the document. A rewrite that held nothing back is kept for the shortest cooldown configured, at least a minute and at most a day. Cached documents carry an `ETag` of their own, and are dated like any other, so clients that revalidate with `If-None-Match` or `If-Modified-Since` , such as Composer, get `304 Not Modified` . The RubyGems compact index and Hex packages filtered by cooldown are kept the same way; they are never served gzip-compressed.
Cache rewritten npm and Composer metadata (#402)
The npm and Composer handlers rewrite every metadata document they serve
so that download URLs point at the proxy: decode the whole document into
generic maps, change the URLs, encode it again, and for Composer expand
the minified format first. That ran on every request, cached metadata
included. With metadata caching on and upstream out of the picture, a
cached request still cost 7 ms for @babel/core, 17 ms for
symfony/console and 155 ms and 134 MB of allocations for typescript, and
an 8 vCPU VM serving cached npm packuments to 20 clients ran the proxy
at 450% CPU for about 1,100 requests a second.
Rewritten documents are now kept in memory, keyed by the ecosystem,
proxy URL, package and a SHA-256 of the raw document, so new bytes from
upstream are rewritten again and nothing is served stale. Requests that
arrive while a document is being rewritten wait for that rewrite rather
than running their own; a waiter leaves when its client does, and the
rewrite still completes and is cached. The denylist is fixed at startup,
so it needs no place in the key. Cooldown filtering depends on the
current time, so with cooldown on the cache is bypassed.
metadata_rewrite_cache_size bounds the cache (default "256MB", least
recently used out first, "0" to rewrite on every request). NewProxy
callers keep the old behaviour unless they set it.
Whole cached requests, measured locally:
@babel/core 7.1 ms -> 0.38 ms, 54,489 -> 119 allocations
typescript 155 ms -> 10.5 ms, 1.37M -> 133 allocations
symfony/console 17.4 ms -> 0.48 ms, 165,909 -> 123 allocations
What remains is reading the raw document from storage and hashing it.
2026-10-03 04:09:15 -07:00
2026-10-29 10:52:39 +01:00
Each cached document is kept gzip-compressed, when that makes it smaller, and served with `Content-Encoding: gzip` to clients that accept it, as npm, pnpm, Yarn and Composer do. A large packument is compressed once rather than on every request, by the proxy or by a reverse proxy in front of it. Only the compressed copy is kept, typically an eighth of the document, and it is decompressed for the clients that don't accept gzip. Documents rewritten outside the cache are served uncompressed.
Size the cache for the metadata of the packages one install asks for. pnpm 12 checks every lockfile entry against the registry on each install unless `trustLockfile` is set, so each install reads the packument of every package in the lockfile; when they don't all fit, each install evicts what the next one needs and rewrites them all again. A lockfile of about 550 npm packages takes about 50 MB.
2026-10-15 12:46:55 +02:00
Cache rewritten npm and Composer metadata (#402)
The npm and Composer handlers rewrite every metadata document they serve
so that download URLs point at the proxy: decode the whole document into
generic maps, change the URLs, encode it again, and for Composer expand
the minified format first. That ran on every request, cached metadata
included. With metadata caching on and upstream out of the picture, a
cached request still cost 7 ms for @babel/core, 17 ms for
symfony/console and 155 ms and 134 MB of allocations for typescript, and
an 8 vCPU VM serving cached npm packuments to 20 clients ran the proxy
at 450% CPU for about 1,100 requests a second.
Rewritten documents are now kept in memory, keyed by the ecosystem,
proxy URL, package and a SHA-256 of the raw document, so new bytes from
upstream are rewritten again and nothing is served stale. Requests that
arrive while a document is being rewritten wait for that rewrite rather
than running their own; a waiter leaves when its client does, and the
rewrite still completes and is cached. The denylist is fixed at startup,
so it needs no place in the key. Cooldown filtering depends on the
current time, so with cooldown on the cache is bypassed.
metadata_rewrite_cache_size bounds the cache (default "256MB", least
recently used out first, "0" to rewrite on every request). NewProxy
callers keep the old behaviour unless they set it.
Whole cached requests, measured locally:
@babel/core 7.1 ms -> 0.38 ms, 54,489 -> 119 allocations
typescript 155 ms -> 10.5 ms, 1.37M -> 133 allocations
symfony/console 17.4 ms -> 0.48 ms, 165,909 -> 123 allocations
What remains is reading the raw document from storage and hashing it.
2026-10-03 04:09:15 -07:00
```yaml
metadata_rewrite_cache_size: "256MB" # default; "0" rewrites on every request
```
Or via environment variable: `PROXY_METADATA_REWRITE_CACHE_SIZE=1GB` .
2026-07-07 17:48:36 +01:00
## Upstream HTTP timeout
Protocol handlers use a shared HTTP client for upstream requests such as metadata fetches and pass-through file downloads. `http_timeout` sets that client's per-request timeout. Raise it if slow upstreams or large metadata responses cause `context deadline exceeded` errors.
```yaml
http_timeout: "30s" # default
```
Or via environment variable: `PROXY_HTTP_TIMEOUT=2m` .
fix(server): tune the shared upstream transport defaults (#351)
* fix(server): tune the shared upstream transport defaults
server.serve builds the shared client with safehttp.New, which clones
Go's default transport: MaxIdleConnsPerHost stays 0 (an effective limit
of two idle connections per host) and ResponseHeaderTimeout stays 0.
Handing that client to fetch.NewFetcher via fetch.WithHTTPClient
replaces the fetcher's own defaults of 10 idle connections per host and
a 60-second response-header timeout.
Set both on the shared transport, matching the fetcher defaults: a
second burst of concurrent cache misses to one registry now reuses its
connections instead of re-dialling most of them, and an upstream that
accepts a request but stalls before sending headers is cut off after 60
seconds rather than only by the client's overall timeout.
Tests measure connection reuse across two concurrent bursts against a
TLS upstream that counts accepted connections (Go default: at most two
reused; tuned: all eight) and assert that a stall before headers fails
with the response-header timeout.
Fixes #327
* fix(server): reconcile http_timeout docs and tighten the transport tests
The http_timeout documentation and the config comment said "0" disables
the upstream timeout entirely. With a fixed 60-second
ResponseHeaderTimeout on the shared transport that is no longer the
whole story, so both now say that waiting for response headers stays
bounded independently of the setting.
Test cleanup from review: the 50ms settle between bursts was dead time
(the transport returns a connection to the idle pool before the body's
final Read returns, so burst returning already means the pool is
settled); the maxNewInBurst sentinel became explicit min/max bounds per
case; the stall test dropped the client.Timeout override and the
elapsed-time assertion, which was redundant with the error-text check
in any realistic run and whose failure message misattributed the cause,
and its comment now says plainly that the field assertions pin
production while the behavioural half runs at a lowered timeout.
* test(server): hold each burst at the upstream instead of sleeping
The reuse test kept a burst in flight with a 100ms handler sleep, so a
process stall longer than that between spawning the goroutines and
their dials let a request finish early and hand its connection to a
sibling. Review reproduced this with forced stalls: the default-transport
case then dialled 5 instead of 6 connections.
The handler now answers only once burstSize requests are waiting at the
same time. With HTTP/1.1 pinned that forces every burst onto burstSize
distinct connections regardless of scheduling, and the measured counts
stay exactly 6 new for Go's default and 0 for the tuned transport, also
under the same forced stalls.
2026-09-16 09:11:06 +02:00
Set to `"0"` to disable the timeout entirely (requests then rely only on the server's write timeout). Independently of this setting, the shared transport gives up on an upstream that has not sent response headers within 60 seconds.
2026-07-07 17:48:36 +01:00
2026-10-13 13:11:49 +02:00
## Hosted Packages
Host generic files and share one push path across hosted ecosystems
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
2026-10-20 19:39:14 +02:00
Operators can push a package archive into the proxy under a name and version. The proxy serves it in place of the upstream archive with the same version, or in addition to the upstream's versions. Use it to ship a patched build of a dependency without running a separate registry. Files that aren't packages, such as prebuilt binaries other builds download, can be pushed as [generic files ](#generic-files ).
2026-10-13 13:11:49 +02:00
The `/api/hosted` endpoints are disabled by default. Because a pushed version replaces the upstream one for every client, enabling them requires a token:
```yaml
hosted:
enabled: true
token: "${PROXY_HOSTED_TOKEN}"
max_upload_size: "100MB"
2026-10-13 13:42:34 +02:00
npm_scopes: ["@acme "]
composer_vendors: ["acme"]
2026-10-13 13:11:49 +02:00
```
| Config | Environment | Description |
|--------|-------------|-------------|
| `hosted.enabled` | `PROXY_HOSTED_ENABLED` | Register the `/api/hosted` endpoints (default `false` ) |
| `hosted.token` | `PROXY_HOSTED_TOKEN` | Bearer token required to push and delete; required when enabled, supports `${VAR}` |
Host generic files and share one push path across hosted ecosystems
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
2026-10-20 19:39:14 +02:00
| `hosted.max_upload_size` | `PROXY_HOSTED_MAX_UPLOAD_SIZE` | Maximum size of one pushed archive or file (default `100MB` ) |
2026-10-13 13:42:34 +02:00
| `hosted.npm_scopes` | `PROXY_HOSTED_NPM_SCOPES` | npm scopes served from pushed packages only (comma-separated in the environment) |
| `hosted.composer_vendors` | `PROXY_HOSTED_COMPOSER_VENDORS` | Composer vendors served from pushed packages only |
2026-10-13 13:11:49 +02:00
`hosted.enabled` cannot be combined with `storage.cache_artifacts: false` .
2026-10-25 17:32:22 +01:00
With `hosted.enabled` off, archives pushed earlier stay in storage but are not served as hosted packages: package metadata doesn't list them, `/generic/hosted/` isn't served, and npm and Composer downloads take the normal proxy path, cooldown included. An upstream copy still never replaces a pushed archive.
2026-10-13 13:11:49 +02:00
### Pushing a patched package
2026-10-13 13:27:27 +02:00
1. Build the archive with the original package name and the version you push in its manifest (`package.json` or `composer.json` ), e.g. `request-2.88.3.tgz` .
2026-10-13 13:11:49 +02:00
2. Push it:
```bash
export PROXY_HOSTED_TOKEN=...
2026-10-13 15:20:57 +02:00
proxy hosted push -url https://proxy.example.com npm request 2.88.3 ./request-2.88.3.tgz
2026-10-13 13:11:49 +02:00
```
or with curl:
```bash
curl -X PUT --data-binary @request -2.88.3.tgz \
-H "Authorization: Bearer $PROXY_HOSTED_TOKEN" \
Host generic files and share one push path across hosted ecosystems
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
2026-10-20 19:39:14 +02:00
"https://proxy.example.com/api/hosted/npm/request/2.88.3?sha256=$(sha256sum request-2.88.3.tgz | cut -d' ' -f1)"
2026-10-13 13:11:49 +02:00
```
Host generic files and share one push path across hosted ecosystems
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
2026-10-20 19:39:14 +02:00
The `sha256` query parameter is optional. When it is set, a push whose bytes don't match is refused with 400. `proxy hosted push` always sends it.
2026-10-13 13:27:27 +02:00
3. Install it as usual, e.g. `npm install request@2.88.3` . The version is listed in the package metadata, so ranges like `^2.88.0` resolve to it too.
2026-10-13 15:20:57 +02:00
4. Check what is pushed: `proxy hosted list npm request` or `GET /api/hosted/npm/request` .
5. Remove it again: `proxy hosted delete npm request 2.88.3` or `DELETE /api/hosted/npm/request/2.88.3` . If the upstream has the version, clients get the upstream archive again.
2026-10-13 13:11:49 +02:00
2026-10-21 12:33:43 +02:00
A pushed file is never replaced: pushing the same version again answers 409, also under a differently cased Composer name, before the body is uploaded when the client sends `Expect: 100-continue` (as `proxy hosted push` does). Clients that pinned the file by its hash would otherwise get different bytes under the same version. To push it again, delete it first.
Host generic files and share one push path across hosted ecosystems
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
2026-10-20 19:39:14 +02:00
2026-10-13 13:11:49 +02:00
### Supported ecosystems
Download handlers look archives up by the filename clients request, so a push is stored under that name:
| Ecosystem | Name | Filename |
|-----------|------|----------|
| `npm` | `{name}` or `@{scope}/{name}` | `{name}-{version}.tgz` , without the scope |
| `composer` | `{vendor}/{package}` | `{package}-{version}.zip` |
Host generic files and share one push path across hosted ecosystems
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
2026-10-20 19:39:14 +02:00
| `generic` | `{name}` , one path segment | named in the push path, see [Generic files ](#generic-files ) |
2026-10-13 13:11:49 +02:00
Other ecosystems are not supported yet.
Host generic files and share one push path across hosted ecosystems
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
2026-10-20 19:39:14 +02:00
npm and Composer archives must contain the package manifest: `package.json` in the top directory of an npm tarball, or `composer.json` at the root or in the single top directory of a Composer zip. Its name, and its version when it has one, must match the push. Otherwise the push is refused with 400.
2026-10-13 13:27:27 +02:00
### Package metadata
Pushed versions are listed in the package metadata clients resolve against, with the dependencies from their manifest:
- A pushed version takes the place of an upstream version with the same number.
2026-10-27 16:19:14 +01:00
- Pushed versions are added next to the upstream ones and are never held back by cooldown. npm `dist-tags` stay as the upstream has them, also in `npm dist-tag ls` .
2026-10-13 13:27:27 +02:00
- A package the upstream doesn't know is served from its pushed versions alone. Its npm `latest` tag is the highest pushed version.
- Pushed Composer dev versions (`dev-*` , `*-dev` ) are listed in the `~dev` metadata file.
- npm entries carry the archive's sha512 as `dist.integrity` . Composer entries carry its sha256 as `dist.reference` and leave `shasum` empty.
Metadata of packages with pushed versions is rewritten on every request instead of being cached, so a push or delete shows up on the next request.
2026-10-13 13:42:34 +02:00
### Hosted-only namespaces
A package the upstream doesn't know is served from its pushed versions, but the proxy still asks the upstream about it on every request. If someone later publishes the same name there, their versions are listed next to yours, and a range such as `^1.0.0` can resolve to one of them (dependency confusion).
For private packages:
1. Prefer a namespace you own on the public registry: an npm scope registered on npmjs.com, or a Composer vendor with packages on Packagist. Nobody else can publish under it.
2. List the namespace in `hosted.npm_scopes` or `hosted.composer_vendors` . Its packages are served from pushed versions alone: the upstream is never asked about them, and anything not pushed answers 404.
Other packages, including ones in other scopes and vendors, are proxied as usual.
Host generic files and share one push path across hosted ecosystems
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
2026-10-20 19:39:14 +02:00
### Generic files
Files that aren't packages, such as a prebuilt binary that other builds download and check against a pinned sha256, are pushed to the `generic` ecosystem. They have no manifest; the name, version and filename come from the path, and a version can hold several files:
```bash
proxy hosted push generic tools 1.2.0 ./tool-linux-arm64.tar.gz
# or
curl -X PUT --data-binary @tool -linux-arm64.tar.gz \
-H "Authorization: Bearer $PROXY_HOSTED_TOKEN" \
https://proxy.example.com/api/hosted/generic/tools/1.2.0/tool-linux-arm64.tar.gz
```
They are downloaded without a token from `/generic/hosted/{name}/{version}/{file}` , with the file's sha256 as `ETag` :
```bash
curl -O https://proxy.example.com/generic/hosted/tools/1.2.0/tool-linux-arm64.tar.gz
```
2026-10-20 20:03:06 +02:00
Generic files never come from an upstream: anything not pushed answers 404. While hosting is enabled, `hosted` can't be used as an `upstream.generic` name. `proxy hosted list generic tools` lists all versions and files. `proxy hosted delete generic tools 1.2.0` removes a version with all its files, and `proxy hosted delete generic tools 1.2.0 tool-linux-arm64.tar.gz` removes only that file.
Host generic files and share one push path across hosted ecosystems
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
2026-10-20 19:39:14 +02:00
2026-10-13 13:11:49 +02:00
### Storage and eviction
Pushed archives are stored like cached ones but marked as local. The cache size limit and storage retention never remove them; only a delete does. Scanning, when enabled, checks pushed archives like fetched ones.
2026-04-01 16:14:07 +01:00
## Mirror API
The `/api/mirror` endpoints are disabled by default. Enable them to allow starting mirror jobs via HTTP:
```yaml
mirror_api: true
```
Or via environment variable: `PROXY_MIRROR_API=true` .
When disabled, the endpoints are not registered and return 404.
2026-10-24 07:48:08 +02:00
Start a mirror job with either PURLs or an inline CycloneDX or SPDX JSON document. Requests without `Content-Type: application/json` get 415, so other websites cannot start jobs through a visitor's browser:
2026-09-02 16:56:29 +05:30
```bash
curl -X POST http://localhost:8080/api/mirror \
-H "Content-Type: application/json" \
-d '{"sbom":{"bomFormat":"CycloneDX","components":[{"purl":"pkg:npm/lodash@4 .17.21"}]}}'
```
2026-10-12 00:20:10 +02:00
`platforms` limits which platforms of container images are mirrored, like the command's `--platform` flag:
```bash
curl -X POST http://localhost:8080/api/mirror \
-H "Content-Type: application/json" \
-d '{"purls":["pkg:docker/library/alpine@3 .20"],"platforms":["linux/amd64"]}'
```
2026-10-30 14:59:52 +01:00
`latest` mirrors only the newest version of PURLs without a version, like the command's `--latest` flag:
```bash
curl -X POST http://localhost:8080/api/mirror \
-H "Content-Type: application/json" \
-d '{"purls":["pkg:npm/lodash"],"latest":true}'
```
2026-10-20 11:35:08 +01:00
## Cache Delete API
Deleting cached artifacts is disabled by default. Enable it to register `DELETE /api/cache` and show Delete buttons on the version page of the web UI:
```yaml
cache_delete: true
```
Or via environment variable: `PROXY_CACHE_DELETE=true` .
The endpoint has no authentication. Before enabling it, restrict `/api/cache` and `/ui` to trusted users at your ingress or reverse proxy. The UI sends its deletes to `/ui/api/cache` , so whatever protects `/ui` protects those too.
Delete every cached file of a version, or only the one named by `filename` :
```bash
curl -X DELETE http://localhost:8080/api/cache/maven/com.google.guava:guava/32.1.3-jre
curl -X DELETE 'http://localhost:8080/api/cache/maven/com.google.guava:guava/32.1.3-jre?filename=guava-32.1.3-jre.jar'
```
The name is the package name the UI shows, for example `group:artifact` for Maven. The response lists the deleted files. The next request for a deleted file fetches it from the upstream again. The stored bytes are removed after the same delay as a replaced artifact, so downloads already in progress can finish.
Pushed archives are not touched: a whole-version delete skips them, and naming one answers 409. Delete those through the [hosted API ](#hosted-packages ).
2026-03-19 21:06:02 +00:00
## Mirror Command
The `proxy mirror` command pre-populates the cache from various sources. It accepts the same storage and database flags as `serve` .
2026-10-22 16:00:58 +02:00
Packages are downloaded from the configured upstreams: npm, cargo, gem, golang, hex, pub, maven and nuget from `upstream.npm` , `upstream.cargo_download` , `upstream.gem` , `upstream.go` , `upstream.hex` , `upstream.pub` , `upstream.maven` and `upstream.nuget` . npm packages in `upstream.npm_routes` come from their route and are cached under the route's keys, so clients get them from the cache as they would after an install. Mirror API jobs do the same.
2026-10-25 12:36:36 +01:00
A PyPI version is mirrored with every file `upstream.pypi` lists for it: the sdist and each wheel, checked against the listed sha256 and cached under the name and version the download handler reads from the filename. Add the `file_name` qualifier to mirror one file, for example `pkg:pypi/zope.interface@7.2?file_name=zope.interface-7.2-cp312-cp312-macosx_11_0_arm64.whl` .
2026-10-25 14:25:40 +01:00
A conda version is mirrored with every build anaconda.org lists for it, across all subdirs, and each build is cached the way `/conda/{channel}/{subdir}/{file}` serves it. Give the channel as the namespace or the `channel` qualifier, for example `pkg:conda/conda-forge/numpy@2.1.0` ; a package without one is refused. The `subdir` , `build` and `type` qualifiers select builds, for example `pkg:conda/conda-forge/numpy@2.1.0?subdir=linux-64&type=conda` . Builds with a listed sha256 are checked against it. Files are listed from anaconda.org, so mirroring conda fails while `upstream.conda` points somewhere else.
2026-10-25 14:36:43 +01:00
A Composer release is mirrored with the dist archive `upstream.composer` lists for it, checked against its shasum when one is listed, and cached under the name `/composer/files/...` serves it as. Give the version as the repository lists it, for example `pkg:composer/monolog/monolog@3.8.1` . Branches such as `dev-main` aren't mirrored, since their archive changes with every push, and neither are packages in `upstream.composer_routes` .
2026-10-25 19:37:22 +01:00
A Helm chart is mirrored from a repository in `upstream.helm` , named as the namespace, for example `pkg:helm/prometheus-community/prometheus@27.1.0` with `prometheus-community` configured. The chart is fetched from the first URL the repository's `index.yaml` lists for that version, checked against the listed digest, and cached under the repository and digest `/helm/{name}/charts/...` serves it from. The index is cached as well, so with [`cache_metadata` ](#metadata-caching ) on `helm pull` works while the repository is down. Without a version every version in the index is mirrored. Charts named without a repository, or from one that isn't configured, are refused, and so are charts the index lists in an OCI registry: helm pulls those through `/v2` , so mirror them as `pkg:oci` images.
2026-10-25 19:37:46 +01:00
A Julia package is mirrored with the source tarball `upstream.julia` serves at `/package/{uuid}/{tree-hash}` , cached under the package's name and tree hash as `/julia/package/...` serves it. The PURL needs the `uuid` qualifier, for example `pkg:julia/Example@0.5.5?uuid=7876af07-990d-54b4-ab0e-23690620f79a` . Only packages of the General registry can be mirrored: the uuid and tree hash are read from General on GitHub, also with another `upstream.julia` , since they are the same on every Pkg server serving General. The tarball isn't checked by the proxy, as when serving; Pkg checks the tree hash of what it unpacks, except on Windows. Dependencies, the artifacts in a package's `Artifacts.toml` and the registry aren't mirrored. An offline `Pkg.add` also needs the registry: one installed on the client, or the one the proxy cached when a client last fetched it, which it serves offline with `cache_metadata` on.
2026-10-26 16:43:42 +01:00
A Swift release is mirrored from `upstream.swift` with its source archive, checked against the SHA-256 checksum in the release metadata and cached under the key `/swift/{scope}/{name}/{version}.zip` serves. Give the registry scope as the namespace, for example `pkg:swift/apple/swift-argument-parser@1.5.0` for `apple.swift-argument-parser` , or the source repository, as SBOM tools write it: `pkg:swift/github.com/apple/swift-argument-parser@1.5.0` is looked up through the upstream's `/identifiers` , and the first identity it lists is mirrored. The release metadata, the package's release list and the release's `Package.swift` with its version-specific manifests are stored in the metadata cache too, so `swift package resolve` works with the upstream unreachable. Serving them, and the archive, offline needs `cache_metadata` on the server, since the archive is served with the checksum from the release metadata.
2026-10-25 21:48:59 +01:00
2026-10-30 08:51:35 +00:00
A CRAN version is mirrored as its source package from `upstream.cran` , for example `pkg:cran/glue@1.8.1` , and cached under the file name, so it is served for both `/cran/src/contrib/{file}` and `/cran/src/contrib/Archive/{name}/{file}` , where CRAN moves it once a newer version is published. The current version is fetched from `src/contrib` and checked against the MD5 sum in `src/contrib/PACKAGES.gz` ; older versions are fetched unchecked from `src/contrib/Archive/{name}/` , as the CRAN handler fetches them. Binary packages aren't mirrored, and neither are the `PACKAGES` indexes, so `install.packages` needs the upstream or an index the proxy already cached; installing the tarball by URL works without either.
2026-10-25 19:40:13 +01:00
2026-03-19 21:06:02 +00:00
| Flag | Default | Description |
|------|---------|-------------|
| `--sbom` | | Path to CycloneDX or SPDX SBOM file |
| `--concurrency` | `4` | Number of parallel downloads |
| `--dry-run` | `false` | Show what would be mirrored without downloading |
2026-10-12 00:20:10 +02:00
| `--platform` | | Comma-separated `os/arch[/variant]` list limiting which platforms of container images are mirrored |
2026-10-30 14:59:52 +01:00
| `--latest` | `false` | Mirror only the newest version of PURLs without a version |
2026-03-19 21:06:02 +00:00
| `--config` | | Path to configuration file |
| `--storage-url` | | Storage URL |
| `--database-driver` | | Database driver |
| `--database-path` | | SQLite database file |
| `--database-url` | | PostgreSQL connection URL |
Positional arguments are treated as PURLs:
```bash
proxy mirror pkg:npm/lodash@4 .17.21 pkg:cargo/serde@1 .0.0
```
2026-10-22 15:38:35 +02:00
### Unversioned PURLs
2026-10-27 11:00:10 +01:00
A PURL without a version, here or in a mirror API job, mirrors every version the configured upstream lists, through the same client, so `upstream.auth` and `upstream.allow_private_hosts` apply. npm packages in `upstream.npm_routes` are listed from their route. Listing works for npm, gem, golang, hex (from `upstream.hex_api` ), pub, pypi, maven, nuget, composer, cran, brew and swift. For cargo and conda the upstream doesn't serve the API used for listing, so versions come from the public registry while the upstream is the default one; with another upstream, and for Composer packages in `upstream.composer_routes` or any other ecosystem, give a version. Julia versions are always listed from the General registry.
2026-10-22 15:38:35 +02:00
2026-10-30 14:59:52 +01:00
With `--latest` , or `"latest": true` in a mirror API job, only the latest version is mirrored, as git-pkgs/registries picks it: the version the registry advertises as latest where it has one, such as npm's `dist-tags.latest` , and otherwise the newest by publish time, leaving out yanked, retracted and deprecated versions. PURLs with a version, including those in an SBOM, are mirrored as given. Run it from cron to keep the newest releases of a list of packages cached:
```bash
proxy mirror --latest pkg:npm/lodash pkg:pypi/requests pkg:cargo/serde
```
2026-10-12 00:20:10 +02:00
### Container images
`pkg:docker` and `pkg:oci` PURLs mirror a container image the way a pull does: the manifest or index, the platform manifests, then config and layer blobs. Everything is stored under the keys a pull through `/v2` reads, so a later `docker pull` or containerd pull is served from cache, also when the upstream is unreachable.
```bash
proxy mirror --platform linux/amd64,linux/arm64 pkg:docker/library/alpine@3 .20
proxy mirror 'pkg:docker/containerd/alpine@3 .14.0?repository_url=ghcr.io'
proxy mirror 'pkg:oci/debian@sha256: ...?repository_url=docker.io/library/debian&tag=bookworm'
```
- The version is a tag or a digest. Unversioned image PURLs fail; mirroring all tags is not supported.
- Without `repository_url` the name is routed like a pull without containerd's `ns` parameter: the default OCI registry, repository prefix routes and `upstream/{name}/` . Single-segment Docker Hub names get the `library/` prefix.
- `repository_url` is resolved like containerd's `ns` parameter, so it has to name the default registry, Docker Hub, a configured `upstream.oci` host or a host listed in `upstream.oci_mirrors` . Other hosts are refused.
- `pkg:oci` needs `repository_url` . With `tag=` the tag is mirrored as well; if it now points to another digest, both are mirrored and a warning is logged. `arch=` limits the platforms when `--platform` is not given.
- Without `--platform` every platform of an index is mirrored, including Windows images, which can be large. Attestation manifests of the selected platforms are mirrored too, because Docker fetches them on every pull.
- Non-distributable (foreign) layers are skipped.
- While `scanning.enabled` is set, the command refuses images, because scanners fetch staged layers from the running server. Use the mirror API instead.
2026-10-25 21:21:19 +01:00
### Homebrew bottles
`pkg:brew` PURLs mirror a homebrew/core formula's bottles, which Homebrew publishes as an image on `upstream.homebrew_artifact` . They are mirrored like that image, under the keys brew reads through `HOMEBREW_ARTIFACT_DOMAIN` , so `brew install` is served from cache, also when the upstream is unreachable:
```bash
proxy mirror --platform linux/arm64 pkg:brew/hello@2 .12.3 pkg:brew/openssl%403
```
- A formula name with `@` is written with `%40` , since `@` starts the version: `pkg:brew/openssl%403` is `openssl@3` , and `pkg:brew/openssl@3` is version 3 of `openssl` .
- The formula's JSON API on `upstream.homebrew_api` gives the image tag for the current version, which carries the formula's revision and the bottles' rebuild (`hello` 2.12.3 is tagged `2.12.3-1` ). Only the current version is described there; another version is used as the tag as given, so give the full tag, such as `2.12.2` or `2.12.3-1` , for older bottles.
- A PURL without a version mirrors the current version.
- `--platform` selects bottles like image platforms: `linux/arm64` , `linux/amd64` , or `darwin/arm64` for every macOS release's arm64 bottle. Without it every bottle of the version is mirrored.
- Dependencies aren't mirrored with a formula; list them too. brew also needs the JSON API, which the proxy keeps with `cache_metadata` .
- Casks, source archives and taps other than homebrew/core aren't mirrored.
2026-10-18 19:33:13 +02:00
### Scheduled refresh
Tags such as `alpine:3.20` move. The server can re-mirror them on its own, so the cache follows the tag even when nobody pulls it or the proxy is offline later:
```yaml
mirror:
schedules:
- images: [pkg:docker/library/alpine@3 .20, pkg:docker/library/nginx@stable ]
interval: 6h
platforms: [linux/amd64]
```
| Setting | Env | Description |
|---------|-----|-------------|
| `mirror.schedules[].images` | - | `pkg:docker` or `pkg:oci` PURLs with a tag, resolved like the `mirror` command |
| `mirror.schedules[].interval` | - | How often the images are mirrored, at least `1m` |
| `mirror.schedules[].platforms` | - | `os/arch[/variant]` entries limiting the platforms; empty mirrors all |
Each run checks the tag with a HEAD request. If the tag still points to the cached digest, nothing else is fetched. If it moved, the new manifests and blobs are fetched first and the tag is switched only when all of them are cached, so a failed run leaves the previous image pullable. Images of one schedule are mirrored one after the other.
2026-10-23 19:07:32 +02:00
- The first run starts within ten minutes of startup, later runs every `interval` ±10 %, so restarts and several instances don't hit the registry at once. Instances sharing a PostgreSQL database run each schedule once per interval: the first to reach it records the run, and the others skip it. A run cut off by a shutdown is not recorded, so the next instance to reach the schedule runs it.
2026-10-18 19:33:13 +02:00
- Digest-pinned PURLs never move and are rejected. A `pkg:oci` digest with `tag=` is accepted; the tag is refreshed and a warning is logged when it no longer matches the digest.
- The previous digest stays cached, so pulls by digest keep working, until eviction or retention removes it.
- Docker Hub does not count HEAD requests against its pull limit, but every changed manifest is a pull. Set `platforms` and `upstream.auth` for `registry-1.docker.io` to stay under the anonymous limit.
- Mirrored layers are scanned when `scanning.enabled` is set. A blocked layer fails the run and keeps the previous image.
- Schedules are read at startup; changes need a restart. They do not need `mirror_api` .
2026-01-29 16:33:09 +00:00
## Docker
### SQLite with Local Storage
```bash
docker compose up
```
### PostgreSQL with Local Storage
```bash
docker compose --profile postgres up
```
### PostgreSQL with S3 (MinIO)
```bash
docker compose --profile s3 up
```
## Example Configurations
### Minimal (defaults)
```yaml
listen: ":8080"
```
### Production with PostgreSQL and S3
```yaml
listen: ":8080"
base_url: "https://proxy.example.com"
storage:
url: "s3://my-cache-bucket"
max_size: "100GB"
database:
driver: "postgres"
url: "postgres://proxy:secret@db .example.com:5432/proxy?sslmode=require"
log:
level: "info"
format: "json"
```
### Private npm Registry
```yaml
listen: ":8080"
base_url: "http://localhost:8080"
upstream:
npm: "https://npm.pkg.github.com"
auth:
npm:
type: bearer
token: "${GITHUB_TOKEN}"
```
