pkg-proxy/README.md

# git-pkgs proxy

A caching proxy for package registries. Speeds up package downloads by caching artifacts locally, reducing bandwidth usage and improving reliability.

## Version Cooldown

Most supply chain attacks rely on speed: a malicious version gets published and consumed by automated pipelines within minutes, before anyone notices. The cooldown feature adds a quarantine period to newly published versions. When enabled, the proxy strips versions from metadata responses until they've aged past a configurable threshold.

```yaml
cooldown:
  default: "3d"              # hide versions published less than 3 days ago
  ecosystems:
    npm: "7d"                # npm gets a longer window
    cargo: "0"               # disable for cargo
  packages:
    "pkg:npm/lodash": "0"    # exempt trusted packages
```

A 3-day cooldown means that when `lodash` publishes version `4.18.0`, your builds keep using `4.17.21` until 3 days have passed. If the new release turns out to be compromised, you were never exposed.

Resolution order: package override, then ecosystem override, then global default. This lets you set a conservative default and carve out exceptions for packages where you need faster updates. See [docs/configuration.md](docs/configuration.md) for the full config reference.

## Artifact Scanning

Cooldown only looks at a version's publish timestamp — it never inspects the actual bytes. Artifact scanning closes that gap: when enabled, every artifact is staged into storage and scanned by one or more external services (trivy, ClamAV, Wiz, or anything else that speaks a small HTTP/JSON contract) before it's committed to the cache and served to clients.

```yaml
scanning:
  enabled: true
  signing_key: ${PROXY_SCANNING_SIGNING_KEY}
  scanners:
    - name: clamav
      url: http://clamav-adapter:8080/scan
      mode: block             # a block verdict deletes the artifact and returns 403
    - name: trivy
      url: http://trivy-adapter:8081/scan
      mode: monitor            # findings are logged, never gate caching
      ecosystems: [npm, pypi]
```

The proxy never uploads artifact bytes to a scanner. Each scanner is notified with package metadata plus a short-lived signed URL; the scanner pulls the bytes itself from the proxy's own storage. Scanners run concurrently, and the first `block`-mode scanner to report a verdict of not-allowed wins immediately, canceling the rest. See [docs/configuration.md](docs/configuration.md) for the full config reference and the scanner HTTP contract.

## Supported Registries

| Registry | Language/Platform | Cooldown | Completed |
|----------|-------------------|:--------:|:---------:|
| npm | JavaScript | Yes | ✓ |
| Cargo | Rust | Yes | ✓ |
| RubyGems | Ruby | Yes | ✓ |
| Go proxy | Go | | ✓ |
| Hex | Elixir | Yes* | ✓ |
| pub.dev | Dart | Yes | ✓ |
| PyPI | Python | Yes | ✓ |
| Maven | Java | | ✓ |
| Gradle Build Cache | Java/Kotlin | | ✓ |
| NuGet | .NET | Yes | ✓ |
| Composer | PHP | Yes | ✓ |
| Conan | C/C++ | | ✓ |
| Conda | Python/R | Yes | ✓ |
| CRAN | R | | ✓ |
| Julia | Julia | | ✓ |
| Swift | Swift | | ✓ |
| Container | Docker/OCI | | ✓ |
| Homebrew | macOS/Linux | | ✓ |
| Debian | Debian/Ubuntu | | ✓ |
| RPM | RHEL/Fedora | | ✓ |
| Alpine | Alpine Linux | | ✓ |
| Arch | Arch Linux | | ✓ |
| Chef | Chef | | ✗ |
| Generic | Any | | ✓ |
| Helm | Kubernetes | Yes | ✓ |
| Vagrant | Vagrant | | ✗ |

Cooldown requires publish timestamps in metadata. Registries without a "Yes" in the cooldown column either don't expose timestamps or haven't been wired up yet.

\* Hex cooldown requires disabling registry signature verification (`HEX_NO_VERIFY_REPO_ORIGIN=1`) since the proxy re-encodes the protobuf payload.

## Install

```bash
brew install git-pkgs/git-pkgs/proxy
```

Or download a binary from the [releases page](https://github.com/git-pkgs/proxy/releases).

### Helm

Install the chart from GHCR, setting the public URL that package-manager clients
will use to reach the proxy:

```bash
helm install proxy oci://ghcr.io/git-pkgs/charts/proxy \
  --set config.data.base_url=https://proxy.example.com
```

The default chart deploys one replica backed by a 10 GiB persistent volume,
using SQLite and filesystem artifact storage under `/data`. See
[`deploy/charts/proxy/values.yaml`](deploy/charts/proxy/values.yaml) for ingress,
external database and object-storage configuration options.

## Quick Start

```bash
# Build from source
go build -o proxy ./cmd/proxy

# Run with defaults (listens on :8080)
./proxy

# Run with custom settings
./proxy -listen :3000 -base-url https://proxy.example.com
```

The proxy is now running. Configure your package managers to use it.

## OpenAPI (Swagger)

This repo uses swaggo to generate an OpenAPI spec from annotated handlers.

Generate the spec:

```bash
go install github.com/swaggo/swag/cmd/swag@latest
go generate ./internal/server
```

Generated files are written to `docs/swagger/`.

When the proxy is running, fetch the live spec from:

- `http://localhost:8080/openapi.json`

Or replace `http://localhost:8080` with your configured base URL. This link is also shown on the dashboard.

## Configuring Package Managers

### npm

Create or edit `~/.npmrc`:

```
registry=http://localhost:8080/npm/
```

Or set per-project in `.npmrc`:

```
registry=http://localhost:8080/npm/
```

Or use environment variable:

```bash
npm_config_registry=http://localhost:8080/npm/ npm install
```

`npm audit`, `pnpm audit`, `yarn npm audit` and `npm audit signatures` work
through the proxy: the audit and signing-key endpoints are passed through to the
configured upstream registry, with upstream authentication applied. Advisories
therefore come from upstream's database, not from the proxy's own vulnerability
data, and versions withheld by [cooldown](#version-cooldown) are not excluded
from the report.

To serve private packages through the same registry URL, see [Private package routes](docs/configuration.md#private-package-routes).

### Cargo

Create or edit `~/.cargo/config.toml`:

```toml
[source.crates-io]
replace-with = "proxy"

[source.proxy]
registry = "sparse+http://localhost:8080/cargo/"
```

Or set per-project in `.cargo/config.toml` in your project root.

### RubyGems / Bundler

Set the gem source in your `Gemfile`:

```ruby
source "http://localhost:8080/gem"
```

Or configure globally:

```bash
gem sources --add http://localhost:8080/gem/
bundle config mirror.https://rubygems.org http://localhost:8080/gem
```

### Go modules

Set the GOPROXY environment variable:

```bash
export GOPROXY=http://localhost:8080/go,direct
```

Or in your shell profile for persistence.

Checksum lookups go through the same URL: the proxy serves `sum.golang.org` at
`/go/sumdb/sum.golang.org/`, and the `go` command uses it without further
setup, so clients need no route of their own to `sum.golang.org`. If the
proxy loses `https://sum.golang.org`, it keeps answering until its cached
copy of the database's signed tree is older than `metadata_ttl`; after that
the `go` command contacts the database directly as before. Set
`upstream.go_sumdbs_offline` on networks where clients cannot do that. Other
checksum databases are listed under `upstream.go_sumdbs`, see
[Go checksum database](docs/configuration.md#go-checksum-database).

### Homebrew

Point Homebrew's JSON API and artifact domain at the proxy:

```bash
export HOMEBREW_API_DOMAIN=http://localhost:8080/homebrew
export HOMEBREW_ARTIFACT_DOMAIN=http://localhost:8080
```

The artifact domain proxies manifests and bottle blobs under `/v2/homebrew/core/`. GHCR routing is limited to that repository. Source archives, cask application downloads, custom tap artifacts, and legacy flat-file bottle mirrors use Homebrew's normal fallback URLs. Keep fallback enabled by leaving `HOMEBREW_ARTIFACT_DOMAIN_NO_FALLBACK` unset.

Enable `cache_metadata` or set `PROXY_CACHE_METADATA=true` to retain Homebrew JSON API responses for offline fallback. Bottle blobs and their OCI manifests are cached without this setting.

The upstreams default to `https://formulae.brew.sh/api` for the JSON API and `https://ghcr.io` for artifacts. To chain this proxy to another proxy, configure its Homebrew endpoints as the upstreams:

```yaml
upstream:
  homebrew_api: "https://upstream-proxy.example.com/homebrew"
  homebrew_artifact: "https://upstream-proxy.example.com"
```

The equivalent environment variables are `PROXY_UPSTREAM_HOMEBREW_API` and `PROXY_UPSTREAM_HOMEBREW_ARTIFACT`.

### Hex (Elixir)

Configure in `~/.hex/hex.config`:

```erlang
{default_url, <<"http://localhost:8080/hex">>}.
```

Or set the environment variable:

```bash
export HEX_MIRROR=http://localhost:8080/hex
```

### pub.dev (Dart/Flutter)

Set the PUB_HOSTED_URL environment variable:

```bash
export PUB_HOSTED_URL=http://localhost:8080/pub
```

### PyPI (pip)

Configure pip to use the proxy:

```bash
pip install --index-url http://localhost:8080/pypi/simple/ package_name
```

Or set in `~/.pip/pip.conf`:

```ini
[global]
index-url = http://localhost:8080/pypi/simple/
```

### Maven

Add to your `~/.m2/settings.xml`:

```xml
<settings>
  <mirrors>
    <mirror>
      <id>proxy</id>
      <mirrorOf>central</mirrorOf>
      <url>http://localhost:8080/maven/</url>
    </mirror>
  </mirrors>
</settings>
```

The `/maven/` endpoint uses Maven Central as primary upstream and falls back to the Gradle Plugin Portal for Gradle plugin marker metadata and related artifacts when the primary upstream returns not found.

For Gradle plugin resolution via the same proxy endpoint:

```kotlin
pluginManagement {
  repositories {
    maven(url = "http://localhost:8080/maven/")
  }
}
```

### Gradle HTTP Build Cache

Configure in `settings.gradle(.kts)`:

```kotlin
buildCache {
  local {
    enabled = false
  }
  remote<HttpBuildCache> {
    url = uri("http://localhost:8080/gradle/")
    push = true
  }
}
```

### NuGet

Configure in `nuget.config`:

```xml
<configuration>
  <packageSources>
    <clear />
    <add key="proxy" value="http://localhost:8080/nuget/v3/index.json" />
  </packageSources>
</configuration>
```

Or use the CLI:

```bash
dotnet nuget add source http://localhost:8080/nuget/v3/index.json -n proxy
```

### Composer (PHP)

Configure in `composer.json`:

```json
{
    "repositories": [
        {
            "type": "composer",
            "url": "http://localhost:8080/composer"
        }
    ]
}
```

Or set globally:

```bash
composer config -g repositories.proxy composer http://localhost:8080/composer
```

To serve private packages through the same URL, see [Private package routes](docs/configuration.md#private-package-routes).

### Conan (C/C++)

Add the proxy as a remote:

```bash
conan remote add proxy http://localhost:8080/conan
conan remote disable conancenter
```

Or configure in `~/.conan2/remotes.json`.

### Conda

Configure in `~/.condarc`:

```yaml
channels:
  - http://localhost:8080/conda/main
  - http://localhost:8080/conda/conda-forge
default_channels:
  - http://localhost:8080/conda/main
```

Or set via command:

```bash
conda config --add channels http://localhost:8080/conda/main
```

### CRAN (R)

Set the repository in R:

```r
options(repos = c(CRAN = "http://localhost:8080/cran"))
```

Or in `~/.Rprofile` for persistence:

```r
local({
  r <- getOption("repos")
  r["CRAN"] <- "http://localhost:8080/cran"
  options(repos = r)
})
```

### Julia

Set the Pkg server before starting Julia:

```bash
export JULIA_PKG_SERVER=http://localhost:8080/julia
```

Or inside a running session:

```julia
ENV["JULIA_PKG_SERVER"] = "http://localhost:8080/julia"
using Pkg; Pkg.update()
```

### Swift

Configure the proxy as the default registry for the current Swift package:

```bash
swift package-registry set --allow-insecure-http http://localhost:8080/swift
```

Registry dependencies use their scoped package identifier in `Package.swift`:

```swift
dependencies: [
    .package(id: "apple.swift-argument-parser", from: "1.2.0")
]
```

The proxy supports dependency resolution and source downloads. Publishing with
`swift package-registry publish` is not supported.

### Docker / Container Registry

Configure Docker to use the proxy as a registry mirror in `/etc/docker/daemon.json`:

```json
{
  "registry-mirrors": ["http://localhost:8080"]
}
```

Then restart Docker:

```bash
sudo systemctl restart docker
```

Or pull images directly:

```bash
docker pull localhost:8080/library/nginx:latest
```

The registry is pull-only. BuildKit cache exports (`--cache-to type=registry`)
and other pushes need a separate registry.

SBOMs, signatures and attestations attached to an image are found through the
OCI 1.1 referrers API (`GET /v2/{name}/referrers/{digest}`). `oras discover`
and `notation verify` use it, and so does cosign for its newer signature
bundles. The proxy forwards these requests and caches the returned index with
the same TTL as other metadata, so they keep working offline. It does not
filter by `artifactType`; it returns the full list and the client filters it,
as the spec allows. Registries without the referrers API (GHCR, for example)
answer 404, and clients then look up the `sha256-<digest>` tag instead, which
the proxy serves like any other manifest. The proxy is pull-only, so attaching
new artifacts through it is not supported.

#### containerd (Kubernetes, k3s, nerdctl)

containerd mirrors send the original registry host in an `ns` query
parameter, so one mirror entry can serve Docker Hub and the registries
configured in `upstream.oci` whose URL has no path. Point containerd's CRI
plugin at a hosts directory in `/etc/containerd/config.toml` and restart
containerd. For containerd 2.x:

```toml
version = 3

[plugins."io.containerd.cri.v1.images".registry]
  config_path = "/etc/containerd/certs.d"
```

For containerd 1.x:

```toml
version = 2

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"
```

Then create `/etc/containerd/certs.d/_default/hosts.toml`:

```toml
[host."http://proxy.example.com:8080"]
  capabilities = ["pull", "resolve"]
```

To check that CRI picked up the directory, confirm the setting containerd
runs with and pull through CRI (on k3s use `k3s crictl`):

```bash
containerd config dump | grep config_path
crictl pull docker.io/library/nginx:latest
```

The proxy logs a `container manifest request` line for the pull and, when
`access_log.path` is set, an access-log entry. `ctr images pull --hosts-dir`
is no substitute for this check: it reads the directory itself and succeeds
even while CRI still has no `config_path`.

`docker.io` and the host of `upstream.oci_default` use the default registry;
the host of each `upstream.oci` URL uses that named registry, and so do the
hosts listed for it in `upstream.oci_mirrors`. Pulls for any other registry get
`404 NAME_UNKNOWN`, and containerd falls back to the registry itself. Existing
per-registry `hosts.toml` files that point at `/v2/upstream/{name}` with
`override_path = true` keep working for that upstream's own host and for
Docker Hub. If the upstream is a mirror of the registry the nodes pull from,
say an Artifactory remote of `ghcr.io`, list that registry in
`upstream.oci_mirrors` (`ghcr: ["ghcr.io"]`); otherwise those pulls get the
404 as well. See [docs/configuration.md](docs/configuration.md) for how
hosts are matched and which entry wins when two share a host. k3s generates
the hosts directory itself from `/etc/rancher/k3s/registries.yaml`; `mirrors:
{"*": {endpoint: ["http://proxy.example.com:8080"]}}` produces the same
`_default` entry.

### Helm

Configure each HTTP chart repository with a name, then add the matching proxy
URL to Helm:

```yaml
upstream:
  helm:
    bitnami: "https://charts.bitnami.com/bitnami"
```

```bash
helm repo add bitnami http://localhost:8080/helm/bitnami
helm repo update
helm pull bitnami/nginx
```

The proxy caches `index.yaml` using the normal metadata-cache settings and
caches chart archives after verifying their SHA-256 digest from the index.

For charts stored in an OCI registry, configure a named OCI upstream and add
the reserved `upstream/{name}` prefix to the chart reference:

```yaml
upstream:
  oci:
    ghcr: "https://ghcr.io"
```

```bash
helm pull oci://localhost:8080/upstream/ghcr/owner/charts/mychart --version 1.0.0 --plain-http
```

### Debian / APT

Configure APT to use the proxy in `/etc/apt/sources.list.d/proxy.list`:

```
deb http://localhost:8080/debian stable main contrib
```

Or in deb822 format, in `/etc/apt/sources.list.d/proxy.sources`:

```
Types: deb
URIs: http://localhost:8080/debian
Suites: stable
Components: main contrib
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
```

Replace your existing sources.list entries (on newer releases also
`debian.sources` or `ubuntu.sources` in `/etc/apt/sources.list.d/`), then:

```bash
sudo apt update
```

There is no proxy key to install. The proxy serves the upstream's signed
`InRelease` and `Release` files unchanged, so APT verifies them with the
upstream archive's own key from the distribution's keyring package:
`debian-archive-keyring` on Debian, `ubuntu-keyring` on Ubuntu
(`/usr/share/keyrings/ubuntu-archive-keyring.gpg`). `Signed-By:` in a
`.sources` file, or `[signed-by=...]` in a `.list` line, restricts the source
to that key. For a third-party upstream, use the key its vendor publishes.

The upstream defaults to `http://deb.debian.org/debian`. To proxy a different APT repository (e.g. Ubuntu), set `upstream.debian` in the config file or `PROXY_UPSTREAM_DEBIAN` in the environment:

```yaml
upstream:
  debian: "http://archive.ubuntu.com/ubuntu"
```

A release's security updates are served by a separate archive, so additional
archives are configured under `upstream.debian_repositories` and served at
`/debian/{name}/`:

```yaml
upstream:
  debian: "http://deb.debian.org/debian"
  debian_repositories:
    security: "https://security.debian.org/debian-security"
```

```
deb http://localhost:8080/debian trixie main
deb http://localhost:8080/debian/security trixie-security main
```

`upstream.debian` keeps serving `/debian/pool/…` and `/debian/dists/…`
unchanged. See [Debian archives](docs/configuration.md#debian-archives).

The version denylist blocks recognized `.deb` downloads but leaves signed APT
indexes unchanged. To keep APT from selecting a denied version, configure
[client-side version pinning](docs/configuration.md#apt-version-pinning).

### RPM / Yum / DNF

Configure yum/dnf to use the proxy in `/etc/yum.repos.d/proxy.repo`:

```ini
[proxy-fedora]
name=Fedora via Proxy
baseurl=http://localhost:8080/rpm/releases/$releasever/Everything/$basearch/os/
enabled=1
gpgcheck=0
```

Then:

```bash
sudo dnf clean all
sudo dnf update
```

### Alpine / apk

Point `/etc/apk/repositories` at the proxy. The default repository name
`alpine` proxies the official mirror (`https://dl-cdn.alpinelinux.org/alpine`):

```
http://localhost:8080/apk/alpine/v3.22/main
http://localhost:8080/apk/alpine/v3.22/community
```

Then:

```bash
apk update
```

Repository indexes (v2 `APKINDEX.tar.gz` and v3 `Packages.adb`), detached
signatures, and packages are served byte-for-byte unchanged, so apk's normal
signature verification keeps working. Indexes use the metadata cache
(`metadata_ttl`, stale fallback); `.apk` packages are stored in the shared
artifact cache and remain available when the upstream is unreachable.

To proxy other mirrors or private repositories, configure named upstreams
under `upstream.apk` (this replaces the built-in default; re-add `alpine` if
you still want it):

```yaml
upstream:
  apk:
    alpine: "https://dl-cdn.alpinelinux.org/alpine"
    private: "https://apk.example.com"
```

```
http://localhost:8080/apk/private
```

apk appends the architecture and index filename to each repository line
itself.

### Arch Linux / pacman

Put the proxy at the top of `/etc/pacman.d/mirrorlist`. The default repository
name `archlinux` proxies the official GeoIP mirror
(`https://geo.mirror.pkgbuild.com`):

```
Server = http://localhost:8080/arch/archlinux/$repo/os/$arch
```

Then:

```bash
pacman -Syu
```

Repository databases (`core.db`, `core.files` and their `.tar.*` forms),
database signatures, packages and package signatures are served byte-for-byte
unchanged, so pacman's signature checks keep working. Databases use the
metadata cache (`metadata_ttl`, stale fallback). Each database refresh also
fetches its signature and stores it for exactly that database, so a signature
is only ever served with the database it signs. Enable `cache_metadata` (or set `PROXY_CACHE_METADATA=true`) so
`pacman -Sy` keeps working while the upstream is down; without it databases are
fetched fresh on every request. Packages and their `.sig` files are stored in the shared artifact
cache and remain available when the upstream is unreachable.

The proxy only starts sending a package once it has the whole file, and pacman
gives up on a download that sends nothing for 10 seconds. The fetch keeps
running and the next try is served from the cache. On a slow upstream, set
`DisableDownloadTimeout` in `/etc/pacman.conf` to avoid the failed first try.

`extra.files` is about 50 MB and is read into memory on each `pacman -F`
request. Raise `metadata_max_size` (100 MB by default) for larger custom
repositories.

A custom repository that rebuilds a package under the same version leaves the
old package and signature in the cache. Bump `pkgrel` instead, or remove the
cached version.

#### Arch Linux ARM

Arch Linux ARM has no built-in repository. Add one under `upstream.arch`, using a
regional mirror from <https://archlinuxarm.org/about/mirrors> that serves HTTPS:

```yaml
upstream:
  arch:
    archlinux: "https://geo.mirror.pkgbuild.com"
    archlinuxarm: "https://<region>.mirror.archlinuxarm.org"
```

Setting `upstream.arch` replaces the built-in default, so keep `archlinux` in the map
if you also proxy x86_64 Arch.

Arch Linux ARM mirrors use `$arch/$repo` instead of `$repo/os/$arch`. In
`/etc/pacman.d/mirrorlist`:

```
Server = http://localhost:8080/arch/archlinuxarm/$arch/$repo
```

Avoid `mirror.archlinuxarm.org` as the upstream. It redirects each request to a
changing regional mirror over plain HTTP, so the database and the packages can come
from mirrors with a different sync state.

### GitHub Releases / mise (aqua backend)

Configure named generic upstreams:

```yaml
upstream:
  generic:
    github: "https://github.com"
    github-api: "https://api.github.com"
```

Then rewrite GitHub URLs in mise's settings (`~/.config/mise/config.toml`, mise ≥ 2025.9.3):

```toml
[settings.url_replacements]
"regex:^https://github\\.com/([^/]+)/([^/]+)/releases/download/(.+)" = "http://localhost:8080/generic/github/$1/$2/releases/download/$3"
"regex:^https://api\\.github\\.com/(.*)" = "http://localhost:8080/generic/github-api/$1"
```

Release assets are cached permanently after the first download and keep
installing while GitHub is down. Tag lookups through `api.github.com` are
cached for `metadata_ttl` and served stale during an outage or rate limit.
Commit a `mise.lock` and install with `mise install --locked` so pinned
installs need no API call at all. Add a bearer token for `https://api.github.com`
under `upstream.auth` if the fleet exceeds GitHub's anonymous rate limit.

Maven, mvnd, Node and Go downloads are cached the same way once their hosts
are configured as generic upstreams; their version-pinned paths are built in
(see [configuration](docs/configuration.md)):

```yaml
upstream:
  generic:
    apache: "https://archive.apache.org"
    nodejs: "https://nodejs.org"
    golang: "https://dl.google.com/go"
```

```toml
[settings]
node.mirror_url = "http://localhost:8080/generic/nodejs/dist/"
go.download_mirror = "http://localhost:8080/generic/golang"

[settings.url_replacements]
"https://archive.apache.org/" = "http://localhost:8080/generic/apache/"
```

Other download hosts need their version-pinned paths declared in
`upstream.generic_artifacts`.

## Configuration

The proxy can be configured via:

1. Command line flags (highest priority)
2. Environment variables
3. Configuration file (YAML or JSON)

### Command Line Flags

```
-config string           Path to configuration file
-listen string           Address to listen on (default ":8080")
-base-url string         Public URL of this proxy (default "http://localhost:8080")
-storage-url string      Storage URL (file://, s3://, gs://, azblob://)
-storage-path string     Path to artifact storage directory (deprecated, use -storage-url)
-database-driver string  Database driver: sqlite or postgres (default "sqlite")
-database-path string    Path to SQLite database file (default "./cache/proxy.db")
-database-url string     PostgreSQL connection URL
-log-level string        Log level: debug, info, warn, error (default "info")
-log-format string       Log format: text, json (default "text")
-access-log string       Path to the JSONL access log
-version                 Print version and exit
```

### Environment Variables

```bash
PROXY_LISTEN=:8080
PROXY_BASE_URL=http://localhost:8080
PROXY_UI_URL=http://localhost:8080  # Optional; defaults to PROXY_BASE_URL
PROXY_STORAGE_URL=file:///var/cache/proxy/artifacts
PROXY_DATABASE_DRIVER=sqlite
PROXY_DATABASE_PATH=./cache/proxy.db
PROXY_DATABASE_URL=postgres://user:pass@localhost/proxy?sslmode=disable
PROXY_LOG_LEVEL=info
PROXY_LOG_FORMAT=text
PROXY_ACCESS_LOG_PATH=/var/log/proxy/access.jsonl
PROXY_UPSTREAM_SWIFT=https://tuist.dev/api/registry/swift
```

### Configuration File

```yaml
listen: ":8080"
base_url: "http://localhost:8080"

storage:
  url: "file:///var/cache/proxy/artifacts"
  max_size: "10GB"  # Optional: evict LRU when exceeded
  retention:
    default: "30d"  # Optional: evict artifacts not downloaded for 30 days

database:
  driver: "sqlite"
  path: "/var/lib/proxy/cache.db"

log:
  level: "info"
  format: "text"

access_log:
  path: "/var/log/proxy/access.jsonl"  # Optional JSONL activity log

# Optional: override upstream URLs
upstream:
  npm: "https://registry.npmjs.org"
  cargo: "https://index.crates.io"
  swift: "https://tuist.dev/api/registry/swift"

# Optional: version cooldown (see above)
cooldown:
  default: "3d"
```

See the [configuration reference](docs/configuration.md#upstream-registries) for every upstream key, environment variable, and default URL.

Run with config file:

```bash
./proxy -config /etc/proxy/config.yaml
```

### PostgreSQL

SQLite is the default and works well for single-node deployments. For multi-node setups or if you prefer a managed database, switch to Postgres:

```yaml
database:
  driver: "postgres"
  url: "postgres://user:password@localhost:5432/proxy?sslmode=disable"
```

Or via environment variables:

```bash
PROXY_DATABASE_DRIVER=postgres
PROXY_DATABASE_URL=postgres://user:password@localhost:5432/proxy?sslmode=disable
```

The proxy creates tables automatically on first run.

### S3 Storage

The proxy can store cached artifacts in S3 or any S3-compatible service (MinIO, R2, etc.) instead of the local filesystem.

```yaml
storage:
  url: "s3://my-bucket-name?region=us-east-1"
```

For S3-compatible services like MinIO:

```yaml
storage:
  url: "s3://my-bucket?endpoint=http://localhost:9000&disableSSL=true&s3ForcePathStyle=true"
```

Set credentials via standard AWS environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`).

### Google Cloud Storage

The proxy can store cached artifacts in a GCS bucket using the `gs://` URL scheme.

```yaml
storage:
  url: "gs://my-bucket-name"
```

Authentication uses [Application Default Credentials](https://docs.cloud.google.com/docs/authentication/application-default-credentials), which means no credentials need to be embedded in the config or environment. Supported sources, in order:

- **GKE Workload Identity** — bind the Kubernetes service account running the proxy to a Google service account that has `roles/storage.objectAdmin` on the bucket. The proxy will use the workload's token automatically.
- **Attached service account** on GCE, Cloud Run, Cloud Functions, etc.
- **`GOOGLE_APPLICATION_CREDENTIALS`** environment variable pointing at a service account JSON key file.
- **`gcloud auth application-default login`** for local development.

#### GKE Workload Identity setup

```bash
# 1. Create a Google service account
gcloud iam service-accounts create git-pkgs-proxy \
  --project=PROJECT_ID

# 2. Grant it access to the bucket
gsutil iam ch \
  serviceAccount:git-pkgs-proxy@PROJECT_ID.iam.gserviceaccount.com:objectAdmin \
  gs://my-bucket-name

# 3. Bind the Kubernetes service account to it
gcloud iam service-accounts add-iam-policy-binding \
  git-pkgs-proxy@PROJECT_ID.iam.gserviceaccount.com \
  --role=roles/iam.workloadIdentityUser \
  --member="serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"

# 4. Annotate the Kubernetes service account
kubectl annotate serviceaccount KSA_NAME \
  --namespace=NAMESPACE \
  iam.gke.io/gcp-service-account=git-pkgs-proxy@PROJECT_ID.iam.gserviceaccount.com
```

#### Direct serve (signed URLs) with Workload Identity

When `direct_serve: true` is enabled, the proxy issues HTTP 302 redirects to presigned GCS URLs. Workload Identity provides no private key, so the GCS backend calls the [IAM Credentials `signBlob` API](https://docs.cloud.google.com/iam/docs/reference/credentials/rest/v1/projects.serviceAccounts/signBlob). Grant the service account the token-creator role on itself:

```bash
gcloud iam service-accounts add-iam-policy-binding \
  git-pkgs-proxy@PROJECT_ID.iam.gserviceaccount.com \
  --role=roles/iam.serviceAccountTokenCreator \
  --member="serviceAccount:git-pkgs-proxy@PROJECT_ID.iam.gserviceaccount.com"
```

## CLI Commands

### serve (default)

Start the proxy server. This is the default command if none is specified.

```bash
proxy serve [flags]
proxy [flags]  # same as 'proxy serve'
```

### mirror

Pre-populate the cache from PURLs, SBOM files, or entire registries. Useful for ensuring offline availability or warming the cache before deployments.

```bash
# Mirror specific package versions
proxy mirror pkg:npm/lodash@4.17.21 pkg:cargo/serde@1.0.0

# Mirror all versions of a package
proxy mirror pkg:npm/lodash

# Mirror from a CycloneDX or SPDX SBOM
proxy mirror --sbom sbom.cdx.json

# Preview what would be mirrored
proxy mirror --dry-run pkg:npm/lodash

# Control parallelism
proxy mirror --concurrency 8 pkg:npm/lodash@4.17.21

# Mirror a container image for one platform
proxy mirror --platform linux/amd64 pkg:docker/library/alpine@3.20
```

The mirror command accepts the same storage and database flags as `serve`. Already-cached artifacts are skipped.

A mirror API is also available when the server is running:

```bash
# Start a mirror job
curl -X POST http://localhost:8080/api/mirror \
  -H "Content-Type: application/json" \
  -d '{"purls": ["pkg:npm/lodash@4.17.21"]}'

# Start a mirror job from an inline CycloneDX or SPDX JSON SBOM
curl -X POST http://localhost:8080/api/mirror \
  -H "Content-Type: application/json" \
  -d '{"sbom":{"bomFormat":"CycloneDX","components":[{"purl":"pkg:npm/lodash@4.17.21"}]}}'

# Check job status
curl http://localhost:8080/api/mirror/mirror-1

# Cancel a running job
curl -X DELETE http://localhost:8080/api/mirror/mirror-1
```

### hosted

Push package archives to a running proxy with `hosted.enabled`, list them and delete them. See [Hosted Packages](docs/configuration.md#hosted-packages).

```bash
export PROXY_URL=https://proxy.example.com PROXY_HOSTED_TOKEN=...
proxy hosted push npm request 2.88.3 ./request-2.88.3.tgz
proxy hosted push composer monolog/monolog 3.5.0 ./monolog-3.5.0.zip
proxy hosted list npm request
proxy hosted delete npm request 2.88.3
```

### stats

Show cache statistics without running the server.

```bash
# Text output
proxy stats

# JSON output
proxy stats -json

# Custom database path
proxy stats -database-path /var/lib/proxy/cache.db

# With PostgreSQL
proxy stats -database-driver postgres -database-url postgres://user:pass@localhost/proxy

# Show top 20 most popular packages
proxy stats -popular 20
```

Example output:

```
Cache Statistics
================

Packages:   45
Versions:   128
Artifacts:  128
Total size: 892.4 MB
Total hits: 1547

Packages by ecosystem:
  npm        32
  cargo      13

Most popular packages:
   1. npm/lodash (342 hits, 24.7 KB)
   2. npm/react (198 hits, 89.3 KB)
   3. cargo/serde (156 hits, 234.1 KB)

Recently cached:
  npm/express@4.18.2 (2024-01-15 14:32, 54.2 KB)
  cargo/tokio@1.35.0 (2024-01-15 14:28, 412.8 KB)
```

## API Endpoints

### Registry Protocols

| Endpoint | Description |
|----------|-------------|
| `GET /` | Dashboard (web UI) |
| `GET /health` | Health check and upstream circuit breaker state (JSON; HTTP 200 healthy, 503 unhealthy) |
| `GET /stats` | Cache statistics (JSON) |
| `GET /metrics` | Prometheus metrics |
| `GET /npm/*` | npm registry protocol |
| `POST /npm/-/npm/v1/security/*` | npm/pnpm/Yarn audit endpoints, passed through to upstream |
| `GET /npm/-/npm/v1/keys` | npm registry signing keys, passed through to upstream |
| `GET /cargo/*` | Cargo sparse index protocol |
| `GET /gem/*` | RubyGems protocol |
| `GET /go/*` | Go module proxy protocol |
| `GET /hex/*` | Hex.pm protocol |
| `GET /pub/*` | pub.dev protocol |
| `GET /pypi/*` | PyPI simple/JSON API |
| `GET /maven/*` | Maven repository protocol |
| `GET /nuget/*` | NuGet V3 API |
| `GET /composer/*` | Composer/Packagist protocol |
| `GET /conan/*` | Conan C/C++ protocol |
| `GET /conda/*` | Conda/Anaconda protocol |
| `GET /cran/*` | CRAN (R) protocol |
| `GET /julia/*` | Julia Pkg server protocol |
| `GET /swift/*` | Swift Package Registry v1 protocol |
| `GET /helm/{repository}/*` | HTTP Helm chart repository protocol |
| `GET /homebrew/*` | Homebrew JSON API |
| `GET /v2/*` | OCI/Docker registry protocol |
| `GET /v2/homebrew/core/*` | Homebrew core bottle manifests and blobs from GHCR |
| `GET /apk/{repository}/*` | Alpine APK repository protocol |
| `GET /arch/{repository}/*` | Arch Linux pacman repository protocol |
| `GET /generic/{name}/*` | Generic HTTP download proxy (GitHub release assets, mise/aqua) |
| `GET /debian/*` | Debian/APT repository protocol (main archive) |
| `GET /debian/{repository}/*` | Debian/APT repository protocol (named archive, e.g. security) |
| `GET /rpm/*` | RPM/Yum repository protocol |

### Mirror API

| Endpoint | Description |
|----------|-------------|
| `POST /api/mirror` | Start a mirror job (JSON body with `purls` or an inline `sbom`) |
| `GET /api/mirror/{id}` | Get job status and progress |
| `DELETE /api/mirror/{id}` | Cancel a running job |

### Hosted API

Registered when `hosted.enabled` is set. Pushing and deleting require `Authorization: Bearer <hosted.token>`.

| Endpoint | Description |
|----------|-------------|
| `PUT /api/hosted/{ecosystem}/{name}/{version}` | Push the request body as the archive for a version (npm and composer) |
| `GET /api/hosted/{ecosystem}/{name}` | List the archives pushed for a package |
| `DELETE /api/hosted/{ecosystem}/{name}/{version}` | Delete a pushed version |

### Enrichment API

The proxy provides REST endpoints for package metadata enrichment, vulnerability scanning, and outdated detection.

| Endpoint | Description |
|----------|-------------|
| `GET /api/package/{ecosystem}/{name}` | Get package metadata |
| `GET /api/package/{ecosystem}/{name}/{version}` | Get version metadata with vulnerabilities |
| `GET /api/vulns/{ecosystem}/{name}` | Get all vulnerabilities for a package |
| `GET /api/vulns/{ecosystem}/{name}/{version}` | Get vulnerabilities for a specific version |
| `POST /api/outdated` | Check multiple packages for outdated versions |
| `POST /api/bulk` | Bulk package metadata lookup |

#### Get Package Metadata

```bash
curl http://localhost:8080/api/package/npm/lodash
```

Response:

```json
{
  "ecosystem": "npm",
  "name": "lodash",
  "latest_version": "4.17.21",
  "license": "MIT",
  "license_category": "permissive",
  "description": "Lodash modular utilities",
  "homepage": "https://lodash.com/",
  "repository": "https://github.com/lodash/lodash",
  "registry_url": "https://registry.npmjs.org"
}
```

#### Get Version with Vulnerabilities

```bash
curl http://localhost:8080/api/package/npm/lodash/4.17.0
```

Response:

```json
{
  "package": {
    "ecosystem": "npm",
    "name": "lodash",
    "latest_version": "4.17.21",
    "license": "MIT",
    "license_category": "permissive"
  },
  "version": {
    "ecosystem": "npm",
    "name": "lodash",
    "version": "4.17.0",
    "license": "MIT",
    "published_at": "2016-06-17T03:59:56Z",
    "yanked": false,
    "is_outdated": true
  },
  "vulnerabilities": [
    {
      "id": "GHSA-p6mc-m468-83gw",
      "summary": "Prototype Pollution in lodash",
      "severity": "HIGH",
      "cvss_score": 7.4,
      "fixed_version": "4.17.12"
    }
  ],
  "is_outdated": true,
  "license_category": "permissive"
}
```

#### Check Outdated Packages

```bash
curl -X POST http://localhost:8080/api/outdated \
  -H "Content-Type: application/json" \
  -d '{
    "packages": [
      {"ecosystem": "npm", "name": "lodash", "version": "4.17.0"},
      {"ecosystem": "pypi", "name": "requests", "version": "2.25.0"}
    ]
  }'
```

Response:

```json
{
  "results": [
    {
      "ecosystem": "npm",
      "name": "lodash",
      "version": "4.17.0",
      "latest_version": "4.17.21",
      "is_outdated": true
    },
    {
      "ecosystem": "pypi",
      "name": "requests",
      "version": "2.25.0",
      "latest_version": "2.31.0",
      "is_outdated": true
    }
  ]
}
```

#### Bulk Package Lookup

```bash
curl -X POST http://localhost:8080/api/bulk \
  -H "Content-Type: application/json" \
  -d '{
    "purls": [
      "pkg:npm/lodash@4.17.21",
      "pkg:pypi/requests@2.28.0"
    ]
  }'
```

Response:

```json
{
  "packages": {
    "pkg:npm/lodash": {
      "ecosystem": "npm",
      "name": "lodash",
      "latest_version": "4.17.21",
      "license": "MIT",
      "license_category": "permissive"
    },
    "pkg:pypi/requests": {
      "ecosystem": "pypi",
      "name": "requests",
      "latest_version": "2.31.0",
      "license": "Apache-2.0",
      "license_category": "permissive"
    }
  }
}
```

### Stats Response (HTTP endpoint)

```json
{
  "cached_artifacts": 142,
  "total_size_bytes": 523456789,
  "total_size": "499.2 MB",
  "storage_url": "file:///path/to/cache/artifacts",
  "database_path": "./cache/proxy.db"
}
```

## How It Works

1. Package manager requests package metadata from the proxy
2. Proxy fetches metadata from upstream, rewrites artifact URLs to point at proxy
3. Package manager requests artifact (tarball, crate, etc.)
4. Proxy checks local cache:
   - **Cache hit**: Serve from local storage
   - **Cache miss**: Fetch from upstream, store locally, serve to client
5. Subsequent requests for the same artifact are served from cache

```
┌─────────────┐     ┌─────────┐     ┌──────────┐
│   npm/cargo │────▶│  proxy  │────▶│ upstream │
│   client    │◀────│         │◀────│ registry │
└─────────────┘     └─────────┘     └──────────┘
                         │
                         ▼
                    ┌─────────┐
                    │  cache  │
                    │ storage │
                    └─────────┘
```

## Web Interface

The proxy serves a web UI under `/ui`. No separate frontend build is needed -- templates and assets are embedded in the binary. `GET /` redirects to `/ui/`. The UI is mounted under its own prefix so a reverse proxy can apply different access rules to it than to the package endpoints (for example, requiring auth for `PathPrefix(/ui)` while leaving `/npm`, `/pypi` etc. open to build machines).

- **Dashboard** (`/ui/`) -- cache stats, popular packages, recently cached artifacts, and vulnerability overview.
- **Analytics** (`/ui/analytics`) -- accumulated download size as a ring broken down by ecosystem with the total in the middle, the cache size, artifact, package and version counts, a per-ecosystem table, the vulnerability overview, and a Runtime card mirroring every counter `/metrics` exposes. See [Analytics](#analytics).
- **Install guide** (`/ui/install`) -- per-ecosystem configuration instructions, so you don't have to look them up here.
- **Package browser** (`/ui/packages`) -- browse all cached packages with filtering by ecosystem and sorting by hits, size, name, or vulnerability count.
- **Search** (`/ui/search?q=...`) -- search cached packages by name.
- **Package detail** (`/ui/package/{ecosystem}/{name}`) -- metadata, license, vulnerabilities, and version list for a package. You can select two versions to compare.
- **Version detail** (`/ui/package/{ecosystem}/{name}/{version}`) -- per-version metadata, integrity hash, artifact cache status, and hit counts.
- **Source browser** (`/ui/package/{ecosystem}/{name}/{version}/browse`) -- browse files inside cached archives with syntax highlighting for text files and image previews.
- **Version diff** (`/ui/package/{ecosystem}/{name}/compare/{v1}...{v2}`) -- side-by-side diff of two cached versions showing added, removed, and changed files.

## Monitoring

The proxy exposes Prometheus metrics at `GET /metrics`. All metric names are prefixed with `proxy_`.

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `proxy_requests_total` | counter | `ecosystem`, `status` | Proxy responses by package ecosystem and HTTP status |
| `proxy_request_duration_seconds` | histogram | `ecosystem`, `status` | Proxy request duration |
| `proxy_cache_hits_total` | counter | `ecosystem` | Cache hits |
| `proxy_cache_misses_total` | counter | `ecosystem` | Cache misses |
| `proxy_cache_size_bytes` | gauge | | Total size of cached artifacts |
| `proxy_cached_artifacts_total` | gauge | | Number of cached artifacts |
| `proxy_artifacts_evicted_total` | counter | `reason`, `ecosystem` | Artifacts evicted by the size limit (`lru`) or by age (`retention`) |
| `proxy_upstream_fetch_duration_seconds` | histogram | `ecosystem` | Time spent fetching from upstream |
| `proxy_upstream_errors_total` | counter | `ecosystem`, `error_type` | Upstream fetch failures |
| `proxy_storage_operation_duration_seconds` | histogram | `operation` | Storage read/write latency |
| `proxy_storage_errors_total` | counter | `operation` | Storage read/write failures |
| `proxy_active_requests` | gauge | | In-flight requests |
| `proxy_health_probe_failures_total` | counter | `step` | Storage health probe failures by failing step (`write`, `size`, `read`, `verify`, `delete`). |
| `proxy_circuit_breaker_state` | gauge | `registry` | Artifact-fetch circuit breaker state per upstream registry (0 closed, 2 open). Published once that registry's breaker has tripped. |
| `proxy_circuit_breaker_trips_total` | counter | `registry` | Circuit breaker trips per upstream registry. |
| `proxy_upstream_rate_limit_holds_total` | counter | `host` | Holds placed on an upstream host after it rate limited the proxy with HTTP 429, or 503 with a `Retry-After`. See `upstream_holds` under [Health Check](#health-check). |
| `proxy_ecosystem_downloaded_bytes` | gauge | `ecosystem` | Accumulated bytes served from cache: cache hits multiplied by the artifact size they served. |
| `proxy_ecosystem_artifact_downloads` | gauge | `ecosystem` | Accumulated artifact downloads served from cache. |
| `proxy_ecosystem_cache_size_bytes` | gauge | `ecosystem` | Size of cached artifacts per ecosystem. |
| `proxy_ecosystem_cached_artifacts` | gauge | `ecosystem` | Number of cached artifacts per ecosystem. |
| `proxy_ecosystem_packages` | gauge | `ecosystem` | Known packages per ecosystem. |
| `proxy_ecosystem_versions` | gauge | `ecosystem` | Known package versions per ecosystem. |
| `proxy_response_bytes_total` | counter | `ecosystem` | Response body bytes written to clients. Route-labelled, see the label caveat below. |
| `proxy_client_requests_total` | counter | `client` | Requests by client tool, from the User-Agent. |
| `proxy_client_response_bytes_total` | counter | `client` | Response bytes by client tool. |

The `ecosystem` label on `proxy_requests_total` and `proxy_request_duration_seconds` is the mounted route a request arrived on, not the ecosystem recorded against the package it served: `/gem` reports as `rubygems`, `/go` as `golang`, `/composer` as `packagist`, `/apk` as `alpine` and `/v2` as `oci`. Anything outside a package route -- the UI, `/health`, `/metrics`, `/stats` -- reports as `other`, and so did `/apk`, `/helm`, `/homebrew`, `/generic` and `/swift` before they were listed; traffic on those five routes now appears under its own name instead.

Cache size, artifact count and the per-ecosystem gauges are refreshed every 60 seconds, from a single pass over the database. Circuit breaker state is read from the fetcher on each scrape of `/metrics` and each `/health` request, so `proxy_circuit_breaker_trips_total` counts the trips visible between those reads — a breaker that opens and recovers entirely between two scrapes is not counted. The remaining metrics update on each request.

The breaker metrics carry one series per upstream host, but only for hosts whose breaker has tripped at least once since startup. A breaker is created per host the proxy fetches artifacts from, and for some ecosystems that host comes from upstream metadata rather than from configuration (composer takes it from a package's `dist.url`, helm from the chart URLs in `index.yaml`), so publishing every host would let upstream content grow the series count for the lifetime of the process. Once a host has tripped it keeps reporting, so a recovery still shows up as a transition to 0 rather than as a series that vanishes. `/health` is not a persistent time series and lists every breaker, tripped or not.

The `registry` label is the host of the URL the artifact was fetched from. Because that URL can come from upstream metadata, it is not always one a host can be read off — a signed `dist.url` that fails to parse, for instance — and such a breaker is labelled `hostless-url-<digest>` instead, where the digest is keyed by a value drawn fresh at startup. Neither `/metrics` nor `/health` requires authentication, so a fetch URL is never published as a label or a key; the digest identifies the breaker for as long as the process runs without revealing the URL behind it or letting a chosen URL be matched against it.

Alert on `proxy_circuit_breaker_state == 2` sustained for more than a few minutes: while a breaker is open, artifact downloads for that upstream fail with HTTP 502 on every cache miss, and only a single probe request per backoff interval reaches the upstream. Cached artifacts keep serving, and so does metadata for the same ecosystem (metadata does not go through the circuit breaker), so installs fail in a way that looks like a partial upstream outage.

#### Accumulated download size

`proxy_ecosystem_downloaded_bytes` is, for every cached artifact, the number of times it was served multiplied by its size. It answers "how much traffic has this proxy actually carried", which is the number that matters when sizing egress or justifying the cache. Two properties are worth knowing before alerting on it.

**It counts cache hits, not upstream fetches.** The request that first pulls an artifact through the proxy is a miss and is not counted; only later hits are. So the accumulated total is also the upstream bandwidth the cache has saved, not the total bytes the proxy has ever sent.

**Eviction removes history.** Evicting an artifact clears its size, so its past hits drop out of the total. That is why these are gauges rather than counters, and why the figure can step downwards. Chart them with `max_over_time` rather than `increase`, and read a drop after an eviction sweep as expected rather than as data loss.

Nothing at the schema level ties `artifacts.version_purl` to a version row, so a cached artifact can end up with no ecosystem to attribute it to. Those are reported under the ecosystem `unattributed` rather than dropped, which keeps the per-ecosystem figures adding up to `proxy_cache_size_bytes` and `proxy_cached_artifacts_total`. A non-zero `unattributed` means the database holds artifact rows whose version or package rows have gone missing.

The `ecosystem` label on these six is taken from the package record and normalized, so aliases collapse: a database carrying both `gem` and `rubygems` rows -- the proxy writes the former, git-pkgs the latter -- reports one `rubygems` series with the two summed.

### Analytics

`/ui/analytics` reports the accumulated download size as a ring broken down by ecosystem, the cache figures from the dashboard, a per-ecosystem table, the vulnerability overview, and a **Runtime** card covering every remaining metric `/metrics` exposes.

The ring shows at most six slices, because part-to-whole stops being readable past that. When more ecosystems are active the smallest are folded into a single "Other" slice; the table below lists every one of them, so nothing is hidden, only summarised.

#### No history is kept

The proxy stores no time series. The page reads the database and the in-process metric registry at request time and reports current state; there is nowhere for it to read yesterday's figures from, and nothing is written for tomorrow. That splits the figures in two, and the page says which is which.

**Database-derived figures survive a restart.** Download volume, cache size and the package, version and artifact counts come from the `artifacts`, `packages` and `versions` tables, so they are as durable as the database.

**Registry-derived figures do not.** Everything in the Runtime card -- request counts and latencies, cache hit rate, upstream and storage errors, circuit breaker state, scan results -- lives only in this process's Prometheus registry and starts from zero on restart. A small number there next to a large one above just means the proxy started recently.

For history, trends and alerting, scrape `/metrics` with Prometheus. That is the intended split: the UI answers "what is true now", Prometheus answers "what happened".

The same figures are available as JSON from `GET /stats`, which reports `downloaded_bytes`, `downloads` and an `ecosystems` array carrying the per-ecosystem breakdown, served from the same 60-second snapshot the page and the gauges read. When the aggregation fails with no snapshot to fall back on, the response carries `stats_unavailable: true` rather than passing zeros off as a count -- the endpoint keeps answering with the artifact count and cache size either way.

#### Three ecosystem label sets

`ecosystem` means three slightly different things across `/metrics`, and queries that join across them need to know which.

**From the package record, normalized.** The six `proxy_ecosystem_*` gauges, `proxy_cache_hits_total`, `proxy_cache_misses_total`, `proxy_artifacts_evicted_total`, `proxy_integrity_failures_total` and the scan metrics. Aliases collapse here: `gem` reads as `rubygems`, `composer` as `packagist`, `go` as `golang`.

**From the request path.** `proxy_requests_total`, `proxy_request_duration_seconds` and `proxy_response_bytes_total`. The names mostly coincide with the normalized ones -- these also report `rubygems`, `packagist` and `golang` -- but the Debian route reports `debian` where the package record says `deb`, and any path that is not a package endpoint reports `other`, which corresponds to no ecosystem at all.

**From the handler's own name.** `proxy_upstream_fetch_duration_seconds` and `proxy_upstream_errors_total`, which report `composer`, `gem` and `go` where the other two sets report `packagist`, `rubygems` and `golang`. These are published series and are deliberately left as they are; renaming them would break existing queries and alerts.

### Request sources

Package managers do not say who invoked them. A request from `pip` or `go` carries a `Host`, an `Accept` and a `User-Agent` -- no `Referer`, no originating URL, nothing naming a repository, pipeline or job. Whatever identity you want has to come from something on the wire, so the proxy attributes requests by the two things always present.

**Address** -- the TCP peer, or the leftmost `X-Forwarded-For` entry when `trust_forwarded_for` is enabled. Enable that only behind a load balancer or ingress that sets the header; any client can send it, so in front of one it lets a caller forge its own attribution and, by cycling synthetic addresses, fill the table and push every genuine caller into the overflow row. The totals stay correct; the attribution is what is lost.

**Client** -- the tool, taken from the leading User-Agent token: `pip`, `npm`, `go`, `docker`, `apt`, `curl` and so on. Anything unrecognised reports as `other`.

Set `ui_request_sources: true` and both appear on `/ui/analytics` under **Runtime -> Request sources**, as a table of the busiest callers by bytes downloaded plus a per-tool breakdown. The table is in-memory and process-lifetime, like the rest of that card. It tracks 200 callers, evicting the least recently seen once full, and summarises everything it is not showing individually -- both evicted callers and those ranked below the display limit -- in a single "other callers" row, so the rows always add up to the totals above them.

**The flag defaults to off because the page is not authenticated.** `/ui` carries no auth of its own -- it is mounted under its own prefix so a reverse proxy *can* gate it separately, as [Behind a Reverse Proxy](#behind-a-reverse-proxy) describes, but nothing makes you -- and until now it exposed only package data. The sources table changes what is on offer: anyone who can reach the proxy can read the addresses of your build fleet, which tool each runs, and how much each pulled. Turn it on once `/ui` is gated, or leave it off and read the same detail from the access log.

**What this can and cannot tell you.** How much an address gives you depends entirely on your network. A fleet of build machines with stable addresses attributes cleanly. Containerised CI usually does not: with Docker or Kubernetes executors every job gets an ephemeral address, and egress is commonly NAT'd behind one gateway, so you get runner-node or gateway granularity, not per-project. If you need per-project attribution the caller has to send something naming itself -- a basic-auth username, or a per-project base URL -- which the proxy does not currently read. Say so and it can be added.

**Why addresses are not Prometheus labels.** Client tool names are exported as `proxy_client_requests_total{client}` because they come from a closed set. Addresses are not exported at all: the caller set is unbounded and outside the proxy's control, and every new address would create a time series that lives forever in your TSDB. The same goes for anything job-scoped -- a pipeline ID must never become a label. Per-address and per-request detail belongs in the access log, which records `remote_ip`, `user_agent`, `client`, `ecosystem` and `bytes` on every line as JSONL, ready for `jq`, Loki or whatever you ship logs to.

### Failure details

The failure counts under **Runtime** on `/ui/analytics` -- upstream errors, storage errors, integrity failures, health probe failures and scan errors -- say how often something failed, not what. Set `ui_failure_details: true` and each count gets a **Latest** list underneath it, newest first: the package URL and filename, the upstream URL or storage path, the error, and how long ago it happened.

The list keeps the last 20 failures per row, in memory only, so it starts empty when the proxy starts, like the counts above it. Credentials, query strings and fragments are stripped from every URL before it is kept, including URLs inside error messages, since that is where registry tokens and signed-URL signatures live.

The flag defaults to off for the same reason as `ui_request_sources`: package names and upstream hosts say what your users depend on, and `/ui` is not authenticated. Package names and URLs are never Prometheus labels; the same detail is in the proxy's log.

### Grafana dashboard

A ready-made dashboard lives at [`deploy/grafana/git-pkgs-proxy.json`](deploy/grafana/git-pkgs-proxy.json). Import it via **Dashboards -> New -> Import** and pick your Prometheus data source when prompted; it has no hardcoded data source UID.

It carries three ecosystem filters rather than one, because `ecosystem` means three different things across `/metrics` -- see the label sets above. **Ecosystem** filters the database-derived gauges, **Route** the request-path counters, and **Upstream** the two upstream fetch metrics. All three are query variables, so they populate from whatever labels your proxy is actually reporting; a panel is on the one its metric belongs to, and the panel descriptions say which.

### Health Check

`/health` returns a structured JSON report of subsystem health. HTTP 200 if all checks pass; 503 if any fail.

```json
{
  "status": "ok",
  "checks": {
    "database": {"status": "ok"},
    "storage":  {"status": "ok"}
  },
  "circuit_breakers": {
    "registry.npmjs.org": "closed",
    "static.crates.io":   "open"
  },
  "upstream_holds": {
    "repo1.maven.org": "2026-10-14T11:15:29Z"
  }
}
```

Failing checks include an `"error"` field. Storage failures also include a `"step"` field identifying which probe step failed (`write`, `size`, `read`, `verify`, `delete`). When the database check fails, the storage entry reports `{"status": "skipped"}` so the response always carries the same key set.

`circuit_breakers` reports the state of each upstream's artifact-fetch circuit breaker (`"open"` or `"closed"`), keyed by upstream host — or by the `hostless-url-<digest>` placeholder described under [Monitoring](#monitoring) where the fetch URL has no host to read. The key is omitted until the proxy has fetched an artifact from at least one upstream, and a host appears only once a breaker has been created for it. Breakers trip after repeated upstream failures and retry the upstream after an exponential backoff. While one is open, artifact downloads for that host return HTTP 502 on a cache miss without contacting the upstream; already-cached artifacts are still served from storage, since the cache is checked before the fetcher. A breaker is reported as `"open"` throughout its backoff, including the half-open window in which it admits one probe request to test recovery. Breaker state is per process and in memory, so a restart clears it, but a restart is not needed for recovery: the backoff keeps retrying for as long as the breaker is open, so it closes on its own once the upstream serves again.

`upstream_holds` lists upstream hosts that answered the proxy with HTTP 429, or 503 with a `Retry-After`, and the time until which the proxy sends them nothing. Every request to a held host, metadata and artifacts alike, fails locally for the duration the upstream asked for (capped at one hour), or for one minute doubling up to fifteen when the upstream gave no `Retry-After`. Cached metadata is served stale during a hold, and anything uncached returns HTTP 503 with the remaining `Retry-After` so a client can pace itself too. Metadata requests do not go through the circuit breaker, so without the hold a rate-limited upstream would keep receiving one request per cache miss, which is exactly the traffic it is asking the proxy to stop sending. The first non-limited response after a hold ends clears it, and `proxy_upstream_rate_limit_holds_total` counts the holds placed. The key is omitted while no host is held.

An open breaker does **not** set `status` to `"error"` or change the HTTP status code: it reports a specific upstream refusing to serve, not this proxy being unfit to receive traffic, and failing the readiness probe over one unhealthy upstream would pull the pod out of rotation for every other ecosystem too. Use `proxy_circuit_breaker_state` for alerting on it.

Storage probe results are cached for `health.storage_probe_interval` (default 30s) to bound the cost of probing remote backends. A probe holds an internal mutex for up to 10 seconds (the hardcoded per-probe timeout), so `/health` is intended as a Kubernetes **readiness** probe rather than a liveness probe — a slow S3 round-trip should pull the pod from rotation, not restart it.

Scrape config for Prometheus:

```yaml
scrape_configs:
  - job_name: git-pkgs-proxy
    static_configs:
      - targets: ["localhost:8080"]
```

## Production Deployment

### Systemd Service

Create `/etc/systemd/system/proxy.service`:

```ini
[Unit]
Description=git-pkgs proxy
After=network.target

[Service]
Type=simple
User=proxy
ExecStart=/usr/local/bin/proxy -config /etc/proxy/config.yaml
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
```

Enable and start:

```bash
sudo systemctl enable proxy
sudo systemctl start proxy
```

### Docker

A Dockerfile is included in the repo. Build and run:

```bash
docker build -t proxy .
docker run -p 8080:8080 -v proxy-data:/data proxy
```

With Postgres and S3:

```bash
docker run -p 8080:8080 \
  -e PROXY_DATABASE_DRIVER=postgres \
  -e PROXY_DATABASE_URL=postgres://user:pass@db:5432/proxy \
  -e PROXY_STORAGE_URL=s3://my-bucket?region=us-east-1 \
  -e AWS_ACCESS_KEY_ID=... \
  -e AWS_SECRET_ACCESS_KEY=... \
  proxy
```

### Behind a Reverse Proxy

When running behind nginx, Apache, or another reverse proxy, set `base_url` to your public URL:

```yaml
base_url: "https://proxy.example.com"
```

If the UI is reached on a different hostname than the package endpoints — for example, the UI exposed publicly on a domain while build machines hit a Docker network alias — set `ui_base_url` separately. `base_url` is the URL package managers and metadata rewriting use; `ui_base_url` is the URL advertised to humans visiting the web UI (canonical/`og:url` tags and the install guide banner):

```yaml
base_url: "http://pkg-proxy:8080"        # internal alias for build machines
ui_base_url: "https://proxy.example.com/ui"  # public UI URL
```

When unset, `ui_base_url` defaults to `base_url`.

> **Warning:** the proxy serves the UI and package endpoints on the same listener. Setting `ui_base_url` only changes what URL the UI advertises to humans; it does not stop package endpoints from being reachable on the same hostname and port. When fronting the proxy with a public reverse proxy, restrict the public route to `PathPrefix(/ui)` (or your proxy's equivalent), otherwise `/npm`, `/pypi`, and the other package endpoints stay exposed alongside the UI.

nginx example, restricting the public host to the UI while leaving package endpoints reachable only on the internal listener:

```nginx
server {
    listen 443 ssl;
    server_name proxy.example.com;

    location /ui/ {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_buffering off;
    }

    location / {
        return 404;
    }
}
```

Traefik example using `PathPrefix(/ui)` so the public router only matches UI traffic:

```yaml
labels:
  traefik.enable: "true"
  traefik.http.services.pkg-proxy.loadbalancer.server.port: "8080"
  traefik.http.routers.pkg-proxy.rule: "Host(`proxy.example.com`) && PathPrefix(`/ui`)"
  traefik.http.routers.pkg-proxy.entrypoints: "websecure"
```

## Cache Management

The proxy stores artifacts in the configured storage directory with this structure:

```
cache/artifacts/
├── npm/
│   └── lodash/
│       └── 4.17.21/
│           └── lodash-4.17.21.tgz
├── cargo/
│   └── serde/
│       └── 1.0.193/
│           └── serde-1.0.193.crate
├── oci/
│   └── library/nginx/
│       └── sha256:abc123.../
│           └── sha256:abc123...
├── deb/
│   └── nginx/
│       └── 1.18.0-6/
│           └── nginx_1.18.0-6_amd64.deb
└── rpm/
    └── nginx/
        └── 1.24.0-1.fc39/
            └── nginx-1.24.0-1.fc39.x86_64.rpm
```

Cache metadata is stored in SQLite (default) or PostgreSQL. To clear a local cache:

```bash
rm -rf ./cache/artifacts/*
rm ./cache/proxy.db
```

The proxy will recreate the database on next start.

## Building from Source

Requirements:

- Go (the project version is declared in `go.mod`)

```bash
git clone https://github.com/git-pkgs/proxy.git
cd proxy
go build -o proxy ./cmd/proxy
```

Run tests:

```bash
go test ./...
```

## License

[GPL-3.0-or-later](LICENSE).