mirror of
https://github.com/git-pkgs/proxy.git
synced 2026-07-06 14:23:20 -04:00
Adds UIBaseURL (env PROXY_UI_URL), advertised separately from BaseURL for deployments where the UI is reached on a public domain while build machines hit a Docker network alias for the package endpoints. Defaults to BaseURL when unset. Consumers: - Logged alongside base_url at startup. - <link rel="canonical"> and og:url / og:title / og:site_name in every UI page, omitted when UIBaseURL is empty. - Banner on the install guide when UIBaseURL differs from BaseURL, clarifying that the URLs in the snippets are the package endpoint and that the UI itself lives elsewhere. Docs call out that the proxy serves UI and package endpoints on the same listener, so reverse proxies fronting the UI publicly must restrict the public route to PathPrefix(/ui) to avoid exposing /npm, /pypi, etc. nginx and Traefik examples both show the path-split pattern.
1036 lines
26 KiB
Markdown
1036 lines
26 KiB
Markdown
# git-pkgs proxy
|
|
|
|
A caching proxy for package registries. Speeds up package downloads by caching artifacts locally, reducing bandwidth usage and improving reliability.
|
|
|
|
## Version Cooldown
|
|
|
|
Most supply chain attacks rely on speed: a malicious version gets published and consumed by automated pipelines within minutes, before anyone notices. The cooldown feature adds a quarantine period to newly published versions. When enabled, the proxy strips versions from metadata responses until they've aged past a configurable threshold.
|
|
|
|
```yaml
|
|
cooldown:
|
|
default: "3d" # hide versions published less than 3 days ago
|
|
ecosystems:
|
|
npm: "7d" # npm gets a longer window
|
|
cargo: "0" # disable for cargo
|
|
packages:
|
|
"pkg:npm/lodash": "0" # exempt trusted packages
|
|
```
|
|
|
|
A 3-day cooldown means that when `lodash` publishes version `4.18.0`, your builds keep using `4.17.21` until 3 days have passed. If the new release turns out to be compromised, you were never exposed.
|
|
|
|
Resolution order: package override, then ecosystem override, then global default. This lets you set a conservative default and carve out exceptions for packages where you need faster updates. See [docs/configuration.md](docs/configuration.md) for the full config reference.
|
|
|
|
## Supported Registries
|
|
|
|
| Registry | Language/Platform | Cooldown | Completed |
|
|
|----------|-------------------|:--------:|:---------:|
|
|
| npm | JavaScript | Yes | ✓ |
|
|
| Cargo | Rust | Yes | ✓ |
|
|
| RubyGems | Ruby | Yes | ✓ |
|
|
| Go proxy | Go | | ✓ |
|
|
| Hex | Elixir | Yes* | ✓ |
|
|
| pub.dev | Dart | Yes | ✓ |
|
|
| PyPI | Python | Yes | ✓ |
|
|
| Maven | Java | | ✓ |
|
|
| Gradle Build Cache | Java/Kotlin | | ✓ |
|
|
| NuGet | .NET | Yes | ✓ |
|
|
| Composer | PHP | Yes | ✓ |
|
|
| Conan | C/C++ | | ✓ |
|
|
| Conda | Python/R | Yes | ✓ |
|
|
| CRAN | R | | ✓ |
|
|
| Julia | Julia | | ✓ |
|
|
| Container | Docker/OCI | | ✓ |
|
|
| Debian | Debian/Ubuntu | | ✓ |
|
|
| RPM | RHEL/Fedora | | ✓ |
|
|
| Alpine | Alpine Linux | | ✗ |
|
|
| Arch | Arch Linux | | ✗ |
|
|
| Chef | Chef | | ✗ |
|
|
| Generic | Any | | ✗ |
|
|
| Helm | Kubernetes | | ✗ |
|
|
| Swift | Swift | | ✗ |
|
|
| Vagrant | Vagrant | | ✗ |
|
|
|
|
Cooldown requires publish timestamps in metadata. Registries without a "Yes" in the cooldown column either don't expose timestamps or haven't been wired up yet.
|
|
|
|
\* Hex cooldown requires disabling registry signature verification (`HEX_NO_VERIFY_REPO_ORIGIN=1`) since the proxy re-encodes the protobuf payload.
|
|
|
|
## Install
|
|
|
|
```bash
|
|
brew install git-pkgs/git-pkgs/proxy
|
|
```
|
|
|
|
Or download a binary from the [releases page](https://github.com/git-pkgs/proxy/releases).
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Build from source
|
|
go build -o proxy ./cmd/proxy
|
|
|
|
# Run with defaults (listens on :8080)
|
|
./proxy
|
|
|
|
# Run with custom settings
|
|
./proxy -listen :3000 -base-url https://proxy.example.com
|
|
```
|
|
|
|
The proxy is now running. Configure your package managers to use it.
|
|
|
|
## OpenAPI (Swagger)
|
|
|
|
This repo uses swaggo to generate an OpenAPI spec from annotated handlers.
|
|
|
|
Generate the spec:
|
|
|
|
```bash
|
|
go install github.com/swaggo/swag/cmd/swag@latest
|
|
go generate ./internal/server
|
|
```
|
|
|
|
Generated files are written to `docs/swagger/`.
|
|
|
|
When the proxy is running, fetch the live spec from:
|
|
|
|
- `http://localhost:8080/openapi.json`
|
|
|
|
Or replace `http://localhost:8080` with your configured base URL. This link is also shown on the dashboard.
|
|
|
|
## Configuring Package Managers
|
|
|
|
### npm
|
|
|
|
Create or edit `~/.npmrc`:
|
|
|
|
```
|
|
registry=http://localhost:8080/npm/
|
|
```
|
|
|
|
Or set per-project in `.npmrc`:
|
|
|
|
```
|
|
registry=http://localhost:8080/npm/
|
|
```
|
|
|
|
Or use environment variable:
|
|
|
|
```bash
|
|
npm_config_registry=http://localhost:8080/npm/ npm install
|
|
```
|
|
|
|
### Cargo
|
|
|
|
Create or edit `~/.cargo/config.toml`:
|
|
|
|
```toml
|
|
[source.crates-io]
|
|
replace-with = "proxy"
|
|
|
|
[source.proxy]
|
|
registry = "sparse+http://localhost:8080/cargo/"
|
|
```
|
|
|
|
Or set per-project in `.cargo/config.toml` in your project root.
|
|
|
|
### RubyGems / Bundler
|
|
|
|
Set the gem source in your `Gemfile`:
|
|
|
|
```ruby
|
|
source "http://localhost:8080/gem"
|
|
```
|
|
|
|
Or configure globally:
|
|
|
|
```bash
|
|
gem sources --add http://localhost:8080/gem/
|
|
bundle config mirror.https://rubygems.org http://localhost:8080/gem
|
|
```
|
|
|
|
### Go modules
|
|
|
|
Set the GOPROXY environment variable:
|
|
|
|
```bash
|
|
export GOPROXY=http://localhost:8080/go,direct
|
|
```
|
|
|
|
Or in your shell profile for persistence.
|
|
|
|
### Hex (Elixir)
|
|
|
|
Configure in `~/.hex/hex.config`:
|
|
|
|
```erlang
|
|
{default_url, <<"http://localhost:8080/hex">>}.
|
|
```
|
|
|
|
Or set the environment variable:
|
|
|
|
```bash
|
|
export HEX_MIRROR=http://localhost:8080/hex
|
|
```
|
|
|
|
### pub.dev (Dart/Flutter)
|
|
|
|
Set the PUB_HOSTED_URL environment variable:
|
|
|
|
```bash
|
|
export PUB_HOSTED_URL=http://localhost:8080/pub
|
|
```
|
|
|
|
### PyPI (pip)
|
|
|
|
Configure pip to use the proxy:
|
|
|
|
```bash
|
|
pip install --index-url http://localhost:8080/pypi/simple/ package_name
|
|
```
|
|
|
|
Or set in `~/.pip/pip.conf`:
|
|
|
|
```ini
|
|
[global]
|
|
index-url = http://localhost:8080/pypi/simple/
|
|
```
|
|
|
|
### Maven
|
|
|
|
Add to your `~/.m2/settings.xml`:
|
|
|
|
```xml
|
|
<settings>
|
|
<mirrors>
|
|
<mirror>
|
|
<id>proxy</id>
|
|
<mirrorOf>central</mirrorOf>
|
|
<url>http://localhost:8080/maven/</url>
|
|
</mirror>
|
|
</mirrors>
|
|
</settings>
|
|
```
|
|
|
|
The `/maven/` endpoint uses Maven Central as primary upstream and falls back to the Gradle Plugin Portal for Gradle plugin marker metadata and related artifacts when the primary upstream returns not found.
|
|
|
|
For Gradle plugin resolution via the same proxy endpoint:
|
|
|
|
```kotlin
|
|
pluginManagement {
|
|
repositories {
|
|
maven(url = "http://localhost:8080/maven/")
|
|
}
|
|
}
|
|
```
|
|
|
|
### Gradle HTTP Build Cache
|
|
|
|
Configure in `settings.gradle(.kts)`:
|
|
|
|
```kotlin
|
|
buildCache {
|
|
local {
|
|
enabled = false
|
|
}
|
|
remote<HttpBuildCache> {
|
|
url = uri("http://localhost:8080/gradle/")
|
|
push = true
|
|
}
|
|
}
|
|
```
|
|
|
|
### NuGet
|
|
|
|
Configure in `nuget.config`:
|
|
|
|
```xml
|
|
<configuration>
|
|
<packageSources>
|
|
<clear />
|
|
<add key="proxy" value="http://localhost:8080/nuget/v3/index.json" />
|
|
</packageSources>
|
|
</configuration>
|
|
```
|
|
|
|
Or use the CLI:
|
|
|
|
```bash
|
|
dotnet nuget add source http://localhost:8080/nuget/v3/index.json -n proxy
|
|
```
|
|
|
|
### Composer (PHP)
|
|
|
|
Configure in `composer.json`:
|
|
|
|
```json
|
|
{
|
|
"repositories": [
|
|
{
|
|
"type": "composer",
|
|
"url": "http://localhost:8080/composer"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Or set globally:
|
|
|
|
```bash
|
|
composer config -g repositories.proxy composer http://localhost:8080/composer
|
|
```
|
|
|
|
### Conan (C/C++)
|
|
|
|
Add the proxy as a remote:
|
|
|
|
```bash
|
|
conan remote add proxy http://localhost:8080/conan
|
|
conan remote disable conancenter
|
|
```
|
|
|
|
Or configure in `~/.conan2/remotes.json`.
|
|
|
|
### Conda
|
|
|
|
Configure in `~/.condarc`:
|
|
|
|
```yaml
|
|
channels:
|
|
- http://localhost:8080/conda/main
|
|
- http://localhost:8080/conda/conda-forge
|
|
default_channels:
|
|
- http://localhost:8080/conda/main
|
|
```
|
|
|
|
Or set via command:
|
|
|
|
```bash
|
|
conda config --add channels http://localhost:8080/conda/main
|
|
```
|
|
|
|
### CRAN (R)
|
|
|
|
Set the repository in R:
|
|
|
|
```r
|
|
options(repos = c(CRAN = "http://localhost:8080/cran"))
|
|
```
|
|
|
|
Or in `~/.Rprofile` for persistence:
|
|
|
|
```r
|
|
local({
|
|
r <- getOption("repos")
|
|
r["CRAN"] <- "http://localhost:8080/cran"
|
|
options(repos = r)
|
|
})
|
|
```
|
|
|
|
### Julia
|
|
|
|
Set the Pkg server before starting Julia:
|
|
|
|
```bash
|
|
export JULIA_PKG_SERVER=http://localhost:8080/julia
|
|
```
|
|
|
|
Or inside a running session:
|
|
|
|
```julia
|
|
ENV["JULIA_PKG_SERVER"] = "http://localhost:8080/julia"
|
|
using Pkg; Pkg.update()
|
|
```
|
|
|
|
### Docker / Container Registry
|
|
|
|
Configure Docker to use the proxy as a registry mirror in `/etc/docker/daemon.json`:
|
|
|
|
```json
|
|
{
|
|
"registry-mirrors": ["http://localhost:8080"]
|
|
}
|
|
```
|
|
|
|
Then restart Docker:
|
|
|
|
```bash
|
|
sudo systemctl restart docker
|
|
```
|
|
|
|
Or pull images directly:
|
|
|
|
```bash
|
|
docker pull localhost:8080/library/nginx:latest
|
|
```
|
|
|
|
### Debian / APT
|
|
|
|
Configure APT to use the proxy in `/etc/apt/sources.list.d/proxy.list`:
|
|
|
|
```
|
|
deb http://localhost:8080/debian stable main contrib
|
|
```
|
|
|
|
Replace your existing sources.list entries, then:
|
|
|
|
```bash
|
|
sudo apt update
|
|
```
|
|
|
|
### RPM / Yum / DNF
|
|
|
|
Configure yum/dnf to use the proxy in `/etc/yum.repos.d/proxy.repo`:
|
|
|
|
```ini
|
|
[proxy-fedora]
|
|
name=Fedora via Proxy
|
|
baseurl=http://localhost:8080/rpm/releases/$releasever/Everything/$basearch/os/
|
|
enabled=1
|
|
gpgcheck=0
|
|
```
|
|
|
|
Then:
|
|
|
|
```bash
|
|
sudo dnf clean all
|
|
sudo dnf update
|
|
```
|
|
|
|
## Configuration
|
|
|
|
The proxy can be configured via:
|
|
|
|
1. Command line flags (highest priority)
|
|
2. Environment variables
|
|
3. Configuration file (YAML or JSON)
|
|
|
|
### Command Line Flags
|
|
|
|
```
|
|
-config string Path to configuration file
|
|
-listen string Address to listen on (default ":8080")
|
|
-base-url string Public URL of this proxy (default "http://localhost:8080")
|
|
-storage-url string Storage URL (file:// or s3://)
|
|
-storage-path string Path to artifact storage directory (deprecated, use -storage-url)
|
|
-database-driver string Database driver: sqlite or postgres (default "sqlite")
|
|
-database-path string Path to SQLite database file (default "./cache/proxy.db")
|
|
-database-url string PostgreSQL connection URL
|
|
-log-level string Log level: debug, info, warn, error (default "info")
|
|
-log-format string Log format: text, json (default "text")
|
|
-version Print version and exit
|
|
```
|
|
|
|
### Environment Variables
|
|
|
|
```bash
|
|
PROXY_LISTEN=:8080
|
|
PROXY_BASE_URL=http://localhost:8080
|
|
PROXY_UI_URL=http://localhost:8080 # Optional; defaults to PROXY_BASE_URL
|
|
PROXY_STORAGE_URL=file:///var/cache/proxy/artifacts
|
|
PROXY_DATABASE_DRIVER=sqlite
|
|
PROXY_DATABASE_PATH=./cache/proxy.db
|
|
PROXY_DATABASE_URL=postgres://user:pass@localhost/proxy?sslmode=disable
|
|
PROXY_LOG_LEVEL=info
|
|
PROXY_LOG_FORMAT=text
|
|
```
|
|
|
|
### Configuration File
|
|
|
|
```yaml
|
|
listen: ":8080"
|
|
base_url: "http://localhost:8080"
|
|
|
|
storage:
|
|
url: "file:///var/cache/proxy/artifacts"
|
|
max_size: "10GB" # Optional: evict LRU when exceeded
|
|
|
|
database:
|
|
driver: "sqlite"
|
|
path: "/var/lib/proxy/cache.db"
|
|
|
|
log:
|
|
level: "info"
|
|
format: "text"
|
|
|
|
# Optional: override upstream URLs
|
|
upstream:
|
|
npm: "https://registry.npmjs.org"
|
|
cargo: "https://index.crates.io"
|
|
|
|
# Optional: version cooldown (see above)
|
|
cooldown:
|
|
default: "3d"
|
|
```
|
|
|
|
Run with config file:
|
|
|
|
```bash
|
|
./proxy -config /etc/proxy/config.yaml
|
|
```
|
|
|
|
### PostgreSQL
|
|
|
|
SQLite is the default and works well for single-node deployments. For multi-node setups or if you prefer a managed database, switch to Postgres:
|
|
|
|
```yaml
|
|
database:
|
|
driver: "postgres"
|
|
url: "postgres://user:password@localhost:5432/proxy?sslmode=disable"
|
|
```
|
|
|
|
Or via environment variables:
|
|
|
|
```bash
|
|
PROXY_DATABASE_DRIVER=postgres
|
|
PROXY_DATABASE_URL=postgres://user:password@localhost:5432/proxy?sslmode=disable
|
|
```
|
|
|
|
The proxy creates tables automatically on first run.
|
|
|
|
### S3 Storage
|
|
|
|
The proxy can store cached artifacts in S3 or any S3-compatible service (MinIO, R2, etc.) instead of the local filesystem.
|
|
|
|
```yaml
|
|
storage:
|
|
url: "s3://my-bucket-name?region=us-east-1"
|
|
```
|
|
|
|
For S3-compatible services like MinIO:
|
|
|
|
```yaml
|
|
storage:
|
|
url: "s3://my-bucket?endpoint=http://localhost:9000&disableSSL=true&s3ForcePathStyle=true"
|
|
```
|
|
|
|
Set credentials via standard AWS environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`).
|
|
|
|
## CLI Commands
|
|
|
|
### serve (default)
|
|
|
|
Start the proxy server. This is the default command if none is specified.
|
|
|
|
```bash
|
|
proxy serve [flags]
|
|
proxy [flags] # same as 'proxy serve'
|
|
```
|
|
|
|
### mirror
|
|
|
|
Pre-populate the cache from PURLs, SBOM files, or entire registries. Useful for ensuring offline availability or warming the cache before deployments.
|
|
|
|
```bash
|
|
# Mirror specific package versions
|
|
proxy mirror pkg:npm/lodash@4.17.21 pkg:cargo/serde@1.0.0
|
|
|
|
# Mirror all versions of a package
|
|
proxy mirror pkg:npm/lodash
|
|
|
|
# Mirror from a CycloneDX or SPDX SBOM
|
|
proxy mirror --sbom sbom.cdx.json
|
|
|
|
# Preview what would be mirrored
|
|
proxy mirror --dry-run pkg:npm/lodash
|
|
|
|
# Control parallelism
|
|
proxy mirror --concurrency 8 pkg:npm/lodash@4.17.21
|
|
```
|
|
|
|
The mirror command accepts the same storage and database flags as `serve`. Already-cached artifacts are skipped.
|
|
|
|
A mirror API is also available when the server is running:
|
|
|
|
```bash
|
|
# Start a mirror job
|
|
curl -X POST http://localhost:8080/api/mirror \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"purls": ["pkg:npm/lodash@4.17.21"]}'
|
|
|
|
# Check job status
|
|
curl http://localhost:8080/api/mirror/mirror-1
|
|
|
|
# Cancel a running job
|
|
curl -X DELETE http://localhost:8080/api/mirror/mirror-1
|
|
```
|
|
|
|
### stats
|
|
|
|
Show cache statistics without running the server.
|
|
|
|
```bash
|
|
# Text output
|
|
proxy stats
|
|
|
|
# JSON output
|
|
proxy stats -json
|
|
|
|
# Custom database path
|
|
proxy stats -database-path /var/lib/proxy/cache.db
|
|
|
|
# With PostgreSQL
|
|
proxy stats -database-driver postgres -database-url postgres://user:pass@localhost/proxy
|
|
|
|
# Show top 20 most popular packages
|
|
proxy stats -popular 20
|
|
```
|
|
|
|
Example output:
|
|
|
|
```
|
|
Cache Statistics
|
|
================
|
|
|
|
Packages: 45
|
|
Versions: 128
|
|
Artifacts: 128
|
|
Total size: 892.4 MB
|
|
Total hits: 1547
|
|
|
|
Packages by ecosystem:
|
|
npm 32
|
|
cargo 13
|
|
|
|
Most popular packages:
|
|
1. npm/lodash (342 hits, 24.7 KB)
|
|
2. npm/react (198 hits, 89.3 KB)
|
|
3. cargo/serde (156 hits, 234.1 KB)
|
|
|
|
Recently cached:
|
|
npm/express@4.18.2 (2024-01-15 14:32, 54.2 KB)
|
|
cargo/tokio@1.35.0 (2024-01-15 14:28, 412.8 KB)
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
### Registry Protocols
|
|
|
|
| Endpoint | Description |
|
|
|----------|-------------|
|
|
| `GET /` | Dashboard (web UI) |
|
|
| `GET /health` | Health check (JSON; HTTP 200 healthy, 503 unhealthy) |
|
|
| `GET /stats` | Cache statistics (JSON) |
|
|
| `GET /metrics` | Prometheus metrics |
|
|
| `GET /npm/*` | npm registry protocol |
|
|
| `GET /cargo/*` | Cargo sparse index protocol |
|
|
| `GET /gem/*` | RubyGems protocol |
|
|
| `GET /go/*` | Go module proxy protocol |
|
|
| `GET /hex/*` | Hex.pm protocol |
|
|
| `GET /pub/*` | pub.dev protocol |
|
|
| `GET /pypi/*` | PyPI simple/JSON API |
|
|
| `GET /maven/*` | Maven repository protocol |
|
|
| `GET /nuget/*` | NuGet V3 API |
|
|
| `GET /composer/*` | Composer/Packagist protocol |
|
|
| `GET /conan/*` | Conan C/C++ protocol |
|
|
| `GET /conda/*` | Conda/Anaconda protocol |
|
|
| `GET /cran/*` | CRAN (R) protocol |
|
|
| `GET /julia/*` | Julia Pkg server protocol |
|
|
| `GET /v2/*` | OCI/Docker registry protocol |
|
|
| `GET /debian/*` | Debian/APT repository protocol |
|
|
| `GET /rpm/*` | RPM/Yum repository protocol |
|
|
|
|
### Mirror API
|
|
|
|
| Endpoint | Description |
|
|
|----------|-------------|
|
|
| `POST /api/mirror` | Start a mirror job (JSON body with `purls`) |
|
|
| `GET /api/mirror/{id}` | Get job status and progress |
|
|
| `DELETE /api/mirror/{id}` | Cancel a running job |
|
|
|
|
### Enrichment API
|
|
|
|
The proxy provides REST endpoints for package metadata enrichment, vulnerability scanning, and outdated detection.
|
|
|
|
| Endpoint | Description |
|
|
|----------|-------------|
|
|
| `GET /api/package/{ecosystem}/{name}` | Get package metadata |
|
|
| `GET /api/package/{ecosystem}/{name}/{version}` | Get version metadata with vulnerabilities |
|
|
| `GET /api/vulns/{ecosystem}/{name}` | Get all vulnerabilities for a package |
|
|
| `GET /api/vulns/{ecosystem}/{name}/{version}` | Get vulnerabilities for a specific version |
|
|
| `POST /api/outdated` | Check multiple packages for outdated versions |
|
|
| `POST /api/bulk` | Bulk package metadata lookup |
|
|
|
|
#### Get Package Metadata
|
|
|
|
```bash
|
|
curl http://localhost:8080/api/package/npm/lodash
|
|
```
|
|
|
|
Response:
|
|
|
|
```json
|
|
{
|
|
"ecosystem": "npm",
|
|
"name": "lodash",
|
|
"latest_version": "4.17.21",
|
|
"license": "MIT",
|
|
"license_category": "permissive",
|
|
"description": "Lodash modular utilities",
|
|
"homepage": "https://lodash.com/",
|
|
"repository": "https://github.com/lodash/lodash",
|
|
"registry_url": "https://registry.npmjs.org"
|
|
}
|
|
```
|
|
|
|
#### Get Version with Vulnerabilities
|
|
|
|
```bash
|
|
curl http://localhost:8080/api/package/npm/lodash/4.17.0
|
|
```
|
|
|
|
Response:
|
|
|
|
```json
|
|
{
|
|
"package": {
|
|
"ecosystem": "npm",
|
|
"name": "lodash",
|
|
"latest_version": "4.17.21",
|
|
"license": "MIT",
|
|
"license_category": "permissive"
|
|
},
|
|
"version": {
|
|
"ecosystem": "npm",
|
|
"name": "lodash",
|
|
"version": "4.17.0",
|
|
"license": "MIT",
|
|
"published_at": "2016-06-17T03:59:56Z",
|
|
"yanked": false,
|
|
"is_outdated": true
|
|
},
|
|
"vulnerabilities": [
|
|
{
|
|
"id": "GHSA-p6mc-m468-83gw",
|
|
"summary": "Prototype Pollution in lodash",
|
|
"severity": "HIGH",
|
|
"cvss_score": 7.4,
|
|
"fixed_version": "4.17.12"
|
|
}
|
|
],
|
|
"is_outdated": true,
|
|
"license_category": "permissive"
|
|
}
|
|
```
|
|
|
|
#### Check Outdated Packages
|
|
|
|
```bash
|
|
curl -X POST http://localhost:8080/api/outdated \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"packages": [
|
|
{"ecosystem": "npm", "name": "lodash", "version": "4.17.0"},
|
|
{"ecosystem": "pypi", "name": "requests", "version": "2.25.0"}
|
|
]
|
|
}'
|
|
```
|
|
|
|
Response:
|
|
|
|
```json
|
|
{
|
|
"results": [
|
|
{
|
|
"ecosystem": "npm",
|
|
"name": "lodash",
|
|
"version": "4.17.0",
|
|
"latest_version": "4.17.21",
|
|
"is_outdated": true
|
|
},
|
|
{
|
|
"ecosystem": "pypi",
|
|
"name": "requests",
|
|
"version": "2.25.0",
|
|
"latest_version": "2.31.0",
|
|
"is_outdated": true
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
#### Bulk Package Lookup
|
|
|
|
```bash
|
|
curl -X POST http://localhost:8080/api/bulk \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"purls": [
|
|
"pkg:npm/lodash@4.17.21",
|
|
"pkg:pypi/requests@2.28.0"
|
|
]
|
|
}'
|
|
```
|
|
|
|
Response:
|
|
|
|
```json
|
|
{
|
|
"packages": {
|
|
"pkg:npm/lodash": {
|
|
"ecosystem": "npm",
|
|
"name": "lodash",
|
|
"latest_version": "4.17.21",
|
|
"license": "MIT",
|
|
"license_category": "permissive"
|
|
},
|
|
"pkg:pypi/requests": {
|
|
"ecosystem": "pypi",
|
|
"name": "requests",
|
|
"latest_version": "2.31.0",
|
|
"license": "Apache-2.0",
|
|
"license_category": "permissive"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Stats Response (HTTP endpoint)
|
|
|
|
```json
|
|
{
|
|
"cached_artifacts": 142,
|
|
"total_size_bytes": 523456789,
|
|
"total_size": "499.2 MB",
|
|
"storage_url": "file:///path/to/cache/artifacts",
|
|
"database_path": "./cache/proxy.db"
|
|
}
|
|
```
|
|
|
|
## How It Works
|
|
|
|
1. Package manager requests package metadata from the proxy
|
|
2. Proxy fetches metadata from upstream, rewrites artifact URLs to point at proxy
|
|
3. Package manager requests artifact (tarball, crate, etc.)
|
|
4. Proxy checks local cache:
|
|
- **Cache hit**: Serve from local storage
|
|
- **Cache miss**: Fetch from upstream, store locally, serve to client
|
|
5. Subsequent requests for the same artifact are served from cache
|
|
|
|
```
|
|
┌─────────────┐ ┌─────────┐ ┌──────────┐
|
|
│ npm/cargo │────▶│ proxy │────▶│ upstream │
|
|
│ client │◀────│ │◀────│ registry │
|
|
└─────────────┘ └─────────┘ └──────────┘
|
|
│
|
|
▼
|
|
┌─────────┐
|
|
│ cache │
|
|
│ storage │
|
|
└─────────┘
|
|
```
|
|
|
|
## Web Interface
|
|
|
|
The proxy serves a web UI under `/ui`. No separate frontend build is needed -- templates and assets are embedded in the binary. `GET /` redirects to `/ui/`. The UI is mounted under its own prefix so a reverse proxy can apply different access rules to it than to the package endpoints (for example, requiring auth for `PathPrefix(/ui)` while leaving `/npm`, `/pypi` etc. open to build machines).
|
|
|
|
- **Dashboard** (`/ui/`) -- cache stats, popular packages, recently cached artifacts, and vulnerability overview.
|
|
- **Install guide** (`/ui/install`) -- per-ecosystem configuration instructions, so you don't have to look them up here.
|
|
- **Package browser** (`/ui/packages`) -- browse all cached packages with filtering by ecosystem and sorting by hits, size, name, or vulnerability count.
|
|
- **Search** (`/ui/search?q=...`) -- search cached packages by name.
|
|
- **Package detail** (`/ui/package/{ecosystem}/{name}`) -- metadata, license, vulnerabilities, and version list for a package. You can select two versions to compare.
|
|
- **Version detail** (`/ui/package/{ecosystem}/{name}/{version}`) -- per-version metadata, integrity hash, artifact cache status, and hit counts.
|
|
- **Source browser** (`/ui/package/{ecosystem}/{name}/{version}/browse`) -- browse files inside cached archives with syntax highlighting for text files and image previews.
|
|
- **Version diff** (`/ui/package/{ecosystem}/{name}/compare/{v1}...{v2}`) -- side-by-side diff of two cached versions showing added, removed, and changed files.
|
|
|
|
## Monitoring
|
|
|
|
The proxy exposes Prometheus metrics at `GET /metrics`. All metric names are prefixed with `proxy_`.
|
|
|
|
| Metric | Type | Labels | Description |
|
|
|--------|------|--------|-------------|
|
|
| `proxy_cache_hits_total` | counter | `ecosystem` | Cache hits |
|
|
| `proxy_cache_misses_total` | counter | `ecosystem` | Cache misses |
|
|
| `proxy_cache_size_bytes` | gauge | | Total size of cached artifacts |
|
|
| `proxy_cached_artifacts_total` | gauge | | Number of cached artifacts |
|
|
| `proxy_upstream_fetch_duration_seconds` | histogram | `ecosystem` | Time spent fetching from upstream |
|
|
| `proxy_upstream_errors_total` | counter | `ecosystem`, `error_type` | Upstream fetch failures |
|
|
| `proxy_storage_operation_duration_seconds` | histogram | `operation` | Storage read/write latency |
|
|
| `proxy_storage_errors_total` | counter | `operation` | Storage read/write failures |
|
|
| `proxy_active_requests` | gauge | | In-flight requests |
|
|
| `proxy_health_probe_failures_total` | counter | `step` | Storage health probe failures by failing step (`write`, `size`, `read`, `verify`, `delete`). |
|
|
|
|
Cache size and artifact count are refreshed every 60 seconds. The remaining metrics update on each request.
|
|
|
|
### Health Check
|
|
|
|
`/health` returns a structured JSON report of subsystem health. HTTP 200 if all checks pass; 503 if any fail.
|
|
|
|
```json
|
|
{
|
|
"status": "ok",
|
|
"checks": {
|
|
"database": {"status": "ok"},
|
|
"storage": {"status": "ok"}
|
|
}
|
|
}
|
|
```
|
|
|
|
Failing checks include an `"error"` field. Storage failures also include a `"step"` field identifying which probe step failed (`write`, `size`, `read`, `verify`, `delete`). When the database check fails, the storage entry reports `{"status": "skipped"}` so the response always carries the same key set.
|
|
|
|
Storage probe results are cached for `health.storage_probe_interval` (default 30s) to bound the cost of probing remote backends. A probe holds an internal mutex for up to 10 seconds (the hardcoded per-probe timeout), so `/health` is intended as a Kubernetes **readiness** probe rather than a liveness probe — a slow S3 round-trip should pull the pod from rotation, not restart it.
|
|
|
|
Scrape config for Prometheus:
|
|
|
|
```yaml
|
|
scrape_configs:
|
|
- job_name: git-pkgs-proxy
|
|
static_configs:
|
|
- targets: ["localhost:8080"]
|
|
```
|
|
|
|
## Production Deployment
|
|
|
|
### Systemd Service
|
|
|
|
Create `/etc/systemd/system/proxy.service`:
|
|
|
|
```ini
|
|
[Unit]
|
|
Description=git-pkgs proxy
|
|
After=network.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=proxy
|
|
ExecStart=/usr/local/bin/proxy -config /etc/proxy/config.yaml
|
|
Restart=always
|
|
RestartSec=5
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
|
|
Enable and start:
|
|
|
|
```bash
|
|
sudo systemctl enable proxy
|
|
sudo systemctl start proxy
|
|
```
|
|
|
|
### Docker
|
|
|
|
A Dockerfile is included in the repo. Build and run:
|
|
|
|
```bash
|
|
docker build -t proxy .
|
|
docker run -p 8080:8080 -v proxy-data:/data proxy
|
|
```
|
|
|
|
With Postgres and S3:
|
|
|
|
```bash
|
|
docker run -p 8080:8080 \
|
|
-e PROXY_DATABASE_DRIVER=postgres \
|
|
-e PROXY_DATABASE_URL=postgres://user:pass@db:5432/proxy \
|
|
-e PROXY_STORAGE_URL=s3://my-bucket?region=us-east-1 \
|
|
-e AWS_ACCESS_KEY_ID=... \
|
|
-e AWS_SECRET_ACCESS_KEY=... \
|
|
proxy
|
|
```
|
|
|
|
### Behind a Reverse Proxy
|
|
|
|
When running behind nginx, Apache, or another reverse proxy, set `base_url` to your public URL:
|
|
|
|
```yaml
|
|
base_url: "https://proxy.example.com"
|
|
```
|
|
|
|
If the UI is reached on a different hostname than the package endpoints — for example, the UI exposed publicly on a domain while build machines hit a Docker network alias — set `ui_base_url` separately. `base_url` is the URL package managers and metadata rewriting use; `ui_base_url` is the URL advertised to humans visiting the web UI (canonical/`og:url` tags and the install guide banner):
|
|
|
|
```yaml
|
|
base_url: "http://pkg-proxy:8080" # internal alias for build machines
|
|
ui_base_url: "https://proxy.example.com/ui" # public UI URL
|
|
```
|
|
|
|
When unset, `ui_base_url` defaults to `base_url`.
|
|
|
|
> **Warning:** the proxy serves the UI and package endpoints on the same listener. Setting `ui_base_url` only changes what URL the UI advertises to humans; it does not stop package endpoints from being reachable on the same hostname and port. When fronting the proxy with a public reverse proxy, restrict the public route to `PathPrefix(/ui)` (or your proxy's equivalent), otherwise `/npm`, `/pypi`, and the other package endpoints stay exposed alongside the UI.
|
|
|
|
nginx example, restricting the public host to the UI while leaving package endpoints reachable only on the internal listener:
|
|
|
|
```nginx
|
|
server {
|
|
listen 443 ssl;
|
|
server_name proxy.example.com;
|
|
|
|
location /ui/ {
|
|
proxy_pass http://127.0.0.1:8080;
|
|
proxy_set_header Host $host;
|
|
proxy_set_header X-Real-IP $remote_addr;
|
|
proxy_buffering off;
|
|
}
|
|
|
|
location / {
|
|
return 404;
|
|
}
|
|
}
|
|
```
|
|
|
|
Traefik example using `PathPrefix(/ui)` so the public router only matches UI traffic:
|
|
|
|
```yaml
|
|
labels:
|
|
traefik.enable: "true"
|
|
traefik.http.services.pkg-proxy.loadbalancer.server.port: "8080"
|
|
traefik.http.routers.pkg-proxy.rule: "Host(`proxy.example.com`) && PathPrefix(`/ui`)"
|
|
traefik.http.routers.pkg-proxy.entrypoints: "websecure"
|
|
```
|
|
|
|
## Cache Management
|
|
|
|
The proxy stores artifacts in the configured storage directory with this structure:
|
|
|
|
```
|
|
cache/artifacts/
|
|
├── npm/
|
|
│ └── lodash/
|
|
│ └── 4.17.21/
|
|
│ └── lodash-4.17.21.tgz
|
|
├── cargo/
|
|
│ └── serde/
|
|
│ └── 1.0.193/
|
|
│ └── serde-1.0.193.crate
|
|
├── oci/
|
|
│ └── library/nginx/
|
|
│ └── sha256:abc123.../
|
|
│ └── sha256:abc123...
|
|
├── deb/
|
|
│ └── nginx/
|
|
│ └── 1.18.0-6/
|
|
│ └── nginx_1.18.0-6_amd64.deb
|
|
└── rpm/
|
|
└── nginx/
|
|
└── 1.24.0-1.fc39/
|
|
└── nginx-1.24.0-1.fc39.x86_64.rpm
|
|
```
|
|
|
|
Cache metadata is stored in SQLite (default) or PostgreSQL. To clear a local cache:
|
|
|
|
```bash
|
|
rm -rf ./cache/artifacts/*
|
|
rm ./cache/proxy.db
|
|
```
|
|
|
|
The proxy will recreate the database on next start.
|
|
|
|
## Building from Source
|
|
|
|
Requirements:
|
|
|
|
- Go 1.25 or later
|
|
|
|
```bash
|
|
git clone https://github.com/git-pkgs/proxy.git
|
|
cd proxy
|
|
go build -o proxy ./cmd/proxy
|
|
```
|
|
|
|
Run tests:
|
|
|
|
```bash
|
|
go test ./...
|
|
```
|
|
|
|
## License
|
|
|
|
GPL-3.0-or-later
|