pkg-proxy/docs/architecture.md

Architecture

This document describes the internal architecture of the git-pkgs proxy.

Overview

The proxy is a caching HTTP server that sits between package manager clients and upstream registries. It intercepts requests, checks a local cache, and either serves cached content or fetches from upstream.

┌─────────────────────────────────────────────────────────────────┐
│                         HTTP Server                              │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │                    Router (ServeMux)                     │    │
│  │  /npm/*  -> NPMHandler                                   │    │
│  │  /cargo/* -> CargoHandler                                │    │
│  │  /health -> healthHandler                                │    │
│  │  /stats  -> statsHandler                                 │    │
│  └─────────────────────────────────────────────────────────┘    │
│                              │                                   │
│                              ▼                                   │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │                         Proxy                            │    │
│  │  - GetOrFetchArtifact()                                  │    │
│  │  - Coordinates DB, Storage, Fetcher                      │    │
│  └─────────────────────────────────────────────────────────┘    │
│         │                    │                    │              │
│         ▼                    ▼                    ▼              │
│  ┌───────────┐       ┌─────────────┐      ┌─────────────┐       │
│  │  Database │       │   Storage   │      │   Upstream  │       │
│  │  (SQLite) │       │ (Filesystem)│      │  (Fetcher)  │       │
│  └───────────┘       └─────────────┘      └─────────────┘       │
└─────────────────────────────────────────────────────────────────┘

Request Flow

Metadata Request (npm example)

1. Client requests GET /npm/lodash
2. NPMHandler receives request
3. Handler fetches metadata from upstream registry.npmjs.org/lodash
4. Handler rewrites tarball URLs in metadata to point at proxy
5. Handler returns modified metadata to client

Metadata is not cached - always fetched fresh. This ensures clients see new versions immediately.

Artifact Download (npm example)

1. Client requests GET /npm/lodash/-/lodash-4.17.21.tgz

2. NPMHandler extracts package name and version from URL

3. Handler calls Proxy.GetOrFetchArtifact()

4. Proxy checks database for cached artifact:

   Cache Hit:

   • Look up artifact record in database
   • Open file from storage
   • Record hit (increment counter, update last_accessed_at)
   • Return reader to handler
   • Handler streams file to client

   Cache Miss:

   • Resolve download URL using Resolver
   • Fetch artifact from upstream using Fetcher
   • Store artifact in Storage (returns size, hash)
   • Create/update database records (package, version, artifact)
   • Open stored file
   • Return reader to handler
   • Handler streams file to client

┌────────┐  GET /npm/lodash/-/lodash-4.17.21.tgz  ┌─────────────┐
│ Client │ ──────────────────────────────────────▶│ NPMHandler  │
└────────┘                                        └──────┬──────┘
                                                         │
                                                         ▼
                                               ┌─────────────────┐
                                               │ Proxy           │
                                               │ GetOrFetch      │
                                               └────────┬────────┘
                                                        │
                                    ┌───────────────────┼───────────────────┐
                                    │                   │                   │
                                    ▼                   ▼                   ▼
                             ┌───────────┐       ┌───────────┐       ┌───────────┐
                             │ Database  │       │  Storage  │       │ Upstream  │
                             │ (lookup)  │       │  (read)   │       │ (fetch)   │
                             └───────────┘       └───────────┘       └───────────┘

Package Structure

internal/database

SQLite database for cache metadata. Uses modernc.org/sqlite (pure Go, no CGO).

Tables:

packages (
    id, purl, ecosystem, name, namespace, latest_version,
    license, description, homepage, repository_url, upstream_url,
    metadata_fetched_at, created_at, updated_at
)

versions (
    id, purl, package_id, version, license, integrity,
    published_at, yanked, metadata_fetched_at, created_at, updated_at
)

artifacts (
    id, version_id, filename, upstream_url, storage_path,
    content_hash, size, content_type, fetched_at,
    hit_count, last_accessed_at, created_at, updated_at
)

Key operations:

• GetPackageByPURL() - Look up package by PURL
• GetVersionByPURL() - Look up version by PURL
• GetArtifact() - Look up artifact by version + filename
• UpsertPackage/Version/Artifact() - Insert or update records
• RecordArtifactHit() - Increment hit counter, update access time
• GetLeastRecentlyUsedArtifacts() - For cache eviction

internal/storage

File storage abstraction. Current implementation uses local filesystem.

Interface:

type Storage interface {
    Store(ctx, path, reader) (size, hash, error)
    Open(ctx, path) (io.ReadCloser, error)
    Exists(ctx, path) (bool, error)
    Delete(ctx, path) error
    Size(ctx, path) (int64, error)
    UsedSpace(ctx) (int64, error)
}

Filesystem implementation:

• Stores files in nested directories: {ecosystem}/{name}/{version}/{filename}
• Atomic writes using temp file + rename
• Computes SHA256 hash during write
• Cleans up empty parent directories on delete

Path structure:

cache/artifacts/
├── npm/
│   ├── lodash/
│   │   └── 4.17.21/
│   │       └── lodash-4.17.21.tgz
│   └── @babel/
│       └── core/
│           └── 7.23.0/
│               └── core-7.23.0.tgz
└── cargo/
    └── serde/
        └── 1.0.193/
            └── serde-1.0.193.crate

internal/upstream

Fetches artifacts from upstream registries.

Fetcher:

• HTTP client with configurable timeout (5 min default for large artifacts)
• Exponential backoff retry on 429 (rate limit) and 5xx errors
• Returns streaming reader (doesn't load into memory)
• Configurable user-agent

Resolver:

• Determines download URL for a package/version
• Handles ecosystem-specific URL patterns:
  • npm: https://registry.npmjs.org/{name}/-/{shortname}-{version}.tgz
  • cargo: https://static.crates.io/crates/{name}/{name}-{version}.crate
  • etc.

internal/handler

HTTP protocol handlers for each registry type.

Proxy (shared):

• GetOrFetchArtifact() - Main cache logic
• Coordinates database, storage, and fetcher
• Handles cache hit/miss flow

NPMHandler:

• handlePackageMetadata() - Proxy + rewrite metadata
• handleDownload() - Serve cached artifact
• Rewrites tarball URLs to point at proxy

CargoHandler:

• handleConfig() - Return registry config
• handleIndex() - Proxy sparse index
• handleDownload() - Serve cached crate

internal/server

HTTP server setup.

• Creates and wires together all components
• Mounts handlers at appropriate paths
• Adds logging middleware
• Health and stats endpoints

internal/config

Configuration loading.

• Supports YAML and JSON files
• Environment variable overrides (PROXY_ prefix)
• Command line flag overrides
• Validation

Extending the Proxy

Adding a New Registry

1. Add URL resolution in upstream/resolver.go
2. Create handler in handler/newregistry.go
3. Mount in server/server.go
4. Add tests

Adding a New Storage Backend

1. Implement storage.Storage interface
2. Add configuration options in config/config.go
3. Add initialization in server/server.go

Cache Eviction

The database tracks hit_count and last_accessed_at for LRU eviction. Query with:

db.GetLeastRecentlyUsedArtifacts(limit)

Eviction can be implemented as:

1. Background goroutine checking GetTotalCacheSize()
2. When over limit, get LRU artifacts
3. Delete from storage and clear database records

Design Decisions

Why SQLite?

• Simple deployment (single file)
• No external dependencies
• Good performance for this workload
• Pure Go driver available (no CGO)

Why rewrite metadata URLs?

• Ensures clients fetch artifacts through proxy
• Alternative: Let clients fetch directly, miss cache opportunity

Why not cache metadata?

• Simplicity - no invalidation logic needed
• Fresh data - new versions visible immediately
• Metadata is small, upstream fetch is fast

Why stream artifacts?

• Memory efficient - don't load large files into RAM
• Better latency - start sending while still receiving

Why atomic writes?

• Prevents serving partial files
• Safe concurrent access
• Clean recovery from crashes