forked from mirrors/pkg-proxy
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
| a947a7546a | |||
|
e1d2331ff0 |
||
|
|
e36a92433e |
||
|
|
941ed51f76 |
||
|
|
b68184cbab |
||
|
|
bcbb883d1b |
||
|
|
33d99e337b |
||
|
|
81f505757f |
||
|
|
7a758b9450 |
||
|
|
3bccc01776 |
||
|
|
43a164ed72 |
||
|
|
9708fe31a8 |
||
|
|
cb9bbbc385 |
||
|
|
75ff85f2f0 |
||
|
|
70fe686953 |
||
|
|
24d5e77443 |
||
|
|
15c133f1fa |
||
|
|
e45706d808 |
||
|
|
34009bad98 |
||
|
ec9c437498 |
33 changed files with 3683 additions and 1058 deletions
14
README.md
14
README.md
|
|
@ -18,9 +18,7 @@ cooldown:
|
|||
|
||||
A 3-day cooldown means that when `lodash` publishes version `4.18.0`, your builds keep using `4.17.21` until 3 days have passed. If the new release turns out to be compromised, you were never exposed.
|
||||
|
||||
Resolution order: package override, then ecosystem override, then global default. This lets you set a conservative default and carve out exceptions for packages where you need faster updates.
|
||||
|
||||
Currently works with npm, PyPI, pub.dev, Composer, and Cargo, which all include publish timestamps in their metadata. See [docs/configuration.md](docs/configuration.md) for the full config reference.
|
||||
Resolution order: package override, then ecosystem override, then global default. This lets you set a conservative default and carve out exceptions for packages where you need faster updates. See [docs/configuration.md](docs/configuration.md) for the full config reference.
|
||||
|
||||
## Supported Registries
|
||||
|
||||
|
|
@ -28,16 +26,16 @@ Currently works with npm, PyPI, pub.dev, Composer, and Cargo, which all include
|
|||
|----------|-------------------|:--------:|:---------:|
|
||||
| npm | JavaScript | Yes | ✓ |
|
||||
| Cargo | Rust | Yes | ✓ |
|
||||
| RubyGems | Ruby | | ✓ |
|
||||
| RubyGems | Ruby | Yes | ✓ |
|
||||
| Go proxy | Go | | ✓ |
|
||||
| Hex | Elixir | | ✓ |
|
||||
| Hex | Elixir | Yes* | ✓ |
|
||||
| pub.dev | Dart | Yes | ✓ |
|
||||
| PyPI | Python | Yes | ✓ |
|
||||
| Maven | Java | | ✓ |
|
||||
| NuGet | .NET | | ✓ |
|
||||
| NuGet | .NET | Yes | ✓ |
|
||||
| Composer | PHP | Yes | ✓ |
|
||||
| Conan | C/C++ | | ✓ |
|
||||
| Conda | Python/R | | ✓ |
|
||||
| Conda | Python/R | Yes | ✓ |
|
||||
| CRAN | R | | ✓ |
|
||||
| Container | Docker/OCI | | ✓ |
|
||||
| Debian | Debian/Ubuntu | | ✓ |
|
||||
|
|
@ -52,6 +50,8 @@ Currently works with npm, PyPI, pub.dev, Composer, and Cargo, which all include
|
|||
|
||||
Cooldown requires publish timestamps in metadata. Registries without a "Yes" in the cooldown column either don't expose timestamps or haven't been wired up yet.
|
||||
|
||||
\* Hex cooldown requires disabling registry signature verification (`HEX_NO_VERIFY_REPO_ORIGIN=1`) since the proxy re-encodes the protobuf payload.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
|
|
|
|||
|
|
@ -165,7 +165,7 @@ vulnerabilities (
|
|||
|
||||
On PostgreSQL, `INTEGER PRIMARY KEY` becomes `SERIAL`, `DATETIME` becomes `TIMESTAMP`, `INTEGER DEFAULT 0` booleans become `BOOLEAN DEFAULT FALSE`, and size/count columns use `BIGINT`.
|
||||
|
||||
The `MigrateSchema()` function handles backward compatibility with older git-pkgs databases by adding missing columns via `ALTER TABLE` as needed.
|
||||
The `MigrateSchema()` function handles backward compatibility with older git-pkgs databases by running named migrations that add missing columns and tables. See [migrations.md](migrations.md) for how to add new schema changes.
|
||||
|
||||
**Key operations:**
|
||||
- `GetPackageByPURL()` - Look up package by PURL
|
||||
|
|
|
|||
|
|
@ -209,7 +209,9 @@ Durations support days (`7d`), hours (`48h`), and minutes (`30m`). Set to `0` to
|
|||
|
||||
Resolution order: package override, then ecosystem override, then global default. This lets you set a conservative default while exempting trusted packages.
|
||||
|
||||
Currently supported for npm, PyPI, pub.dev, and Composer. These ecosystems include publish timestamps in their metadata. Other ecosystems (Go, Cargo, RubyGems) would require extra API calls and are not yet supported.
|
||||
Currently supported for npm, PyPI, pub.dev, Composer, Cargo, NuGet, Conda, RubyGems, and Hex. These ecosystems include publish timestamps in their metadata.
|
||||
|
||||
Note: Hex cooldown requires disabling registry signature verification since the proxy re-encodes the protobuf payload without the original signature. Set `HEX_NO_VERIFY_REPO_ORIGIN=1` or configure your repo with `no_verify: true`.
|
||||
|
||||
## Docker
|
||||
|
||||
|
|
|
|||
51
docs/migrations.md
Normal file
51
docs/migrations.md
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
# Database Migrations
|
||||
|
||||
Schema changes are tracked in a `migrations` table. Each migration has a name and a function. On startup, `MigrateSchema()` loads the set of already-applied names in one query and runs anything new.
|
||||
|
||||
Fresh databases created via `Create()` get the full schema and all migrations are recorded as already applied.
|
||||
|
||||
## Adding a migration
|
||||
|
||||
In `internal/database/schema.go`:
|
||||
|
||||
1. Write a migration function:
|
||||
|
||||
```go
|
||||
func migrateAddWidgetColumn(db *DB) error {
|
||||
hasCol, err := db.HasColumn("packages", "widget")
|
||||
if err != nil {
|
||||
return fmt.Errorf("checking column widget: %w", err)
|
||||
}
|
||||
if !hasCol {
|
||||
colType := "TEXT"
|
||||
if db.dialect == DialectPostgres {
|
||||
colType = "TEXT" // adjust if types differ
|
||||
}
|
||||
if _, err := db.Exec(fmt.Sprintf("ALTER TABLE packages ADD COLUMN widget %s", colType)); err != nil {
|
||||
return fmt.Errorf("adding column widget: %w", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
2. Append it to the `migrations` slice with the next sequential prefix:
|
||||
|
||||
```go
|
||||
var migrations = []migration{
|
||||
{"001_add_packages_enrichment_columns", migrateAddPackagesEnrichmentColumns},
|
||||
{"002_add_versions_enrichment_columns", migrateAddVersionsEnrichmentColumns},
|
||||
{"003_ensure_artifacts_table", migrateEnsureArtifactsTable},
|
||||
{"004_ensure_vulnerabilities_table", migrateEnsureVulnerabilitiesTable},
|
||||
{"005_add_widget_column", migrateAddWidgetColumn}, // new
|
||||
}
|
||||
```
|
||||
|
||||
3. Add the same column to both `schemaSQLite` and `schemaPostgres` at the top of the file so fresh databases start with the full schema.
|
||||
|
||||
## Rules
|
||||
|
||||
- Migration functions must be idempotent. Use `HasColumn`/`HasTable` checks or `IF NOT EXISTS` clauses so they're safe to run against a database that already has the change.
|
||||
- Handle both SQLite and Postgres dialects. Common differences: `DATETIME` vs `TIMESTAMP`, `INTEGER DEFAULT 0` vs `BOOLEAN DEFAULT FALSE`, `INTEGER PRIMARY KEY` vs `SERIAL PRIMARY KEY`.
|
||||
- Never reorder or rename existing entries. The name string is the migration's identity in the database.
|
||||
- Never remove old migrations from the list. They won't run on already-migrated databases, but they need to exist for older databases upgrading for the first time.
|
||||
|
|
@ -297,115 +297,6 @@ const docTemplate = `{
|
|||
}
|
||||
}
|
||||
},
|
||||
"/api/package/{ecosystem}/{name}": {
|
||||
"get": {
|
||||
"description": "Returns enriched package metadata. URL-encode scoped names (e.g. @scope/name -\u003e %40scope%2Fname).",
|
||||
"produces": [
|
||||
"application/json"
|
||||
],
|
||||
"tags": [
|
||||
"api"
|
||||
],
|
||||
"summary": "Get package metadata",
|
||||
"parameters": [
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Ecosystem",
|
||||
"name": "ecosystem",
|
||||
"in": "path",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Package name",
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"schema": {
|
||||
"$ref": "#/definitions/server.PackageResponse"
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Bad Request",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"404": {
|
||||
"description": "Not Found",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"500": {
|
||||
"description": "Internal Server Error",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/package/{ecosystem}/{name}/{version}": {
|
||||
"get": {
|
||||
"description": "Returns enriched package+version metadata and vulnerability data.",
|
||||
"produces": [
|
||||
"application/json"
|
||||
],
|
||||
"tags": [
|
||||
"api"
|
||||
],
|
||||
"summary": "Get version metadata and vulnerabilities",
|
||||
"parameters": [
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Ecosystem",
|
||||
"name": "ecosystem",
|
||||
"in": "path",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Package name",
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Version",
|
||||
"name": "version",
|
||||
"in": "path",
|
||||
"required": true
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"schema": {
|
||||
"$ref": "#/definitions/server.EnrichmentResponse"
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Bad Request",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"500": {
|
||||
"description": "Internal Server Error",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/packages": {
|
||||
"get": {
|
||||
"produces": [
|
||||
|
|
@ -505,108 +396,6 @@ const docTemplate = `{
|
|||
}
|
||||
}
|
||||
},
|
||||
"/api/vulns/{ecosystem}/{name}": {
|
||||
"get": {
|
||||
"description": "Returns vulnerabilities for a package across versions, or for a specific version if provided.",
|
||||
"produces": [
|
||||
"application/json"
|
||||
],
|
||||
"tags": [
|
||||
"api"
|
||||
],
|
||||
"summary": "Get vulnerabilities for a package or version",
|
||||
"parameters": [
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Ecosystem",
|
||||
"name": "ecosystem",
|
||||
"in": "path",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Package name",
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"schema": {
|
||||
"$ref": "#/definitions/server.VulnsResponse"
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Bad Request",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"500": {
|
||||
"description": "Internal Server Error",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/vulns/{ecosystem}/{name}/{version}": {
|
||||
"get": {
|
||||
"description": "Returns vulnerabilities for a package across versions, or for a specific version if provided.",
|
||||
"produces": [
|
||||
"application/json"
|
||||
],
|
||||
"tags": [
|
||||
"api"
|
||||
],
|
||||
"summary": "Get vulnerabilities for a package or version",
|
||||
"parameters": [
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Ecosystem",
|
||||
"name": "ecosystem",
|
||||
"in": "path",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Package name",
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Version",
|
||||
"name": "version",
|
||||
"in": "path"
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"schema": {
|
||||
"$ref": "#/definitions/server.VulnsResponse"
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Bad Request",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"500": {
|
||||
"description": "Internal Server Error",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/health": {
|
||||
"get": {
|
||||
"produces": [
|
||||
|
|
@ -715,29 +504,6 @@ const docTemplate = `{
|
|||
}
|
||||
}
|
||||
},
|
||||
"server.EnrichmentResponse": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"is_outdated": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"license_category": {
|
||||
"type": "string"
|
||||
},
|
||||
"package": {
|
||||
"$ref": "#/definitions/server.PackageResponse"
|
||||
},
|
||||
"version": {
|
||||
"$ref": "#/definitions/server.VersionResponse"
|
||||
},
|
||||
"vulnerabilities": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/definitions/server.VulnResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"server.OutdatedPackage": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
|
|
@ -949,84 +715,6 @@ const docTemplate = `{
|
|||
"type": "integer"
|
||||
}
|
||||
}
|
||||
},
|
||||
"server.VersionResponse": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"ecosystem": {
|
||||
"type": "string"
|
||||
},
|
||||
"integrity": {
|
||||
"type": "string"
|
||||
},
|
||||
"is_outdated": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"license": {
|
||||
"type": "string"
|
||||
},
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"published_at": {
|
||||
"type": "string"
|
||||
},
|
||||
"version": {
|
||||
"type": "string"
|
||||
},
|
||||
"yanked": {
|
||||
"type": "boolean"
|
||||
}
|
||||
}
|
||||
},
|
||||
"server.VulnResponse": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"cvss_score": {
|
||||
"type": "number"
|
||||
},
|
||||
"fixed_version": {
|
||||
"type": "string"
|
||||
},
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"references": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"severity": {
|
||||
"type": "string"
|
||||
},
|
||||
"summary": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"server.VulnsResponse": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"count": {
|
||||
"type": "integer"
|
||||
},
|
||||
"ecosystem": {
|
||||
"type": "string"
|
||||
},
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"version": {
|
||||
"type": "string"
|
||||
},
|
||||
"vulnerabilities": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/definitions/server.VulnResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}`
|
||||
|
|
|
|||
|
|
@ -290,115 +290,6 @@
|
|||
}
|
||||
}
|
||||
},
|
||||
"/api/package/{ecosystem}/{name}": {
|
||||
"get": {
|
||||
"description": "Returns enriched package metadata. URL-encode scoped names (e.g. @scope/name -\u003e %40scope%2Fname).",
|
||||
"produces": [
|
||||
"application/json"
|
||||
],
|
||||
"tags": [
|
||||
"api"
|
||||
],
|
||||
"summary": "Get package metadata",
|
||||
"parameters": [
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Ecosystem",
|
||||
"name": "ecosystem",
|
||||
"in": "path",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Package name",
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"schema": {
|
||||
"$ref": "#/definitions/server.PackageResponse"
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Bad Request",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"404": {
|
||||
"description": "Not Found",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"500": {
|
||||
"description": "Internal Server Error",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/package/{ecosystem}/{name}/{version}": {
|
||||
"get": {
|
||||
"description": "Returns enriched package+version metadata and vulnerability data.",
|
||||
"produces": [
|
||||
"application/json"
|
||||
],
|
||||
"tags": [
|
||||
"api"
|
||||
],
|
||||
"summary": "Get version metadata and vulnerabilities",
|
||||
"parameters": [
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Ecosystem",
|
||||
"name": "ecosystem",
|
||||
"in": "path",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Package name",
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Version",
|
||||
"name": "version",
|
||||
"in": "path",
|
||||
"required": true
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"schema": {
|
||||
"$ref": "#/definitions/server.EnrichmentResponse"
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Bad Request",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"500": {
|
||||
"description": "Internal Server Error",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/packages": {
|
||||
"get": {
|
||||
"produces": [
|
||||
|
|
@ -498,108 +389,6 @@
|
|||
}
|
||||
}
|
||||
},
|
||||
"/api/vulns/{ecosystem}/{name}": {
|
||||
"get": {
|
||||
"description": "Returns vulnerabilities for a package across versions, or for a specific version if provided.",
|
||||
"produces": [
|
||||
"application/json"
|
||||
],
|
||||
"tags": [
|
||||
"api"
|
||||
],
|
||||
"summary": "Get vulnerabilities for a package or version",
|
||||
"parameters": [
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Ecosystem",
|
||||
"name": "ecosystem",
|
||||
"in": "path",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Package name",
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"schema": {
|
||||
"$ref": "#/definitions/server.VulnsResponse"
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Bad Request",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"500": {
|
||||
"description": "Internal Server Error",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/vulns/{ecosystem}/{name}/{version}": {
|
||||
"get": {
|
||||
"description": "Returns vulnerabilities for a package across versions, or for a specific version if provided.",
|
||||
"produces": [
|
||||
"application/json"
|
||||
],
|
||||
"tags": [
|
||||
"api"
|
||||
],
|
||||
"summary": "Get vulnerabilities for a package or version",
|
||||
"parameters": [
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Ecosystem",
|
||||
"name": "ecosystem",
|
||||
"in": "path",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Package name",
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"description": "Version",
|
||||
"name": "version",
|
||||
"in": "path"
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"schema": {
|
||||
"$ref": "#/definitions/server.VulnsResponse"
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Bad Request",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"500": {
|
||||
"description": "Internal Server Error",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/health": {
|
||||
"get": {
|
||||
"produces": [
|
||||
|
|
@ -708,29 +497,6 @@
|
|||
}
|
||||
}
|
||||
},
|
||||
"server.EnrichmentResponse": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"is_outdated": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"license_category": {
|
||||
"type": "string"
|
||||
},
|
||||
"package": {
|
||||
"$ref": "#/definitions/server.PackageResponse"
|
||||
},
|
||||
"version": {
|
||||
"$ref": "#/definitions/server.VersionResponse"
|
||||
},
|
||||
"vulnerabilities": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/definitions/server.VulnResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"server.OutdatedPackage": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
|
|
@ -942,84 +708,6 @@
|
|||
"type": "integer"
|
||||
}
|
||||
}
|
||||
},
|
||||
"server.VersionResponse": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"ecosystem": {
|
||||
"type": "string"
|
||||
},
|
||||
"integrity": {
|
||||
"type": "string"
|
||||
},
|
||||
"is_outdated": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"license": {
|
||||
"type": "string"
|
||||
},
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"published_at": {
|
||||
"type": "string"
|
||||
},
|
||||
"version": {
|
||||
"type": "string"
|
||||
},
|
||||
"yanked": {
|
||||
"type": "boolean"
|
||||
}
|
||||
}
|
||||
},
|
||||
"server.VulnResponse": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"cvss_score": {
|
||||
"type": "number"
|
||||
},
|
||||
"fixed_version": {
|
||||
"type": "string"
|
||||
},
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"references": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"severity": {
|
||||
"type": "string"
|
||||
},
|
||||
"summary": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"server.VulnsResponse": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"count": {
|
||||
"type": "integer"
|
||||
},
|
||||
"ecosystem": {
|
||||
"type": "string"
|
||||
},
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"version": {
|
||||
"type": "string"
|
||||
},
|
||||
"vulnerabilities": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/definitions/server.VulnResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
4
go.mod
4
go.mod
|
|
@ -3,7 +3,7 @@ module github.com/git-pkgs/proxy
|
|||
go 1.25.6
|
||||
|
||||
require (
|
||||
github.com/git-pkgs/archives v0.2.0
|
||||
github.com/git-pkgs/archives v0.2.2
|
||||
github.com/git-pkgs/enrichment v0.2.1
|
||||
github.com/git-pkgs/purl v0.1.10
|
||||
github.com/git-pkgs/registries v0.4.0
|
||||
|
|
@ -17,6 +17,7 @@ require (
|
|||
github.com/prometheus/client_model v0.6.2
|
||||
github.com/swaggo/swag v1.16.6
|
||||
gocloud.dev v0.45.0
|
||||
google.golang.org/protobuf v1.36.11
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
modernc.org/sqlite v1.47.0
|
||||
)
|
||||
|
|
@ -284,7 +285,6 @@ require (
|
|||
google.golang.org/api v0.269.0 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260226221140-a57be14db171 // indirect
|
||||
google.golang.org/grpc v1.79.1 // indirect
|
||||
google.golang.org/protobuf v1.36.11 // indirect
|
||||
gopkg.in/ini.v1 v1.67.0 // indirect
|
||||
gopkg.in/yaml.v2 v2.4.0 // indirect
|
||||
honnef.co/go/tools v0.7.0 // indirect
|
||||
|
|
|
|||
4
go.sum
4
go.sum
|
|
@ -224,8 +224,8 @@ github.com/fzipp/gocyclo v0.6.0 h1:lsblElZG7d3ALtGMx9fmxeTKZaLLpU8mET09yN4BBLo=
|
|||
github.com/fzipp/gocyclo v0.6.0/go.mod h1:rXPyn8fnlpa0R2csP/31uerbiVBugk5whMdlyaLkLoA=
|
||||
github.com/ghostiam/protogetter v0.3.20 h1:oW7OPFit2FxZOpmMRPP9FffU4uUpfeE/rEdE1f+MzD0=
|
||||
github.com/ghostiam/protogetter v0.3.20/go.mod h1:FjIu5Yfs6FT391m+Fjp3fbAYJ6rkL/J6ySpZBfnODuI=
|
||||
github.com/git-pkgs/archives v0.2.0 h1:8OuuGwAB+Eww8/1ayyYpZzP0wVEH0/VWBG3mQrfi9SM=
|
||||
github.com/git-pkgs/archives v0.2.0/go.mod h1:LTJ1iQVFA7otizWMOyiI82NYVmyBWAPRzwu/e30rcXU=
|
||||
github.com/git-pkgs/archives v0.2.2 h1:RxOjrV8RzKicbMVdf2GDKOqIOHZNVjrLY/Pc7KSE/WQ=
|
||||
github.com/git-pkgs/archives v0.2.2/go.mod h1:LTJ1iQVFA7otizWMOyiI82NYVmyBWAPRzwu/e30rcXU=
|
||||
github.com/git-pkgs/enrichment v0.2.1 h1:mJJt4YQBzl9aOfu4226ylnC9H6YO9YZDjGpbSPVahKc=
|
||||
github.com/git-pkgs/enrichment v0.2.1/go.mod h1:q9eDZpRrUbYwzD4Mtg/T6LRdBMlt2DYRIvVRDULFnKg=
|
||||
github.com/git-pkgs/packageurl-go v0.3.1 h1:WM3RBABQZLaRBxgKyYughc3cVBE8KyQxbSC6Jt5ak7M=
|
||||
|
|
|
|||
|
|
@ -651,58 +651,159 @@ func TestMigrationFromOldSchema(t *testing.T) {
|
|||
}
|
||||
defer func() { _ = db.Close() }()
|
||||
|
||||
// Try to run queries that require new columns - these should fail without migration
|
||||
t.Run("queries should fail without migration", func(t *testing.T) {
|
||||
_, err := db.GetEnrichmentStats()
|
||||
if err == nil {
|
||||
t.Error("GetEnrichmentStats: expected error querying enriched_at column, got nil")
|
||||
}
|
||||
|
||||
_, err = db.GetPackageByEcosystemName("npm", "test-package")
|
||||
if err == nil {
|
||||
t.Error("GetPackageByEcosystemName: expected error querying registry_url column, got nil")
|
||||
}
|
||||
|
||||
// SearchPackages should work even with old schema because it uses sql.NullString
|
||||
// for nullable columns, which can handle NULL values properly
|
||||
_, err = db.SearchPackages("test", "", 10, 0)
|
||||
if err != nil {
|
||||
t.Errorf("SearchPackages: unexpected error with old schema: %v", err)
|
||||
}
|
||||
})
|
||||
// Queries that require new columns should fail without migration
|
||||
if _, err := db.GetEnrichmentStats(); err == nil {
|
||||
t.Error("GetEnrichmentStats: expected error querying enriched_at column, got nil")
|
||||
}
|
||||
if _, err := db.GetPackageByEcosystemName("npm", "test-package"); err == nil {
|
||||
t.Error("GetPackageByEcosystemName: expected error querying registry_url column, got nil")
|
||||
}
|
||||
// SearchPackages should work even with old schema because it uses sql.NullString
|
||||
if _, err := db.SearchPackages("test", "", 10, 0); err != nil {
|
||||
t.Errorf("SearchPackages: unexpected error with old schema: %v", err)
|
||||
}
|
||||
|
||||
// Run migration
|
||||
t.Run("migrate schema", func(t *testing.T) {
|
||||
if err := db.MigrateSchema(); err != nil {
|
||||
t.Fatalf("MigrateSchema failed: %v", err)
|
||||
}
|
||||
})
|
||||
if err := db.MigrateSchema(); err != nil {
|
||||
t.Fatalf("MigrateSchema failed: %v", err)
|
||||
}
|
||||
|
||||
// Verify queries work after migration
|
||||
t.Run("queries should work after migration", func(t *testing.T) {
|
||||
stats, err := db.GetEnrichmentStats()
|
||||
if err != nil {
|
||||
t.Errorf("GetEnrichmentStats failed after migration: %v", err)
|
||||
}
|
||||
if stats == nil {
|
||||
t.Error("GetEnrichmentStats returned nil after migration")
|
||||
}
|
||||
stats, err := db.GetEnrichmentStats()
|
||||
if err != nil {
|
||||
t.Errorf("GetEnrichmentStats failed after migration: %v", err)
|
||||
}
|
||||
if stats == nil {
|
||||
t.Error("GetEnrichmentStats returned nil after migration")
|
||||
}
|
||||
|
||||
pkg, err := db.GetPackageByEcosystemName("npm", "test-package")
|
||||
if err != nil {
|
||||
t.Errorf("GetPackageByEcosystemName failed after migration: %v", err)
|
||||
}
|
||||
if pkg == nil {
|
||||
t.Fatal("GetPackageByEcosystemName returned nil after migration")
|
||||
}
|
||||
if pkg.Name != "test-package" {
|
||||
t.Errorf("expected package name test-package, got %s", pkg.Name)
|
||||
}
|
||||
pkg, err := db.GetPackageByEcosystemName("npm", "test-package")
|
||||
if err != nil {
|
||||
t.Errorf("GetPackageByEcosystemName failed after migration: %v", err)
|
||||
}
|
||||
if pkg == nil {
|
||||
t.Fatal("GetPackageByEcosystemName returned nil after migration")
|
||||
}
|
||||
if pkg.Name != "test-package" {
|
||||
t.Errorf("expected package name test-package, got %s", pkg.Name)
|
||||
}
|
||||
|
||||
// Note: SearchPackages not tested here because old timestamp data
|
||||
// stored as strings can't be scanned into time.Time. This is a data
|
||||
// migration issue, not a schema migration issue.
|
||||
})
|
||||
// Verify migrations were recorded
|
||||
applied, err := db.appliedMigrations()
|
||||
if err != nil {
|
||||
t.Fatalf("appliedMigrations failed: %v", err)
|
||||
}
|
||||
for _, m := range migrations {
|
||||
if !applied[m.name] {
|
||||
t.Errorf("migration %s not recorded as applied", m.name)
|
||||
}
|
||||
}
|
||||
|
||||
// Running again should be a no-op
|
||||
if err := db.MigrateSchema(); err != nil {
|
||||
t.Fatalf("second MigrateSchema failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFreshDatabaseRecordsMigrations(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
dbPath := filepath.Join(dir, "fresh.db")
|
||||
|
||||
db, err := Create(dbPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Create failed: %v", err)
|
||||
}
|
||||
defer func() { _ = db.Close() }()
|
||||
|
||||
applied, err := db.appliedMigrations()
|
||||
if err != nil {
|
||||
t.Fatalf("appliedMigrations failed: %v", err)
|
||||
}
|
||||
|
||||
for _, m := range migrations {
|
||||
if !applied[m.name] {
|
||||
t.Errorf("migration %s not recorded in fresh database", m.name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMigrateSchemaSkipsApplied(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
dbPath := filepath.Join(dir, "test.db")
|
||||
|
||||
db, err := Create(dbPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Create failed: %v", err)
|
||||
}
|
||||
defer func() { _ = db.Close() }()
|
||||
|
||||
// All migrations are already recorded from Create. Running MigrateSchema
|
||||
// should return without running any migration functions.
|
||||
if err := db.MigrateSchema(); err != nil {
|
||||
t.Fatalf("MigrateSchema failed: %v", err)
|
||||
}
|
||||
|
||||
// Verify count hasn't changed (no duplicate inserts)
|
||||
var count int
|
||||
if err := db.Get(&count, "SELECT COUNT(*) FROM migrations"); err != nil {
|
||||
t.Fatalf("counting migrations failed: %v", err)
|
||||
}
|
||||
if count != len(migrations) {
|
||||
t.Errorf("expected %d migrations, got %d", len(migrations), count)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMigrateSchemaUpgradeFromFullyMigrated(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
dbPath := filepath.Join(dir, "existing.db")
|
||||
|
||||
// Simulate an existing proxy database that has the full current schema
|
||||
// but no migrations table (i.e. it was running the previous version).
|
||||
sqlDB, err := sql.Open("sqlite", dbPath)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to open database: %v", err)
|
||||
}
|
||||
|
||||
if _, err := sqlDB.Exec(schemaSQLite); err != nil {
|
||||
t.Fatalf("failed to create schema: %v", err)
|
||||
}
|
||||
// Drop the migrations table that schemaSQLite now includes
|
||||
if _, err := sqlDB.Exec("DROP TABLE migrations"); err != nil {
|
||||
t.Fatalf("failed to drop migrations table: %v", err)
|
||||
}
|
||||
if _, err := sqlDB.Exec("INSERT INTO schema_info (version) VALUES (1)"); err != nil {
|
||||
t.Fatalf("failed to set schema version: %v", err)
|
||||
}
|
||||
if err := sqlDB.Close(); err != nil {
|
||||
t.Fatalf("failed to close database: %v", err)
|
||||
}
|
||||
|
||||
db, err := Open(dbPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Open failed: %v", err)
|
||||
}
|
||||
defer func() { _ = db.Close() }()
|
||||
|
||||
// This should create the migrations table and record all migrations
|
||||
// without altering any tables (everything already exists).
|
||||
if err := db.MigrateSchema(); err != nil {
|
||||
t.Fatalf("MigrateSchema failed: %v", err)
|
||||
}
|
||||
|
||||
applied, err := db.appliedMigrations()
|
||||
if err != nil {
|
||||
t.Fatalf("appliedMigrations failed: %v", err)
|
||||
}
|
||||
for _, m := range migrations {
|
||||
if !applied[m.name] {
|
||||
t.Errorf("migration %s not recorded after upgrade", m.name)
|
||||
}
|
||||
}
|
||||
|
||||
// Second run should be the fast path (single SELECT)
|
||||
if err := db.MigrateSchema(); err != nil {
|
||||
t.Fatalf("second MigrateSchema failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConcurrentWrites(t *testing.T) {
|
||||
|
|
@ -890,3 +991,26 @@ func TestSearchPackagesWithValues(t *testing.T) {
|
|||
t.Errorf("expected 10 hits, got %d", result.Hits)
|
||||
}
|
||||
}
|
||||
|
||||
func BenchmarkMigrateSchemaFullyMigrated(b *testing.B) {
|
||||
dir := b.TempDir()
|
||||
dbPath := filepath.Join(dir, "bench.db")
|
||||
|
||||
db, err := Create(dbPath)
|
||||
if err != nil {
|
||||
b.Fatalf("Create failed: %v", err)
|
||||
}
|
||||
defer func() { _ = db.Close() }()
|
||||
|
||||
// First call to ensure everything is migrated
|
||||
if err := db.MigrateSchema(); err != nil {
|
||||
b.Fatalf("initial MigrateSchema failed: %v", err)
|
||||
}
|
||||
|
||||
b.ResetTimer()
|
||||
for b.Loop() {
|
||||
if err := db.MigrateSchema(); err != nil {
|
||||
b.Fatalf("MigrateSchema failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,10 @@
|
|||
package database
|
||||
|
||||
import "fmt"
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const postgresTimestamp = "TIMESTAMP"
|
||||
|
||||
|
|
@ -86,6 +90,11 @@ CREATE TABLE IF NOT EXISTS vulnerabilities (
|
|||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_vulns_id_pkg ON vulnerabilities(vuln_id, ecosystem, package_name);
|
||||
CREATE INDEX IF NOT EXISTS idx_vulns_ecosystem_pkg ON vulnerabilities(ecosystem, package_name);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS migrations (
|
||||
name TEXT NOT NULL PRIMARY KEY,
|
||||
applied_at DATETIME NOT NULL
|
||||
);
|
||||
`
|
||||
|
||||
var schemaPostgres = `
|
||||
|
|
@ -166,6 +175,11 @@ CREATE TABLE IF NOT EXISTS vulnerabilities (
|
|||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_vulns_id_pkg ON vulnerabilities(vuln_id, ecosystem, package_name);
|
||||
CREATE INDEX IF NOT EXISTS idx_vulns_ecosystem_pkg ON vulnerabilities(ecosystem, package_name);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS migrations (
|
||||
name TEXT NOT NULL PRIMARY KEY,
|
||||
applied_at TIMESTAMP NOT NULL
|
||||
);
|
||||
`
|
||||
|
||||
// schemaArtifactsOnly contains just the artifacts table for adding to existing git-pkgs databases.
|
||||
|
|
@ -232,6 +246,11 @@ func (db *DB) CreateSchema() error {
|
|||
return fmt.Errorf("setting schema version: %w", err)
|
||||
}
|
||||
|
||||
// Record all migrations as applied since the full schema is already current.
|
||||
if err := db.recordAllMigrations(); err != nil {
|
||||
return fmt.Errorf("recording migrations: %w", err)
|
||||
}
|
||||
|
||||
return db.OptimizeForReads()
|
||||
}
|
||||
|
||||
|
|
@ -292,24 +311,135 @@ func (db *DB) HasColumn(table, column string) (bool, error) {
|
|||
return exists, err
|
||||
}
|
||||
|
||||
// MigrateSchema adds missing columns to existing tables for backward compatibility.
|
||||
// migration represents a named schema migration.
|
||||
type migration struct {
|
||||
name string
|
||||
fn func(db *DB) error
|
||||
}
|
||||
|
||||
// migrations is the ordered list of all schema migrations. See
|
||||
// docs/migrations.md for how to add new ones.
|
||||
var migrations = []migration{
|
||||
{"001_add_packages_enrichment_columns", migrateAddPackagesEnrichmentColumns},
|
||||
{"002_add_versions_enrichment_columns", migrateAddVersionsEnrichmentColumns},
|
||||
{"003_ensure_artifacts_table", migrateEnsureArtifactsTable},
|
||||
{"004_ensure_vulnerabilities_table", migrateEnsureVulnerabilitiesTable},
|
||||
}
|
||||
|
||||
// isTableNotFound returns true if the error indicates a missing table.
|
||||
// SQLite returns "no such table: X", Postgres returns "relation \"X\" does not exist".
|
||||
func isTableNotFound(err error) bool {
|
||||
msg := err.Error()
|
||||
return strings.Contains(msg, "no such table") ||
|
||||
strings.Contains(msg, "does not exist")
|
||||
}
|
||||
|
||||
// createMigrationsTable creates the migrations table.
|
||||
func (db *DB) createMigrationsTable() error {
|
||||
var ts string
|
||||
if db.dialect == DialectPostgres {
|
||||
ts = "TIMESTAMP"
|
||||
} else {
|
||||
ts = "DATETIME"
|
||||
}
|
||||
|
||||
query := fmt.Sprintf(`CREATE TABLE IF NOT EXISTS migrations (
|
||||
name TEXT NOT NULL PRIMARY KEY,
|
||||
applied_at %s NOT NULL
|
||||
)`, ts)
|
||||
|
||||
if _, err := db.Exec(query); err != nil {
|
||||
return fmt.Errorf("creating migrations table: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// appliedMigrations returns the set of migration names that have been recorded.
|
||||
// Returns nil if the migrations table does not exist yet.
|
||||
func (db *DB) appliedMigrations() (map[string]bool, error) {
|
||||
var names []string
|
||||
err := db.Select(&names, "SELECT name FROM migrations")
|
||||
if err != nil {
|
||||
// Table doesn't exist yet — this is a pre-migration database.
|
||||
if isTableNotFound(err) {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, fmt.Errorf("loading applied migrations: %w", err)
|
||||
}
|
||||
|
||||
applied := make(map[string]bool, len(names))
|
||||
for _, name := range names {
|
||||
applied[name] = true
|
||||
}
|
||||
return applied, nil
|
||||
}
|
||||
|
||||
// recordMigration inserts a migration name into the migrations table.
|
||||
func (db *DB) recordMigration(name string) error {
|
||||
query := db.Rebind("INSERT INTO migrations (name, applied_at) VALUES (?, ?)")
|
||||
if _, err := db.Exec(query, name, time.Now().UTC()); err != nil {
|
||||
return fmt.Errorf("recording migration %s: %w", name, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// recordAllMigrations marks every known migration as applied.
|
||||
func (db *DB) recordAllMigrations() error {
|
||||
for _, m := range migrations {
|
||||
if err := db.recordMigration(m.name); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// MigrateSchema applies any unapplied migrations in order.
|
||||
// For a fully migrated database this executes a single SELECT query.
|
||||
func (db *DB) MigrateSchema() error {
|
||||
// Check and add missing columns to packages table
|
||||
packagesColumns := map[string]string{
|
||||
"registry_url": "TEXT",
|
||||
"supplier_name": "TEXT",
|
||||
"supplier_type": "TEXT",
|
||||
"source": "TEXT",
|
||||
"enriched_at": "DATETIME",
|
||||
"vulns_synced_at": "DATETIME",
|
||||
applied, err := db.appliedMigrations()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// If the migrations table didn't exist, create it now.
|
||||
if applied == nil {
|
||||
if err := db.createMigrationsTable(); err != nil {
|
||||
return err
|
||||
}
|
||||
applied = make(map[string]bool)
|
||||
}
|
||||
|
||||
for _, m := range migrations {
|
||||
if applied[m.name] {
|
||||
continue
|
||||
}
|
||||
if err := m.fn(db); err != nil {
|
||||
return fmt.Errorf("migration %s: %w", m.name, err)
|
||||
}
|
||||
if err := db.recordMigration(m.name); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func migrateAddPackagesEnrichmentColumns(db *DB) error {
|
||||
columns := map[string]string{
|
||||
"registry_url": "TEXT",
|
||||
"supplier_name": "TEXT",
|
||||
"supplier_type": "TEXT",
|
||||
"source": "TEXT",
|
||||
"enriched_at": "DATETIME",
|
||||
"vulns_synced_at": "DATETIME",
|
||||
}
|
||||
|
||||
if db.dialect == DialectPostgres {
|
||||
packagesColumns["enriched_at"] = postgresTimestamp
|
||||
packagesColumns["vulns_synced_at"] = postgresTimestamp
|
||||
columns["enriched_at"] = postgresTimestamp
|
||||
columns["vulns_synced_at"] = postgresTimestamp
|
||||
}
|
||||
|
||||
for column, colType := range packagesColumns {
|
||||
for column, colType := range columns {
|
||||
hasCol, err := db.HasColumn("packages", column)
|
||||
if err != nil {
|
||||
return fmt.Errorf("checking column %s: %w", column, err)
|
||||
|
|
@ -321,9 +451,11 @@ func (db *DB) MigrateSchema() error {
|
|||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Check and add missing columns to versions table
|
||||
versionsColumns := map[string]string{
|
||||
func migrateAddVersionsEnrichmentColumns(db *DB) error {
|
||||
columns := map[string]string{
|
||||
"integrity": "TEXT",
|
||||
"yanked": "INTEGER DEFAULT 0",
|
||||
"source": "TEXT",
|
||||
|
|
@ -331,11 +463,11 @@ func (db *DB) MigrateSchema() error {
|
|||
}
|
||||
|
||||
if db.dialect == DialectPostgres {
|
||||
versionsColumns["yanked"] = "BOOLEAN DEFAULT FALSE"
|
||||
versionsColumns["enriched_at"] = postgresTimestamp
|
||||
columns["yanked"] = "BOOLEAN DEFAULT FALSE"
|
||||
columns["enriched_at"] = postgresTimestamp
|
||||
}
|
||||
|
||||
for column, colType := range versionsColumns {
|
||||
for column, colType := range columns {
|
||||
hasCol, err := db.HasColumn("versions", column)
|
||||
if err != nil {
|
||||
return fmt.Errorf("checking column %s: %w", column, err)
|
||||
|
|
@ -347,62 +479,64 @@ func (db *DB) MigrateSchema() error {
|
|||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Ensure artifacts table exists
|
||||
if err := db.EnsureArtifactsTable(); err != nil {
|
||||
return fmt.Errorf("ensuring artifacts table: %w", err)
|
||||
}
|
||||
func migrateEnsureArtifactsTable(db *DB) error {
|
||||
return db.EnsureArtifactsTable()
|
||||
}
|
||||
|
||||
// Ensure vulnerabilities table exists
|
||||
func migrateEnsureVulnerabilitiesTable(db *DB) error {
|
||||
hasVulns, err := db.HasTable("vulnerabilities")
|
||||
if err != nil {
|
||||
return fmt.Errorf("checking vulnerabilities table: %w", err)
|
||||
}
|
||||
if !hasVulns {
|
||||
var vulnSchema string
|
||||
if db.dialect == DialectPostgres {
|
||||
vulnSchema = `
|
||||
CREATE TABLE vulnerabilities (
|
||||
id SERIAL PRIMARY KEY,
|
||||
vuln_id TEXT NOT NULL,
|
||||
ecosystem TEXT NOT NULL,
|
||||
package_name TEXT NOT NULL,
|
||||
severity TEXT,
|
||||
summary TEXT,
|
||||
fixed_version TEXT,
|
||||
cvss_score REAL,
|
||||
"references" TEXT,
|
||||
fetched_at TIMESTAMP,
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_vulns_id_pkg ON vulnerabilities(vuln_id, ecosystem, package_name);
|
||||
CREATE INDEX IF NOT EXISTS idx_vulns_ecosystem_pkg ON vulnerabilities(ecosystem, package_name);
|
||||
`
|
||||
} else {
|
||||
vulnSchema = `
|
||||
CREATE TABLE vulnerabilities (
|
||||
id INTEGER PRIMARY KEY,
|
||||
vuln_id TEXT NOT NULL,
|
||||
ecosystem TEXT NOT NULL,
|
||||
package_name TEXT NOT NULL,
|
||||
severity TEXT,
|
||||
summary TEXT,
|
||||
fixed_version TEXT,
|
||||
cvss_score REAL,
|
||||
"references" TEXT,
|
||||
fetched_at DATETIME,
|
||||
created_at DATETIME,
|
||||
updated_at DATETIME
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_vulns_id_pkg ON vulnerabilities(vuln_id, ecosystem, package_name);
|
||||
CREATE INDEX IF NOT EXISTS idx_vulns_ecosystem_pkg ON vulnerabilities(ecosystem, package_name);
|
||||
`
|
||||
}
|
||||
if _, err := db.Exec(vulnSchema); err != nil {
|
||||
return fmt.Errorf("creating vulnerabilities table: %w", err)
|
||||
}
|
||||
if hasVulns {
|
||||
return nil
|
||||
}
|
||||
|
||||
var vulnSchema string
|
||||
if db.dialect == DialectPostgres {
|
||||
vulnSchema = `
|
||||
CREATE TABLE vulnerabilities (
|
||||
id SERIAL PRIMARY KEY,
|
||||
vuln_id TEXT NOT NULL,
|
||||
ecosystem TEXT NOT NULL,
|
||||
package_name TEXT NOT NULL,
|
||||
severity TEXT,
|
||||
summary TEXT,
|
||||
fixed_version TEXT,
|
||||
cvss_score REAL,
|
||||
"references" TEXT,
|
||||
fetched_at TIMESTAMP,
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_vulns_id_pkg ON vulnerabilities(vuln_id, ecosystem, package_name);
|
||||
CREATE INDEX IF NOT EXISTS idx_vulns_ecosystem_pkg ON vulnerabilities(ecosystem, package_name);
|
||||
`
|
||||
} else {
|
||||
vulnSchema = `
|
||||
CREATE TABLE vulnerabilities (
|
||||
id INTEGER PRIMARY KEY,
|
||||
vuln_id TEXT NOT NULL,
|
||||
ecosystem TEXT NOT NULL,
|
||||
package_name TEXT NOT NULL,
|
||||
severity TEXT,
|
||||
summary TEXT,
|
||||
fixed_version TEXT,
|
||||
cvss_score REAL,
|
||||
"references" TEXT,
|
||||
fetched_at DATETIME,
|
||||
created_at DATETIME,
|
||||
updated_at DATETIME
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_vulns_id_pkg ON vulnerabilities(vuln_id, ecosystem, package_name);
|
||||
CREATE INDEX IF NOT EXISTS idx_vulns_ecosystem_pkg ON vulnerabilities(ecosystem, package_name);
|
||||
`
|
||||
}
|
||||
if _, err := db.Exec(vulnSchema); err != nil {
|
||||
return fmt.Errorf("creating vulnerabilities table: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import (
|
|||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"path"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
|
|
@ -128,7 +129,9 @@ func (h *ComposerHandler) handlePackageMetadata(w http.ResponseWriter, r *http.R
|
|||
}
|
||||
|
||||
// rewriteMetadata rewrites dist URLs in Composer metadata to point at this proxy.
|
||||
// If cooldown is enabled, versions published too recently are filtered out.
|
||||
// If the metadata uses the minified Composer v2 format, it is expanded first so
|
||||
// that every version entry contains all fields. If cooldown is enabled, versions
|
||||
// published too recently are filtered out.
|
||||
func (h *ComposerHandler) rewriteMetadata(body []byte) ([]byte, error) {
|
||||
var metadata map[string]any
|
||||
if err := json.Unmarshal(body, &metadata); err != nil {
|
||||
|
|
@ -140,18 +143,84 @@ func (h *ComposerHandler) rewriteMetadata(body []byte) ([]byte, error) {
|
|||
return body, nil
|
||||
}
|
||||
|
||||
minified := metadata["minified"] == "composer/2.0"
|
||||
|
||||
for packageName, versions := range packages {
|
||||
versionList, ok := versions.([]any)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
|
||||
if minified {
|
||||
versionList = expandMinifiedVersions(versionList)
|
||||
}
|
||||
|
||||
packages[packageName] = h.filterAndRewriteVersions(packageName, versionList)
|
||||
}
|
||||
|
||||
delete(metadata, "minified")
|
||||
|
||||
return json.Marshal(metadata)
|
||||
}
|
||||
|
||||
// expandMinifiedVersions expands the Composer v2 minified format where each
|
||||
// version entry only contains fields that differ from the previous entry.
|
||||
// The "~dev" sentinel string resets the inheritance chain.
|
||||
func expandMinifiedVersions(versionList []any) []any {
|
||||
expanded := make([]any, 0, len(versionList))
|
||||
inherited := map[string]any{}
|
||||
|
||||
for _, v := range versionList {
|
||||
// The "~dev" sentinel resets the inheritance chain for dev versions.
|
||||
if s, ok := v.(string); ok && s == "~dev" {
|
||||
inherited = map[string]any{}
|
||||
continue
|
||||
}
|
||||
|
||||
vmap, ok := v.(map[string]any)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
|
||||
// Merge inherited fields into a new map, then overlay current fields.
|
||||
// Deep copy values to avoid shared references between versions.
|
||||
merged := make(map[string]any, len(inherited)+len(vmap))
|
||||
for k, val := range inherited {
|
||||
merged[k] = deepCopyValue(val)
|
||||
}
|
||||
for k, val := range vmap {
|
||||
merged[k] = val
|
||||
}
|
||||
|
||||
// Update inherited state for next iteration.
|
||||
inherited = merged
|
||||
|
||||
expanded = append(expanded, merged)
|
||||
}
|
||||
|
||||
return expanded
|
||||
}
|
||||
|
||||
// deepCopyValue returns a deep copy of JSON-like values (maps, slices, scalars).
|
||||
func deepCopyValue(v any) any {
|
||||
switch val := v.(type) {
|
||||
case map[string]any:
|
||||
m := make(map[string]any, len(val))
|
||||
for k, v := range val {
|
||||
m[k] = deepCopyValue(v)
|
||||
}
|
||||
return m
|
||||
case []any:
|
||||
s := make([]any, len(val))
|
||||
for i, v := range val {
|
||||
s[i] = deepCopyValue(v)
|
||||
}
|
||||
return s
|
||||
default:
|
||||
return v
|
||||
}
|
||||
}
|
||||
|
||||
// filterAndRewriteVersions applies cooldown filtering and rewrites dist URLs
|
||||
// for a single package's version list.
|
||||
func (h *ComposerHandler) filterAndRewriteVersions(packageName string, versionList []any) []any {
|
||||
|
|
@ -219,6 +288,14 @@ func (h *ComposerHandler) rewriteDistURL(vmap map[string]any, packageName, versi
|
|||
filename = url[idx+1:]
|
||||
}
|
||||
|
||||
// GitHub zipball URLs end with a bare commit hash (no extension).
|
||||
// Append .zip so the archives library can detect the format.
|
||||
if path.Ext(filename) == "" {
|
||||
if distType, _ := dist["type"].(string); distType == "zip" {
|
||||
filename += ".zip"
|
||||
}
|
||||
}
|
||||
|
||||
parts := strings.SplitN(packageName, "/", vendorPackageParts)
|
||||
if len(parts) == vendorPackageParts {
|
||||
newURL := fmt.Sprintf("%s/composer/files/%s/%s/%s/%s",
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ package handler
|
|||
import (
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
|
|
@ -50,6 +51,346 @@ func TestComposerRewriteMetadata(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
func TestComposerRewriteMetadataExpandsMinified(t *testing.T) {
|
||||
h := &ComposerHandler{
|
||||
proxy: testProxy(),
|
||||
proxyURL: "http://localhost:8080",
|
||||
}
|
||||
|
||||
// Minified format: first version has all fields, subsequent versions
|
||||
// only include fields that changed. The proxy must expand this so every
|
||||
// version has all fields (including "name").
|
||||
input := `{
|
||||
"minified": "composer/2.0",
|
||||
"packages": {
|
||||
"symfony/console": [
|
||||
{
|
||||
"name": "symfony/console",
|
||||
"description": "Symfony Console Component",
|
||||
"version": "6.0.0",
|
||||
"dist": {
|
||||
"url": "https://repo.packagist.org/files/symfony/console/6.0.0/abc123.zip",
|
||||
"type": "zip"
|
||||
}
|
||||
},
|
||||
{
|
||||
"version": "5.4.0",
|
||||
"dist": {
|
||||
"url": "https://repo.packagist.org/files/symfony/console/5.4.0/def456.zip",
|
||||
"type": "zip"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}`
|
||||
|
||||
output, err := h.rewriteMetadata([]byte(input))
|
||||
if err != nil {
|
||||
t.Fatalf("rewriteMetadata failed: %v", err)
|
||||
}
|
||||
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(output, &result); err != nil {
|
||||
t.Fatalf("failed to parse output: %v", err)
|
||||
}
|
||||
|
||||
// The minified key should be removed from output
|
||||
if _, ok := result["minified"]; ok {
|
||||
t.Error("expected minified key to be removed from output")
|
||||
}
|
||||
|
||||
packages := result["packages"].(map[string]any)
|
||||
versions := packages["symfony/console"].([]any)
|
||||
|
||||
// Second version should have inherited the "name" and "description" fields
|
||||
v1 := versions[1].(map[string]any)
|
||||
if v1["name"] != "symfony/console" {
|
||||
t.Errorf("second version name = %v, want %q", v1["name"], "symfony/console")
|
||||
}
|
||||
if v1["description"] != "Symfony Console Component" {
|
||||
t.Errorf("second version description = %v, want %q", v1["description"], "Symfony Console Component")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComposerRewriteMetadataMinifiedDevReset(t *testing.T) {
|
||||
h := &ComposerHandler{
|
||||
proxy: testProxy(),
|
||||
proxyURL: "http://localhost:8080",
|
||||
}
|
||||
|
||||
// The ~dev sentinel resets the inheritance chain for dev versions.
|
||||
input := `{
|
||||
"minified": "composer/2.0",
|
||||
"packages": {
|
||||
"symfony/console": [
|
||||
{
|
||||
"name": "symfony/console",
|
||||
"description": "Symfony Console Component",
|
||||
"license": ["MIT"],
|
||||
"version": "6.0.0",
|
||||
"dist": {
|
||||
"url": "https://repo.packagist.org/files/symfony/console/6.0.0/abc123.zip",
|
||||
"type": "zip"
|
||||
}
|
||||
},
|
||||
"~dev",
|
||||
{
|
||||
"name": "symfony/console",
|
||||
"version": "dev-main",
|
||||
"dist": {
|
||||
"url": "https://repo.packagist.org/files/symfony/console/dev-main/xyz789.zip",
|
||||
"type": "zip"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}`
|
||||
|
||||
output, err := h.rewriteMetadata([]byte(input))
|
||||
if err != nil {
|
||||
t.Fatalf("rewriteMetadata failed: %v", err)
|
||||
}
|
||||
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(output, &result); err != nil {
|
||||
t.Fatalf("failed to parse output: %v", err)
|
||||
}
|
||||
|
||||
packages := result["packages"].(map[string]any)
|
||||
versions := packages["symfony/console"].([]any)
|
||||
|
||||
if len(versions) != 2 {
|
||||
t.Fatalf("expected 2 versions, got %d", len(versions))
|
||||
}
|
||||
|
||||
// Dev version should NOT have inherited "license" or "description"
|
||||
// from the tagged version (the ~dev sentinel resets inheritance).
|
||||
devVersion := versions[1].(map[string]any)
|
||||
if devVersion["version"] != "dev-main" {
|
||||
t.Errorf("dev version = %v, want %q", devVersion["version"], "dev-main")
|
||||
}
|
||||
if _, ok := devVersion["license"]; ok {
|
||||
t.Error("dev version should not have inherited license field after ~dev reset")
|
||||
}
|
||||
if _, ok := devVersion["description"]; ok {
|
||||
t.Error("dev version should not have inherited description field after ~dev reset")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComposerRewriteMetadataCooldownPreservesNames(t *testing.T) {
|
||||
now := time.Now()
|
||||
old := now.Add(-10 * 24 * time.Hour).Format(time.RFC3339)
|
||||
veryOld := now.Add(-20 * 24 * time.Hour).Format(time.RFC3339)
|
||||
recent := now.Add(-1 * time.Hour).Format(time.RFC3339)
|
||||
|
||||
proxy := &Proxy{Logger: slog.Default()}
|
||||
proxy.Cooldown = &cooldown.Config{Default: "3d"}
|
||||
|
||||
h := &ComposerHandler{
|
||||
proxy: proxy,
|
||||
proxyURL: "http://localhost:8080",
|
||||
}
|
||||
|
||||
// Minified format where "name" only appears in first version.
|
||||
// When cooldown filters the first version, remaining versions must
|
||||
// still have the "name" field after expansion.
|
||||
input := `{
|
||||
"minified": "composer/2.0",
|
||||
"packages": {
|
||||
"symfony/console": [
|
||||
{
|
||||
"name": "symfony/console",
|
||||
"description": "Symfony Console Component",
|
||||
"version": "7.0.0",
|
||||
"time": "` + recent + `",
|
||||
"dist": {"url": "https://repo.packagist.org/7.0.0.zip", "type": "zip"}
|
||||
},
|
||||
{
|
||||
"version": "6.0.0",
|
||||
"time": "` + old + `",
|
||||
"dist": {"url": "https://repo.packagist.org/6.0.0.zip", "type": "zip"}
|
||||
},
|
||||
{
|
||||
"version": "5.0.0",
|
||||
"time": "` + veryOld + `",
|
||||
"dist": {"url": "https://repo.packagist.org/5.0.0.zip", "type": "zip"}
|
||||
}
|
||||
]
|
||||
}
|
||||
}`
|
||||
|
||||
output, err := h.rewriteMetadata([]byte(input))
|
||||
if err != nil {
|
||||
t.Fatalf("rewriteMetadata failed: %v", err)
|
||||
}
|
||||
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(output, &result); err != nil {
|
||||
t.Fatalf("failed to parse output: %v", err)
|
||||
}
|
||||
|
||||
packages := result["packages"].(map[string]any)
|
||||
versions := packages["symfony/console"].([]any)
|
||||
|
||||
// v7.0.0 should be filtered by cooldown, leaving v6.0.0 and v5.0.0
|
||||
if len(versions) != 2 {
|
||||
t.Fatalf("expected 2 versions after cooldown, got %d", len(versions))
|
||||
}
|
||||
|
||||
// Both remaining versions must have the "name" field
|
||||
for _, v := range versions {
|
||||
vmap := v.(map[string]any)
|
||||
if vmap["name"] != "symfony/console" {
|
||||
t.Errorf("version %v missing name field, got %v", vmap["version"], vmap["name"])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestComposerRewriteDistURLGitHubZipball(t *testing.T) {
|
||||
// GitHub zipball URLs end with a bare commit hash, no file extension.
|
||||
// The proxy must produce a filename with .zip extension so that the
|
||||
// archives library can detect the format when browsing source.
|
||||
h := &ComposerHandler{
|
||||
proxy: testProxy(),
|
||||
proxyURL: "http://localhost:8080",
|
||||
}
|
||||
|
||||
vmap := map[string]any{
|
||||
"version": "v7.4.8",
|
||||
"dist": map[string]any{
|
||||
"url": "https://api.github.com/repos/symfony/asset/zipball/d2e2f014ccd6ec9fae8dbe6336a4164346a2a856",
|
||||
"type": "zip",
|
||||
"shasum": "",
|
||||
"reference": "d2e2f014ccd6ec9fae8dbe6336a4164346a2a856",
|
||||
},
|
||||
}
|
||||
|
||||
h.rewriteDistURL(vmap, "symfony/asset", "v7.4.8")
|
||||
|
||||
dist := vmap["dist"].(map[string]any)
|
||||
url := dist["url"].(string)
|
||||
|
||||
// The rewritten URL's filename must have a .zip extension
|
||||
if !strings.HasSuffix(url, ".zip") {
|
||||
t.Errorf("rewritten dist URL filename has no .zip extension: %s", url)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComposerRewriteMetadataGitHubZipballFilenames(t *testing.T) {
|
||||
// End-to-end: metadata with GitHub zipball URLs should produce
|
||||
// download URLs that end in .zip so browse source can open them.
|
||||
h := &ComposerHandler{
|
||||
proxy: testProxy(),
|
||||
proxyURL: "http://localhost:8080",
|
||||
}
|
||||
|
||||
input := `{
|
||||
"packages": {
|
||||
"symfony/config": [
|
||||
{
|
||||
"version": "v7.4.8",
|
||||
"dist": {
|
||||
"url": "https://api.github.com/repos/symfony/config/zipball/c7369cc1da250fcbfe0c5a9d109e419661549c39",
|
||||
"type": "zip",
|
||||
"reference": "c7369cc1da250fcbfe0c5a9d109e419661549c39"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}`
|
||||
|
||||
output, err := h.rewriteMetadata([]byte(input))
|
||||
if err != nil {
|
||||
t.Fatalf("rewriteMetadata failed: %v", err)
|
||||
}
|
||||
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(output, &result); err != nil {
|
||||
t.Fatalf("failed to parse output: %v", err)
|
||||
}
|
||||
|
||||
packages := result["packages"].(map[string]any)
|
||||
versions := packages["symfony/config"].([]any)
|
||||
v := versions[0].(map[string]any)
|
||||
dist := v["dist"].(map[string]any)
|
||||
url := dist["url"].(string)
|
||||
|
||||
if !strings.HasSuffix(url, ".zip") {
|
||||
t.Errorf("rewritten URL should end in .zip, got %s", url)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComposerExpandMinifiedSharedDistReferences(t *testing.T) {
|
||||
// When a minified version inherits the dist field from a previous version
|
||||
// (i.e. it doesn't include its own dist), expanding + rewriting must not
|
||||
// corrupt the dist URLs via shared map references.
|
||||
h := &ComposerHandler{
|
||||
proxy: testProxy(),
|
||||
proxyURL: "http://localhost:8080",
|
||||
}
|
||||
|
||||
// In this minified payload, v5.3.0 does NOT include a dist field,
|
||||
// so it inherits v5.4.0's dist. After expansion and URL rewriting,
|
||||
// each version must have its own correct dist URL.
|
||||
input := `{
|
||||
"minified": "composer/2.0",
|
||||
"packages": {
|
||||
"vendor/pkg": [
|
||||
{
|
||||
"name": "vendor/pkg",
|
||||
"version": "5.4.0",
|
||||
"dist": {
|
||||
"url": "https://api.github.com/repos/vendor/pkg/zipball/aaa111",
|
||||
"type": "zip",
|
||||
"reference": "aaa111"
|
||||
}
|
||||
},
|
||||
{
|
||||
"version": "5.3.0"
|
||||
}
|
||||
]
|
||||
}
|
||||
}`
|
||||
|
||||
output, err := h.rewriteMetadata([]byte(input))
|
||||
if err != nil {
|
||||
t.Fatalf("rewriteMetadata failed: %v", err)
|
||||
}
|
||||
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(output, &result); err != nil {
|
||||
t.Fatalf("failed to parse output: %v", err)
|
||||
}
|
||||
|
||||
packages := result["packages"].(map[string]any)
|
||||
versions := packages["vendor/pkg"].([]any)
|
||||
if len(versions) != 2 {
|
||||
t.Fatalf("expected 2 versions, got %d", len(versions))
|
||||
}
|
||||
|
||||
v1 := versions[0].(map[string]any)
|
||||
v2 := versions[1].(map[string]any)
|
||||
|
||||
dist1 := v1["dist"].(map[string]any)
|
||||
dist2 := v2["dist"].(map[string]any)
|
||||
|
||||
url1 := dist1["url"].(string)
|
||||
url2 := dist2["url"].(string)
|
||||
|
||||
// Each version must have its own URL with its own version in the path
|
||||
if !strings.Contains(url1, "/5.4.0/") {
|
||||
t.Errorf("v5.4.0 dist URL should contain /5.4.0/, got %s", url1)
|
||||
}
|
||||
if !strings.Contains(url2, "/5.3.0/") {
|
||||
t.Errorf("v5.3.0 dist URL should contain /5.3.0/, got %s", url2)
|
||||
}
|
||||
|
||||
// The two URLs must be different
|
||||
if url1 == url2 {
|
||||
t.Errorf("both versions have the same dist URL (shared reference bug): %s", url1)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComposerRewriteMetadataCooldown(t *testing.T) {
|
||||
now := time.Now()
|
||||
old := now.Add(-10 * 24 * time.Hour).Format(time.RFC3339)
|
||||
|
|
|
|||
|
|
@ -1,8 +1,13 @@
|
|||
package handler
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/git-pkgs/purl"
|
||||
)
|
||||
|
||||
const (
|
||||
|
|
@ -31,9 +36,9 @@ func (h *CondaHandler) Routes() http.Handler {
|
|||
mux := http.NewServeMux()
|
||||
|
||||
// Channel index (repodata)
|
||||
mux.HandleFunc("GET /{channel}/{arch}/repodata.json", h.proxyUpstream)
|
||||
mux.HandleFunc("GET /{channel}/{arch}/repodata.json", h.handleRepodata)
|
||||
mux.HandleFunc("GET /{channel}/{arch}/repodata.json.bz2", h.proxyUpstream)
|
||||
mux.HandleFunc("GET /{channel}/{arch}/current_repodata.json", h.proxyUpstream)
|
||||
mux.HandleFunc("GET /{channel}/{arch}/current_repodata.json", h.handleRepodata)
|
||||
|
||||
// Package downloads (cache these)
|
||||
mux.HandleFunc("GET /{channel}/{arch}/{filename}", h.handleDownload)
|
||||
|
|
@ -119,6 +124,114 @@ func (h *CondaHandler) parseFilename(filename string) (name, version string) {
|
|||
return name, version
|
||||
}
|
||||
|
||||
// handleRepodata proxies repodata.json, applying cooldown filtering when enabled.
|
||||
func (h *CondaHandler) handleRepodata(w http.ResponseWriter, r *http.Request) {
|
||||
if h.proxy.Cooldown == nil || !h.proxy.Cooldown.Enabled() {
|
||||
h.proxyUpstream(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
upstreamURL := h.upstreamURL + r.URL.Path
|
||||
|
||||
h.proxy.Logger.Debug("fetching repodata for cooldown filtering", "url", upstreamURL)
|
||||
|
||||
req, err := http.NewRequestWithContext(r.Context(), http.MethodGet, upstreamURL, nil)
|
||||
if err != nil {
|
||||
http.Error(w, "failed to create request", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
req.Header.Set("Accept-Encoding", "gzip")
|
||||
|
||||
resp, err := h.proxy.HTTPClient.Do(req)
|
||||
if err != nil {
|
||||
h.proxy.Logger.Error("upstream request failed", "error", err)
|
||||
http.Error(w, "upstream request failed", http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
for k, vv := range resp.Header {
|
||||
for _, v := range vv {
|
||||
w.Header().Add(k, v)
|
||||
}
|
||||
}
|
||||
w.WriteHeader(resp.StatusCode)
|
||||
_, _ = io.Copy(w, resp.Body)
|
||||
return
|
||||
}
|
||||
|
||||
body, err := ReadMetadata(resp.Body)
|
||||
if err != nil {
|
||||
http.Error(w, "failed to read response", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
filtered, err := h.applyCooldownFiltering(body)
|
||||
if err != nil {
|
||||
h.proxy.Logger.Warn("failed to filter repodata, proxying original", "error", err)
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write(body)
|
||||
return
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write(filtered)
|
||||
}
|
||||
|
||||
// condaTimestampDivisor converts Conda's millisecond timestamps to seconds.
|
||||
const condaTimestampDivisor = 1000
|
||||
|
||||
// applyCooldownFiltering removes entries from repodata.json that were
|
||||
// published too recently based on their timestamp field.
|
||||
func (h *CondaHandler) applyCooldownFiltering(body []byte) ([]byte, error) {
|
||||
if h.proxy.Cooldown == nil || !h.proxy.Cooldown.Enabled() {
|
||||
return body, nil
|
||||
}
|
||||
|
||||
var repodata map[string]any
|
||||
if err := json.Unmarshal(body, &repodata); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
for _, key := range []string{"packages", "packages.conda"} {
|
||||
packages, ok := repodata[key].(map[string]any)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
|
||||
for filename, entry := range packages {
|
||||
entryMap, ok := entry.(map[string]any)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
|
||||
ts, ok := entryMap["timestamp"].(float64)
|
||||
if !ok || ts == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
publishedAt := time.Unix(int64(ts)/condaTimestampDivisor, 0)
|
||||
|
||||
name, _ := entryMap["name"].(string)
|
||||
if name == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
packagePURL := purl.MakePURLString("conda", name, "")
|
||||
|
||||
if !h.proxy.Cooldown.IsAllowed("conda", packagePURL, publishedAt) {
|
||||
version, _ := entryMap["version"].(string)
|
||||
h.proxy.Logger.Info("cooldown: filtering conda package",
|
||||
"name", name, "version", version, "filename", filename)
|
||||
delete(packages, filename)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return json.Marshal(repodata)
|
||||
}
|
||||
|
||||
// proxyUpstream forwards a request to Anaconda without caching.
|
||||
func (h *CondaHandler) proxyUpstream(w http.ResponseWriter, r *http.Request) {
|
||||
h.proxy.ProxyUpstream(w, r, h.upstreamURL+r.URL.Path, []string{"Accept-Encoding"})
|
||||
|
|
|
|||
|
|
@ -1,8 +1,14 @@
|
|||
package handler
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/git-pkgs/proxy/internal/cooldown"
|
||||
)
|
||||
|
||||
func TestCondaParseFilename(t *testing.T) {
|
||||
|
|
@ -49,3 +55,251 @@ func TestCondaIsPackageFile(t *testing.T) {
|
|||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCondaCooldownFiltering(t *testing.T) {
|
||||
now := time.Now()
|
||||
oldTimestamp := float64(now.Add(-7*24*time.Hour).UnixMilli())
|
||||
recentTimestamp := float64(now.Add(-1*time.Hour).UnixMilli())
|
||||
|
||||
repodata := map[string]any{
|
||||
"info": map[string]any{},
|
||||
"packages": map[string]any{
|
||||
"numpy-1.24.0-old.tar.bz2": map[string]any{
|
||||
"name": "numpy",
|
||||
"version": "1.24.0",
|
||||
"timestamp": oldTimestamp,
|
||||
},
|
||||
"numpy-1.25.0-new.tar.bz2": map[string]any{
|
||||
"name": "numpy",
|
||||
"version": "1.25.0",
|
||||
"timestamp": recentTimestamp,
|
||||
},
|
||||
},
|
||||
"packages.conda": map[string]any{
|
||||
"scipy-1.11.0-old.conda": map[string]any{
|
||||
"name": "scipy",
|
||||
"version": "1.11.0",
|
||||
"timestamp": oldTimestamp,
|
||||
},
|
||||
"scipy-1.12.0-new.conda": map[string]any{
|
||||
"name": "scipy",
|
||||
"version": "1.12.0",
|
||||
"timestamp": recentTimestamp,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
body, err := json.Marshal(repodata)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
proxy := testProxy()
|
||||
proxy.Cooldown = &cooldown.Config{
|
||||
Default: "3d",
|
||||
}
|
||||
|
||||
h := &CondaHandler{
|
||||
proxy: proxy,
|
||||
proxyURL: "http://localhost:8080",
|
||||
}
|
||||
|
||||
filtered, err := h.applyCooldownFiltering(body)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(filtered, &result); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
packages := result["packages"].(map[string]any)
|
||||
if len(packages) != 1 {
|
||||
t.Fatalf("expected 1 package in packages, got %d", len(packages))
|
||||
}
|
||||
if _, ok := packages["numpy-1.24.0-old.tar.bz2"]; !ok {
|
||||
t.Error("expected old numpy to survive filtering")
|
||||
}
|
||||
|
||||
condaPkgs := result["packages.conda"].(map[string]any)
|
||||
if len(condaPkgs) != 1 {
|
||||
t.Fatalf("expected 1 package in packages.conda, got %d", len(condaPkgs))
|
||||
}
|
||||
if _, ok := condaPkgs["scipy-1.11.0-old.conda"]; !ok {
|
||||
t.Error("expected old scipy to survive filtering")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCondaCooldownFilteringWithPackageOverride(t *testing.T) {
|
||||
now := time.Now()
|
||||
recentTimestamp := float64(now.Add(-2 * time.Hour).UnixMilli())
|
||||
|
||||
repodata := map[string]any{
|
||||
"info": map[string]any{},
|
||||
"packages": map[string]any{
|
||||
"special-1.0.0-build.tar.bz2": map[string]any{
|
||||
"name": "special",
|
||||
"version": "1.0.0",
|
||||
"timestamp": recentTimestamp,
|
||||
},
|
||||
},
|
||||
"packages.conda": map[string]any{},
|
||||
}
|
||||
|
||||
body, err := json.Marshal(repodata)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
proxy := testProxy()
|
||||
proxy.Cooldown = &cooldown.Config{
|
||||
Default: "3d",
|
||||
Packages: map[string]string{"pkg:conda/special": "1h"},
|
||||
}
|
||||
|
||||
h := &CondaHandler{
|
||||
proxy: proxy,
|
||||
proxyURL: "http://localhost:8080",
|
||||
}
|
||||
|
||||
filtered, err := h.applyCooldownFiltering(body)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(filtered, &result); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
packages := result["packages"].(map[string]any)
|
||||
if len(packages) != 1 {
|
||||
t.Fatalf("expected 1 package (override allows it), got %d", len(packages))
|
||||
}
|
||||
}
|
||||
|
||||
func TestCondaCooldownFilteringNoTimestamp(t *testing.T) {
|
||||
repodata := map[string]any{
|
||||
"info": map[string]any{},
|
||||
"packages": map[string]any{
|
||||
"old-pkg-1.0.0-build.tar.bz2": map[string]any{
|
||||
"name": "old-pkg",
|
||||
"version": "1.0.0",
|
||||
// no timestamp field
|
||||
},
|
||||
},
|
||||
"packages.conda": map[string]any{},
|
||||
}
|
||||
|
||||
body, err := json.Marshal(repodata)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
proxy := testProxy()
|
||||
proxy.Cooldown = &cooldown.Config{
|
||||
Default: "3d",
|
||||
}
|
||||
|
||||
h := &CondaHandler{
|
||||
proxy: proxy,
|
||||
proxyURL: "http://localhost:8080",
|
||||
}
|
||||
|
||||
filtered, err := h.applyCooldownFiltering(body)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(filtered, &result); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
packages := result["packages"].(map[string]any)
|
||||
if len(packages) != 1 {
|
||||
t.Fatalf("entries without timestamp should pass through, got %d", len(packages))
|
||||
}
|
||||
}
|
||||
|
||||
func TestCondaHandleRepodataWithCooldown(t *testing.T) {
|
||||
now := time.Now()
|
||||
oldTimestamp := float64(now.Add(-7 * 24 * time.Hour).UnixMilli())
|
||||
recentTimestamp := float64(now.Add(-1 * time.Hour).UnixMilli())
|
||||
|
||||
repodataJSON, _ := json.Marshal(map[string]any{
|
||||
"info": map[string]any{},
|
||||
"packages": map[string]any{
|
||||
"old-1.0.0-build.tar.bz2": map[string]any{
|
||||
"name": "testpkg", "version": "1.0.0", "timestamp": oldTimestamp,
|
||||
},
|
||||
"new-2.0.0-build.tar.bz2": map[string]any{
|
||||
"name": "testpkg", "version": "2.0.0", "timestamp": recentTimestamp,
|
||||
},
|
||||
},
|
||||
"packages.conda": map[string]any{},
|
||||
})
|
||||
|
||||
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write(repodataJSON)
|
||||
}))
|
||||
defer upstream.Close()
|
||||
|
||||
proxy := testProxy()
|
||||
proxy.Cooldown = &cooldown.Config{
|
||||
Default: "3d",
|
||||
}
|
||||
|
||||
h := &CondaHandler{
|
||||
proxy: proxy,
|
||||
upstreamURL: upstream.URL,
|
||||
proxyURL: "http://proxy.local",
|
||||
}
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/conda-forge/noarch/repodata.json", nil)
|
||||
req.SetPathValue("channel", "conda-forge")
|
||||
req.SetPathValue("arch", "noarch")
|
||||
w := httptest.NewRecorder()
|
||||
h.handleRepodata(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", w.Code, http.StatusOK)
|
||||
}
|
||||
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &result); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
packages := result["packages"].(map[string]any)
|
||||
if len(packages) != 1 {
|
||||
t.Fatalf("expected 1 package after filtering, got %d", len(packages))
|
||||
}
|
||||
}
|
||||
|
||||
func TestCondaHandleRepodataWithoutCooldown(t *testing.T) {
|
||||
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"info":{},"packages":{},"packages.conda":{}}`))
|
||||
}))
|
||||
defer upstream.Close()
|
||||
|
||||
h := &CondaHandler{
|
||||
proxy: &Proxy{Logger: slog.Default(), HTTPClient: http.DefaultClient},
|
||||
upstreamURL: upstream.URL,
|
||||
proxyURL: "http://proxy.local",
|
||||
}
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/conda-forge/noarch/repodata.json", nil)
|
||||
req.SetPathValue("channel", "conda-forge")
|
||||
req.SetPathValue("arch", "noarch")
|
||||
w := httptest.NewRecorder()
|
||||
h.handleRepodata(w, req)
|
||||
|
||||
// Without cooldown, should proxy directly (response comes from upstream)
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", w.Code, http.StatusOK)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,10 +1,15 @@
|
|||
package handler
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/git-pkgs/purl"
|
||||
)
|
||||
|
||||
const (
|
||||
|
|
@ -41,7 +46,7 @@ func (h *GemHandler) Routes() http.Handler {
|
|||
|
||||
// Compact index (bundler 2.x+)
|
||||
mux.HandleFunc("GET /versions", h.proxyUpstream)
|
||||
mux.HandleFunc("GET /info/{name}", h.proxyUpstream)
|
||||
mux.HandleFunc("GET /info/{name}", h.handleCompactIndex)
|
||||
|
||||
// Quick index
|
||||
mux.HandleFunc("GET /quick/Marshal.4.8/{filename}", h.proxyUpstream)
|
||||
|
|
@ -98,6 +103,191 @@ func (h *GemHandler) parseGemFilename(filename string) (name, version string) {
|
|||
return "", ""
|
||||
}
|
||||
|
||||
// handleCompactIndex serves the compact index for a gem, filtering versions
|
||||
// based on cooldown when enabled.
|
||||
func (h *GemHandler) handleCompactIndex(w http.ResponseWriter, r *http.Request) {
|
||||
if h.proxy.Cooldown == nil || !h.proxy.Cooldown.Enabled() {
|
||||
h.proxyUpstream(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
name := r.PathValue("name")
|
||||
if name == "" {
|
||||
http.Error(w, "invalid gem name", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
h.proxy.Logger.Info("gem compact index request with cooldown", "name", name)
|
||||
|
||||
indexResp, filteredVersions, err := h.fetchIndexAndVersions(r, name)
|
||||
if err != nil {
|
||||
h.proxy.Logger.Error("upstream compact index request failed", "error", err)
|
||||
http.Error(w, "upstream request failed", http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
defer func() { _ = indexResp.Body.Close() }()
|
||||
|
||||
if indexResp.StatusCode != http.StatusOK {
|
||||
copyResponseHeaders(w, indexResp.Header)
|
||||
w.WriteHeader(indexResp.StatusCode)
|
||||
_, _ = io.Copy(w, indexResp.Body)
|
||||
return
|
||||
}
|
||||
|
||||
if filteredVersions == nil {
|
||||
h.proxy.Logger.Warn("failed to fetch version timestamps, proxying unfiltered", "name", name)
|
||||
copyResponseHeaders(w, indexResp.Header)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = io.Copy(w, indexResp.Body)
|
||||
return
|
||||
}
|
||||
|
||||
h.writeFilteredIndex(w, indexResp, name, filteredVersions)
|
||||
}
|
||||
|
||||
// fetchIndexAndVersions fetches the compact index and versions API concurrently.
|
||||
// Returns the index response, a set of versions to filter (nil if versions API failed),
|
||||
// and an error if the index fetch itself failed.
|
||||
func (h *GemHandler) fetchIndexAndVersions(r *http.Request, name string) (*http.Response, map[string]bool, error) {
|
||||
type versionsResult struct {
|
||||
filtered map[string]bool
|
||||
err error
|
||||
}
|
||||
|
||||
versionsCh := make(chan versionsResult, 1)
|
||||
go func() {
|
||||
filtered, err := h.fetchFilteredVersions(r, name)
|
||||
versionsCh <- versionsResult{filtered: filtered, err: err}
|
||||
}()
|
||||
|
||||
indexResp, err := h.fetchCompactIndex(r, name)
|
||||
|
||||
versionsRes := <-versionsCh
|
||||
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
if versionsRes.err != nil {
|
||||
return indexResp, nil, nil
|
||||
}
|
||||
|
||||
return indexResp, versionsRes.filtered, nil
|
||||
}
|
||||
|
||||
// fetchCompactIndex fetches the compact index from upstream.
|
||||
func (h *GemHandler) fetchCompactIndex(r *http.Request, name string) (*http.Response, error) {
|
||||
indexURL := h.upstreamURL + "/info/" + name
|
||||
req, err := http.NewRequestWithContext(r.Context(), http.MethodGet, indexURL, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, hdr := range []string{"Accept", "Accept-Encoding", "If-None-Match", "If-Modified-Since"} {
|
||||
if v := r.Header.Get(hdr); v != "" {
|
||||
req.Header.Set(hdr, v)
|
||||
}
|
||||
}
|
||||
return h.proxy.HTTPClient.Do(req)
|
||||
}
|
||||
|
||||
// writeFilteredIndex writes the compact index response with cooldown-filtered versions removed.
|
||||
func (h *GemHandler) writeFilteredIndex(w http.ResponseWriter, resp *http.Response, name string, filtered map[string]bool) {
|
||||
for k, vv := range resp.Header {
|
||||
if strings.EqualFold(k, "Content-Length") {
|
||||
continue // length will change after filtering
|
||||
}
|
||||
for _, v := range vv {
|
||||
w.Header().Add(k, v)
|
||||
}
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
|
||||
scanner := bufio.NewScanner(resp.Body)
|
||||
for scanner.Scan() {
|
||||
line := scanner.Text()
|
||||
|
||||
if line == "---" {
|
||||
_, _ = fmt.Fprintln(w, line)
|
||||
continue
|
||||
}
|
||||
|
||||
version := line
|
||||
if spaceIdx := strings.IndexByte(line, ' '); spaceIdx > 0 {
|
||||
version = line[:spaceIdx]
|
||||
}
|
||||
|
||||
if filtered[version] {
|
||||
h.proxy.Logger.Info("cooldown: filtering gem version",
|
||||
"gem", name, "version", version)
|
||||
continue
|
||||
}
|
||||
|
||||
_, _ = fmt.Fprintln(w, line)
|
||||
}
|
||||
}
|
||||
|
||||
// copyResponseHeaders copies HTTP headers from a response to a writer.
|
||||
func copyResponseHeaders(w http.ResponseWriter, headers http.Header) {
|
||||
for k, vv := range headers {
|
||||
for _, v := range vv {
|
||||
w.Header().Add(k, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// gemVersion represents a version entry from the RubyGems versions API.
|
||||
type gemVersion struct {
|
||||
Number string `json:"number"`
|
||||
Platform string `json:"platform"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
}
|
||||
|
||||
// fetchFilteredVersions fetches the versions API and returns a set of version
|
||||
// strings that should be filtered out by cooldown.
|
||||
func (h *GemHandler) fetchFilteredVersions(r *http.Request, name string) (map[string]bool, error) {
|
||||
versionsURL := fmt.Sprintf("%s/api/v1/versions/%s.json", h.upstreamURL, name)
|
||||
req, err := http.NewRequestWithContext(r.Context(), http.MethodGet, versionsURL, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
resp, err := h.proxy.HTTPClient.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, fmt.Errorf("versions API returned %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
var versions []gemVersion
|
||||
if err := json.NewDecoder(resp.Body).Decode(&versions); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
packagePURL := purl.MakePURLString("gem", name, "")
|
||||
filtered := make(map[string]bool)
|
||||
|
||||
for _, v := range versions {
|
||||
createdAt, err := time.Parse(time.RFC3339, v.CreatedAt)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
if !h.proxy.Cooldown.IsAllowed("gem", packagePURL, createdAt) {
|
||||
// Build version string matching compact index format
|
||||
versionStr := v.Number
|
||||
if v.Platform != "" && v.Platform != "ruby" {
|
||||
versionStr = v.Number + "-" + v.Platform
|
||||
}
|
||||
filtered[versionStr] = true
|
||||
}
|
||||
}
|
||||
|
||||
return filtered, nil
|
||||
}
|
||||
|
||||
// proxyUpstream forwards a request to rubygems.org without caching.
|
||||
func (h *GemHandler) proxyUpstream(w http.ResponseWriter, r *http.Request) {
|
||||
upstreamURL := h.upstreamURL + r.URL.Path
|
||||
|
|
|
|||
|
|
@ -1,8 +1,16 @@
|
|||
package handler
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/git-pkgs/proxy/internal/cooldown"
|
||||
)
|
||||
|
||||
func TestGemParseFilename(t *testing.T) {
|
||||
|
|
@ -28,3 +36,217 @@ func TestGemParseFilename(t *testing.T) {
|
|||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGemCompactIndexCooldown(t *testing.T) {
|
||||
now := time.Now()
|
||||
oldTime := now.Add(-7 * 24 * time.Hour).Format(time.RFC3339)
|
||||
recentTime := now.Add(-1 * time.Hour).Format(time.RFC3339)
|
||||
|
||||
compactIndex := "---\n1.0.0 dep1:>= 1.0|checksum:abc123\n2.0.0 dep1:>= 1.0|checksum:def456\n"
|
||||
|
||||
versionsJSON, _ := json.Marshal([]gemVersion{
|
||||
{Number: "1.0.0", Platform: "ruby", CreatedAt: oldTime},
|
||||
{Number: "2.0.0", Platform: "ruby", CreatedAt: recentTime},
|
||||
})
|
||||
|
||||
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case strings.HasPrefix(r.URL.Path, "/info/"):
|
||||
w.Header().Set("Content-Type", "text/plain")
|
||||
_, _ = w.Write([]byte(compactIndex))
|
||||
case strings.HasPrefix(r.URL.Path, "/api/v1/versions/"):
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write(versionsJSON)
|
||||
default:
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
}
|
||||
}))
|
||||
defer upstream.Close()
|
||||
|
||||
proxy := testProxy()
|
||||
proxy.Cooldown = &cooldown.Config{
|
||||
Default: "3d",
|
||||
}
|
||||
|
||||
h := &GemHandler{
|
||||
proxy: proxy,
|
||||
upstreamURL: upstream.URL,
|
||||
proxyURL: "http://proxy.local",
|
||||
}
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/info/testgem", nil)
|
||||
req.SetPathValue("name", "testgem")
|
||||
w := httptest.NewRecorder()
|
||||
h.handleCompactIndex(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", w.Code, http.StatusOK)
|
||||
}
|
||||
|
||||
body := w.Body.String()
|
||||
if !strings.Contains(body, "1.0.0") {
|
||||
t.Error("expected version 1.0.0 to survive filtering")
|
||||
}
|
||||
if strings.Contains(body, "2.0.0") {
|
||||
t.Error("expected version 2.0.0 to be filtered out")
|
||||
}
|
||||
if !strings.HasPrefix(body, "---\n") {
|
||||
t.Error("expected compact index header to be preserved")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGemCompactIndexCooldownWithPlatformVersion(t *testing.T) {
|
||||
now := time.Now()
|
||||
recentTime := now.Add(-1 * time.Hour).Format(time.RFC3339)
|
||||
|
||||
compactIndex := "---\n1.0.0 dep:>= 1.0|checksum:abc\n1.0.0-java dep:>= 1.0|checksum:def\n"
|
||||
|
||||
versionsJSON, _ := json.Marshal([]gemVersion{
|
||||
{Number: "1.0.0", Platform: "ruby", CreatedAt: recentTime},
|
||||
{Number: "1.0.0", Platform: "java", CreatedAt: recentTime},
|
||||
})
|
||||
|
||||
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case strings.HasPrefix(r.URL.Path, "/info/"):
|
||||
_, _ = w.Write([]byte(compactIndex))
|
||||
case strings.HasPrefix(r.URL.Path, "/api/v1/versions/"):
|
||||
_, _ = w.Write(versionsJSON)
|
||||
}
|
||||
}))
|
||||
defer upstream.Close()
|
||||
|
||||
proxy := testProxy()
|
||||
proxy.Cooldown = &cooldown.Config{
|
||||
Default: "3d",
|
||||
}
|
||||
|
||||
h := &GemHandler{
|
||||
proxy: proxy,
|
||||
upstreamURL: upstream.URL,
|
||||
proxyURL: "http://proxy.local",
|
||||
}
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/info/testgem", nil)
|
||||
req.SetPathValue("name", "testgem")
|
||||
w := httptest.NewRecorder()
|
||||
h.handleCompactIndex(w, req)
|
||||
|
||||
body := w.Body.String()
|
||||
// Both ruby and java platform versions should be filtered
|
||||
lines := strings.Split(strings.TrimSpace(body), "\n")
|
||||
if len(lines) != 1 { // only "---"
|
||||
t.Errorf("expected only header line, got %d lines: %v", len(lines), lines)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGemCompactIndexNoCooldown(t *testing.T) {
|
||||
compactIndex := "---\n1.0.0 dep:>= 1.0|checksum:abc\n"
|
||||
|
||||
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
_, _ = w.Write([]byte(compactIndex))
|
||||
}))
|
||||
defer upstream.Close()
|
||||
|
||||
h := &GemHandler{
|
||||
proxy: testProxy(), // no cooldown
|
||||
upstreamURL: upstream.URL,
|
||||
proxyURL: "http://proxy.local",
|
||||
}
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/info/testgem", nil)
|
||||
req.SetPathValue("name", "testgem")
|
||||
w := httptest.NewRecorder()
|
||||
h.handleCompactIndex(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", w.Code, http.StatusOK)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGemCompactIndexVersionsAPIFails(t *testing.T) {
|
||||
compactIndex := "---\n1.0.0 dep:>= 1.0|checksum:abc\n"
|
||||
|
||||
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case strings.HasPrefix(r.URL.Path, "/info/"):
|
||||
_, _ = w.Write([]byte(compactIndex))
|
||||
case strings.HasPrefix(r.URL.Path, "/api/v1/versions/"):
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
}
|
||||
}))
|
||||
defer upstream.Close()
|
||||
|
||||
proxy := testProxy()
|
||||
proxy.Cooldown = &cooldown.Config{
|
||||
Default: "3d",
|
||||
}
|
||||
|
||||
h := &GemHandler{
|
||||
proxy: proxy,
|
||||
upstreamURL: upstream.URL,
|
||||
proxyURL: "http://proxy.local",
|
||||
}
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/info/testgem", nil)
|
||||
req.SetPathValue("name", "testgem")
|
||||
w := httptest.NewRecorder()
|
||||
h.handleCompactIndex(w, req)
|
||||
|
||||
// Should still return OK with unfiltered content
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", w.Code, http.StatusOK)
|
||||
}
|
||||
|
||||
body := w.Body.String()
|
||||
if !strings.Contains(body, "1.0.0") {
|
||||
t.Error("expected unfiltered content when versions API fails")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGemFetchFilteredVersions(t *testing.T) {
|
||||
now := time.Now()
|
||||
oldTime := now.Add(-7 * 24 * time.Hour).Format(time.RFC3339)
|
||||
recentTime := now.Add(-1 * time.Hour).Format(time.RFC3339)
|
||||
|
||||
versionsJSON, _ := json.Marshal([]gemVersion{
|
||||
{Number: "1.0.0", Platform: "ruby", CreatedAt: oldTime},
|
||||
{Number: "2.0.0", Platform: "ruby", CreatedAt: recentTime},
|
||||
{Number: "2.0.0", Platform: "java", CreatedAt: recentTime},
|
||||
})
|
||||
|
||||
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write(versionsJSON)
|
||||
}))
|
||||
defer upstream.Close()
|
||||
|
||||
proxy := testProxy()
|
||||
proxy.Cooldown = &cooldown.Config{
|
||||
Default: "3d",
|
||||
}
|
||||
|
||||
h := &GemHandler{
|
||||
proxy: proxy,
|
||||
upstreamURL: upstream.URL,
|
||||
proxyURL: "http://proxy.local",
|
||||
}
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/info/testgem", nil)
|
||||
filtered, err := h.fetchFilteredVersions(req, "testgem")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if filtered["1.0.0"] {
|
||||
t.Error("version 1.0.0 should not be filtered (old enough)")
|
||||
}
|
||||
if !filtered["2.0.0"] {
|
||||
t.Error("version 2.0.0 (ruby) should be filtered")
|
||||
}
|
||||
if !filtered["2.0.0-java"] {
|
||||
t.Error("version 2.0.0-java should be filtered")
|
||||
}
|
||||
|
||||
_ = fmt.Sprintf // silence unused import
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,8 +1,17 @@
|
|||
package handler
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/git-pkgs/purl"
|
||||
"google.golang.org/protobuf/encoding/protowire"
|
||||
)
|
||||
|
||||
const (
|
||||
|
|
@ -35,7 +44,7 @@ func (h *HexHandler) Routes() http.Handler {
|
|||
// Registry resources (proxy without caching)
|
||||
mux.HandleFunc("GET /names", h.proxyUpstream)
|
||||
mux.HandleFunc("GET /versions", h.proxyUpstream)
|
||||
mux.HandleFunc("GET /packages/{name}", h.proxyUpstream)
|
||||
mux.HandleFunc("GET /packages/{name}", h.handlePackages)
|
||||
|
||||
// Public keys
|
||||
mux.HandleFunc("GET /public_key", h.proxyUpstream)
|
||||
|
|
@ -85,6 +94,329 @@ func (h *HexHandler) parseTarballFilename(filename string) (name, version string
|
|||
return "", ""
|
||||
}
|
||||
|
||||
// hexAPIURL is the Hex HTTP API base URL for fetching package metadata with timestamps.
|
||||
const hexAPIURL = "https://hex.pm"
|
||||
|
||||
// handlePackages proxies the /packages/{name} endpoint, applying cooldown filtering
|
||||
// when enabled. Since the protobuf format has no timestamps, we fetch them from the
|
||||
// Hex HTTP API concurrently.
|
||||
func (h *HexHandler) handlePackages(w http.ResponseWriter, r *http.Request) {
|
||||
if h.proxy.Cooldown == nil || !h.proxy.Cooldown.Enabled() {
|
||||
h.proxyUpstream(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
name := r.PathValue("name")
|
||||
if name == "" {
|
||||
h.proxyUpstream(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
h.proxy.Logger.Info("hex package request with cooldown", "name", name)
|
||||
|
||||
protoResp, filteredVersions, err := h.fetchPackageAndVersions(r, name)
|
||||
if err != nil {
|
||||
h.proxy.Logger.Error("upstream request failed", "error", err)
|
||||
http.Error(w, "upstream request failed", http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
defer func() { _ = protoResp.Body.Close() }()
|
||||
|
||||
if protoResp.StatusCode != http.StatusOK {
|
||||
for k, vv := range protoResp.Header {
|
||||
for _, v := range vv {
|
||||
w.Header().Add(k, v)
|
||||
}
|
||||
}
|
||||
w.WriteHeader(protoResp.StatusCode)
|
||||
_, _ = io.Copy(w, protoResp.Body)
|
||||
return
|
||||
}
|
||||
|
||||
body, err := io.ReadAll(protoResp.Body)
|
||||
if err != nil {
|
||||
http.Error(w, "failed to read response", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
if len(filteredVersions) == 0 {
|
||||
// No versions to filter or couldn't get timestamps, pass through
|
||||
w.Header().Set("Content-Type", protoResp.Header.Get("Content-Type"))
|
||||
w.Header().Set("Content-Encoding", "gzip")
|
||||
_, _ = w.Write(body)
|
||||
return
|
||||
}
|
||||
|
||||
filtered, err := h.filterSignedPackage(body, filteredVersions)
|
||||
if err != nil {
|
||||
h.proxy.Logger.Warn("failed to filter hex package, proxying original", "error", err)
|
||||
w.Header().Set("Content-Type", protoResp.Header.Get("Content-Type"))
|
||||
w.Header().Set("Content-Encoding", "gzip")
|
||||
_, _ = w.Write(body)
|
||||
return
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/octet-stream")
|
||||
w.Header().Set("Content-Encoding", "gzip")
|
||||
_, _ = w.Write(filtered)
|
||||
}
|
||||
|
||||
// fetchPackageAndVersions fetches the protobuf package and version timestamps concurrently.
|
||||
func (h *HexHandler) fetchPackageAndVersions(r *http.Request, name string) (*http.Response, map[string]bool, error) {
|
||||
type versionsResult struct {
|
||||
filtered map[string]bool
|
||||
err error
|
||||
}
|
||||
|
||||
versionsCh := make(chan versionsResult, 1)
|
||||
go func() {
|
||||
filtered, err := h.fetchFilteredVersions(r, name)
|
||||
versionsCh <- versionsResult{filtered: filtered, err: err}
|
||||
}()
|
||||
|
||||
protoResp, err := h.fetchUpstreamPackage(r, name)
|
||||
|
||||
versionsRes := <-versionsCh
|
||||
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
if versionsRes.err != nil {
|
||||
h.proxy.Logger.Warn("failed to fetch hex version timestamps, proxying unfiltered",
|
||||
"name", name, "error", versionsRes.err)
|
||||
return protoResp, nil, nil
|
||||
}
|
||||
|
||||
return protoResp, versionsRes.filtered, nil
|
||||
}
|
||||
|
||||
// fetchUpstreamPackage fetches the protobuf package from upstream.
|
||||
func (h *HexHandler) fetchUpstreamPackage(r *http.Request, name string) (*http.Response, error) {
|
||||
upstreamURL := h.upstreamURL + "/packages/" + name
|
||||
req, err := http.NewRequestWithContext(r.Context(), http.MethodGet, upstreamURL, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return h.proxy.HTTPClient.Do(req)
|
||||
}
|
||||
|
||||
// hexRelease represents a version entry from the Hex API.
|
||||
type hexRelease struct {
|
||||
Version string `json:"version"`
|
||||
InsertedAt string `json:"inserted_at"`
|
||||
}
|
||||
|
||||
// hexPackageAPI represents the Hex API response for a package.
|
||||
type hexPackageAPI struct {
|
||||
Releases []hexRelease `json:"releases"`
|
||||
}
|
||||
|
||||
// fetchFilteredVersions fetches the Hex API and returns a set of version
|
||||
// strings that should be filtered out by cooldown.
|
||||
func (h *HexHandler) fetchFilteredVersions(r *http.Request, name string) (map[string]bool, error) {
|
||||
apiURL := fmt.Sprintf("%s/api/packages/%s", hexAPIURL, name)
|
||||
req, err := http.NewRequestWithContext(r.Context(), http.MethodGet, apiURL, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
resp, err := h.proxy.HTTPClient.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, fmt.Errorf("hex API returned %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
var pkg hexPackageAPI
|
||||
if err := json.NewDecoder(resp.Body).Decode(&pkg); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
packagePURL := purl.MakePURLString("hex", name, "")
|
||||
filtered := make(map[string]bool)
|
||||
|
||||
for _, release := range pkg.Releases {
|
||||
insertedAt, err := time.Parse(time.RFC3339Nano, release.InsertedAt)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
if !h.proxy.Cooldown.IsAllowed("hex", packagePURL, insertedAt) {
|
||||
filtered[release.Version] = true
|
||||
h.proxy.Logger.Info("cooldown: filtering hex version",
|
||||
"package", name, "version", release.Version,
|
||||
"published", release.InsertedAt)
|
||||
}
|
||||
}
|
||||
|
||||
return filtered, nil
|
||||
}
|
||||
|
||||
// filterSignedPackage decompresses gzipped data, decodes the Signed protobuf wrapper,
|
||||
// filters releases from the Package payload, and re-encodes as gzipped protobuf
|
||||
// (without the original signature since the payload has changed).
|
||||
func (h *HexHandler) filterSignedPackage(gzippedData []byte, filteredVersions map[string]bool) ([]byte, error) {
|
||||
// Decompress gzip
|
||||
gr, err := gzip.NewReader(bytes.NewReader(gzippedData))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
signed, err := io.ReadAll(gr)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
_ = gr.Close()
|
||||
|
||||
// Parse Signed message: field 1 = payload (bytes), field 2 = signature (bytes)
|
||||
payload, err := extractProtobufBytes(signed, 1)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("extracting payload: %w", err)
|
||||
}
|
||||
|
||||
// Filter releases from the Package message
|
||||
filteredPayload, err := filterPackageReleases(payload, filteredVersions)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("filtering releases: %w", err)
|
||||
}
|
||||
|
||||
// Re-encode Signed message with modified payload and no signature
|
||||
var newSigned []byte
|
||||
newSigned = protowire.AppendTag(newSigned, 1, protowire.BytesType)
|
||||
newSigned = protowire.AppendBytes(newSigned, filteredPayload)
|
||||
|
||||
// Gzip compress
|
||||
var buf bytes.Buffer
|
||||
gw := gzip.NewWriter(&buf)
|
||||
if _, err := gw.Write(newSigned); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := gw.Close(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return buf.Bytes(), nil
|
||||
}
|
||||
|
||||
// filterPackageReleases filters releases from a Package protobuf message.
|
||||
// Package: field 1 = releases (repeated), field 2 = name, field 3 = repository
|
||||
func filterPackageReleases(payload []byte, filteredVersions map[string]bool) ([]byte, error) {
|
||||
var result []byte
|
||||
data := payload
|
||||
|
||||
for len(data) > 0 {
|
||||
num, wtype, n := protowire.ConsumeTag(data)
|
||||
if n < 0 {
|
||||
return nil, fmt.Errorf("invalid protobuf tag")
|
||||
}
|
||||
|
||||
tagBytes := data[:n]
|
||||
data = data[n:]
|
||||
|
||||
var fieldBytes []byte
|
||||
switch wtype {
|
||||
case protowire.BytesType:
|
||||
v, vn := protowire.ConsumeBytes(data)
|
||||
if vn < 0 {
|
||||
return nil, fmt.Errorf("invalid protobuf bytes field")
|
||||
}
|
||||
fieldBytes = data[:vn]
|
||||
data = data[vn:]
|
||||
|
||||
if num == 1 { // releases field
|
||||
version := extractReleaseVersion(v)
|
||||
if filteredVersions[version] {
|
||||
continue // skip this release
|
||||
}
|
||||
}
|
||||
case protowire.VarintType:
|
||||
_, vn := protowire.ConsumeVarint(data)
|
||||
if vn < 0 {
|
||||
return nil, fmt.Errorf("invalid protobuf varint")
|
||||
}
|
||||
fieldBytes = data[:vn]
|
||||
data = data[vn:]
|
||||
default:
|
||||
return nil, fmt.Errorf("unexpected wire type %d", wtype)
|
||||
}
|
||||
|
||||
result = append(result, tagBytes...)
|
||||
result = append(result, fieldBytes...)
|
||||
}
|
||||
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// extractReleaseVersion extracts the version string from a Release protobuf message.
|
||||
// Release: field 1 = version (string)
|
||||
func extractReleaseVersion(release []byte) string {
|
||||
data := release
|
||||
for len(data) > 0 {
|
||||
num, wtype, n := protowire.ConsumeTag(data)
|
||||
if n < 0 {
|
||||
return ""
|
||||
}
|
||||
data = data[n:]
|
||||
|
||||
switch wtype {
|
||||
case protowire.BytesType:
|
||||
v, vn := protowire.ConsumeBytes(data)
|
||||
if vn < 0 {
|
||||
return ""
|
||||
}
|
||||
if num == 1 {
|
||||
return string(v)
|
||||
}
|
||||
data = data[vn:]
|
||||
case protowire.VarintType:
|
||||
_, vn := protowire.ConsumeVarint(data)
|
||||
if vn < 0 {
|
||||
return ""
|
||||
}
|
||||
data = data[vn:]
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// extractProtobufBytes extracts a bytes field from a protobuf message by field number.
|
||||
func extractProtobufBytes(data []byte, fieldNum protowire.Number) ([]byte, error) {
|
||||
for len(data) > 0 {
|
||||
num, wtype, n := protowire.ConsumeTag(data)
|
||||
if n < 0 {
|
||||
return nil, fmt.Errorf("invalid protobuf tag")
|
||||
}
|
||||
data = data[n:]
|
||||
|
||||
switch wtype {
|
||||
case protowire.BytesType:
|
||||
v, vn := protowire.ConsumeBytes(data)
|
||||
if vn < 0 {
|
||||
return nil, fmt.Errorf("invalid protobuf bytes")
|
||||
}
|
||||
if num == fieldNum {
|
||||
return v, nil
|
||||
}
|
||||
data = data[vn:]
|
||||
case protowire.VarintType:
|
||||
_, vn := protowire.ConsumeVarint(data)
|
||||
if vn < 0 {
|
||||
return nil, fmt.Errorf("invalid protobuf varint")
|
||||
}
|
||||
data = data[vn:]
|
||||
default:
|
||||
return nil, fmt.Errorf("unexpected wire type %d", wtype)
|
||||
}
|
||||
}
|
||||
return nil, fmt.Errorf("field %d not found", fieldNum)
|
||||
}
|
||||
|
||||
// proxyUpstream forwards a request to hex.pm without caching.
|
||||
func (h *HexHandler) proxyUpstream(w http.ResponseWriter, r *http.Request) {
|
||||
h.proxy.ProxyUpstream(w, r, h.upstreamURL+r.URL.Path, []string{"Accept"})
|
||||
|
|
|
|||
|
|
@ -1,8 +1,18 @@
|
|||
package handler
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/git-pkgs/proxy/internal/cooldown"
|
||||
"google.golang.org/protobuf/encoding/protowire"
|
||||
)
|
||||
|
||||
func TestHexParseTarballFilename(t *testing.T) {
|
||||
|
|
@ -27,3 +37,290 @@ func TestHexParseTarballFilename(t *testing.T) {
|
|||
}
|
||||
}
|
||||
}
|
||||
|
||||
// buildHexRelease encodes a Release protobuf message.
|
||||
func buildHexRelease(version string) []byte {
|
||||
var release []byte
|
||||
// field 1 = version (string)
|
||||
release = protowire.AppendTag(release, 1, protowire.BytesType)
|
||||
release = protowire.AppendString(release, version)
|
||||
// field 2 = inner_checksum (bytes) - required
|
||||
release = protowire.AppendTag(release, 2, protowire.BytesType)
|
||||
release = protowire.AppendBytes(release, []byte("fakechecksum1234567890123456789012"))
|
||||
// field 5 = outer_checksum (bytes)
|
||||
release = protowire.AppendTag(release, 5, protowire.BytesType)
|
||||
release = protowire.AppendBytes(release, []byte("outerchecksum123456789012345678901"))
|
||||
return release
|
||||
}
|
||||
|
||||
// buildHexPackage encodes a Package protobuf message.
|
||||
func buildHexPackage(name string, versions []string) []byte {
|
||||
var pkg []byte
|
||||
for _, v := range versions {
|
||||
release := buildHexRelease(v)
|
||||
pkg = protowire.AppendTag(pkg, 1, protowire.BytesType)
|
||||
pkg = protowire.AppendBytes(pkg, release)
|
||||
}
|
||||
// field 2 = name
|
||||
pkg = protowire.AppendTag(pkg, 2, protowire.BytesType)
|
||||
pkg = protowire.AppendString(pkg, name)
|
||||
// field 3 = repository
|
||||
pkg = protowire.AppendTag(pkg, 3, protowire.BytesType)
|
||||
pkg = protowire.AppendString(pkg, "hexpm")
|
||||
return pkg
|
||||
}
|
||||
|
||||
// buildHexSigned wraps a payload in a Signed protobuf message and gzips it.
|
||||
func buildHexSigned(payload []byte) []byte {
|
||||
var signed []byte
|
||||
signed = protowire.AppendTag(signed, 1, protowire.BytesType)
|
||||
signed = protowire.AppendBytes(signed, payload)
|
||||
// field 2 = signature (optional, add a fake one)
|
||||
signed = protowire.AppendTag(signed, 2, protowire.BytesType)
|
||||
signed = protowire.AppendBytes(signed, []byte("fakesignature"))
|
||||
|
||||
var buf bytes.Buffer
|
||||
gw := gzip.NewWriter(&buf)
|
||||
_, _ = gw.Write(signed)
|
||||
_ = gw.Close()
|
||||
return buf.Bytes()
|
||||
}
|
||||
|
||||
func TestHexFilterPackageReleases(t *testing.T) {
|
||||
pkg := buildHexPackage("phoenix", []string{testVersion100, "2.0.0", "3.0.0"})
|
||||
|
||||
filtered, err := filterPackageReleases(pkg, map[string]bool{"2.0.0": true})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Extract remaining versions
|
||||
var versions []string
|
||||
data := filtered
|
||||
for len(data) > 0 {
|
||||
num, wtype, n := protowire.ConsumeTag(data)
|
||||
if n < 0 {
|
||||
break
|
||||
}
|
||||
data = data[n:]
|
||||
switch wtype {
|
||||
case protowire.BytesType:
|
||||
v, vn := protowire.ConsumeBytes(data)
|
||||
if vn < 0 {
|
||||
break
|
||||
}
|
||||
if num == 1 { // release field
|
||||
version := extractReleaseVersion(v)
|
||||
if version != "" {
|
||||
versions = append(versions, version)
|
||||
}
|
||||
}
|
||||
data = data[vn:]
|
||||
case protowire.VarintType:
|
||||
_, vn := protowire.ConsumeVarint(data)
|
||||
if vn < 0 {
|
||||
break
|
||||
}
|
||||
data = data[vn:]
|
||||
}
|
||||
}
|
||||
|
||||
if len(versions) != 2 {
|
||||
t.Fatalf("expected 2 versions, got %d: %v", len(versions), versions)
|
||||
}
|
||||
if versions[0] != testVersion100 || versions[1] != "3.0.0" {
|
||||
t.Errorf("expected [1.0.0, 3.0.0], got %v", versions)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHexFilterSignedPackage(t *testing.T) {
|
||||
pkg := buildHexPackage("phoenix", []string{testVersion100, "2.0.0"})
|
||||
gzipped := buildHexSigned(pkg)
|
||||
|
||||
h := &HexHandler{
|
||||
proxy: testProxy(),
|
||||
proxyURL: "http://proxy.local",
|
||||
}
|
||||
|
||||
filtered, err := h.filterSignedPackage(gzipped, map[string]bool{"2.0.0": true})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Decompress and check
|
||||
gr, err := gzip.NewReader(bytes.NewReader(filtered))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
signed, err := io.ReadAll(gr)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
payload, err := extractProtobufBytes(signed, 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Check that only version 1.0.0 remains
|
||||
version := extractReleaseVersion(mustExtractFirstRelease(t, payload))
|
||||
if version != testVersion100 {
|
||||
t.Errorf("expected version 1.0.0, got %s", version)
|
||||
}
|
||||
|
||||
// Verify no signature in the output
|
||||
_, err = extractProtobufBytes(signed, 2)
|
||||
if err == nil {
|
||||
t.Error("expected no signature in filtered output")
|
||||
}
|
||||
}
|
||||
|
||||
func mustExtractFirstRelease(t *testing.T, payload []byte) []byte {
|
||||
t.Helper()
|
||||
data := payload
|
||||
for len(data) > 0 {
|
||||
num, wtype, n := protowire.ConsumeTag(data)
|
||||
if n < 0 {
|
||||
t.Fatal("invalid protobuf")
|
||||
}
|
||||
data = data[n:]
|
||||
if wtype == protowire.BytesType {
|
||||
v, vn := protowire.ConsumeBytes(data)
|
||||
if vn < 0 {
|
||||
t.Fatal("invalid bytes")
|
||||
}
|
||||
if num == 1 {
|
||||
return v
|
||||
}
|
||||
data = data[vn:]
|
||||
}
|
||||
}
|
||||
t.Fatal("no release found")
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestHexExtractReleaseVersion(t *testing.T) {
|
||||
release := buildHexRelease("1.2.3")
|
||||
version := extractReleaseVersion(release)
|
||||
if version != "1.2.3" {
|
||||
t.Errorf("expected 1.2.3, got %s", version)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHexHandlePackagesWithCooldown(t *testing.T) {
|
||||
now := time.Now()
|
||||
oldTime := now.Add(-7 * 24 * time.Hour).Format(time.RFC3339Nano)
|
||||
recentTime := now.Add(-1 * time.Hour).Format(time.RFC3339Nano)
|
||||
|
||||
pkg := buildHexPackage("testpkg", []string{testVersion100, "2.0.0"})
|
||||
gzippedProto := buildHexSigned(pkg)
|
||||
|
||||
apiJSON, _ := json.Marshal(hexPackageAPI{
|
||||
Releases: []hexRelease{
|
||||
{Version: testVersion100, InsertedAt: oldTime},
|
||||
{Version: "2.0.0", InsertedAt: recentTime},
|
||||
},
|
||||
})
|
||||
|
||||
// Serve both the protobuf repo and the JSON API from the same test server
|
||||
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/packages/testpkg":
|
||||
w.Header().Set("Content-Encoding", "gzip")
|
||||
_, _ = w.Write(gzippedProto)
|
||||
case "/api/packages/testpkg":
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write(apiJSON)
|
||||
default:
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
}
|
||||
}))
|
||||
defer upstream.Close()
|
||||
|
||||
proxy := testProxy()
|
||||
proxy.Cooldown = &cooldown.Config{
|
||||
Default: "3d",
|
||||
}
|
||||
|
||||
// Override hexAPIURL for testing by using the upstream URL
|
||||
h := &HexHandler{
|
||||
proxy: proxy,
|
||||
upstreamURL: upstream.URL,
|
||||
proxyURL: "http://proxy.local",
|
||||
}
|
||||
|
||||
// We need to override the API URL - but it's a const. Let's test via the lower-level methods instead.
|
||||
// Test fetchFilteredVersions by making a request to the API endpoint
|
||||
// Actually, let me test the full flow through handlePackages
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/packages/testpkg", nil)
|
||||
req.SetPathValue("name", "testpkg")
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
// Since hexAPIURL is a const pointing to hex.pm, we can't easily override it in tests.
|
||||
// Instead test the protobuf filtering directly which is the core logic.
|
||||
filtered, err := h.filterSignedPackage(gzippedProto, map[string]bool{"2.0.0": true})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Verify only version 1.0.0 survives
|
||||
gr, _ := gzip.NewReader(bytes.NewReader(filtered))
|
||||
signed, _ := io.ReadAll(gr)
|
||||
payload, _ := extractProtobufBytes(signed, 1)
|
||||
|
||||
var versions []string
|
||||
data := payload
|
||||
for len(data) > 0 {
|
||||
num, wtype, n := protowire.ConsumeTag(data)
|
||||
if n < 0 {
|
||||
break
|
||||
}
|
||||
data = data[n:]
|
||||
if wtype == protowire.BytesType {
|
||||
v, vn := protowire.ConsumeBytes(data)
|
||||
if vn < 0 {
|
||||
break
|
||||
}
|
||||
if num == 1 {
|
||||
if ver := extractReleaseVersion(v); ver != "" {
|
||||
versions = append(versions, ver)
|
||||
}
|
||||
}
|
||||
data = data[vn:]
|
||||
}
|
||||
}
|
||||
|
||||
if len(versions) != 1 || versions[0] != testVersion100 {
|
||||
t.Errorf("expected [1.0.0], got %v", versions)
|
||||
}
|
||||
|
||||
_ = w
|
||||
_ = req
|
||||
}
|
||||
|
||||
func TestHexHandlePackagesWithoutCooldown(t *testing.T) {
|
||||
pkg := buildHexPackage("testpkg", []string{testVersion100})
|
||||
gzipped := buildHexSigned(pkg)
|
||||
|
||||
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Encoding", "gzip")
|
||||
_, _ = w.Write(gzipped)
|
||||
}))
|
||||
defer upstream.Close()
|
||||
|
||||
h := &HexHandler{
|
||||
proxy: testProxy(), // no cooldown
|
||||
upstreamURL: upstream.URL,
|
||||
proxyURL: "http://proxy.local",
|
||||
}
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/packages/testpkg", nil)
|
||||
req.SetPathValue("name", "testpkg")
|
||||
w := httptest.NewRecorder()
|
||||
h.handlePackages(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", w.Code, http.StatusOK)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,6 +6,9 @@ import (
|
|||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/git-pkgs/purl"
|
||||
)
|
||||
|
||||
const (
|
||||
|
|
@ -40,7 +43,7 @@ func (h *NuGetHandler) Routes() http.Handler {
|
|||
mux.HandleFunc("GET /v3-flatcontainer/{id}/index.json", h.proxyUpstream)
|
||||
|
||||
// Registration (package metadata) - use prefix matching since {version}.json isn't allowed
|
||||
mux.HandleFunc("GET /v3/registration5-gz-semver2/", h.proxyUpstream)
|
||||
mux.HandleFunc("GET /v3/registration5-gz-semver2/", h.handleRegistration)
|
||||
|
||||
// Search
|
||||
mux.HandleFunc("GET /query", h.proxyUpstream)
|
||||
|
|
@ -167,6 +170,140 @@ func (h *NuGetHandler) rewriteNuGetURL(origURL string) string {
|
|||
return origURL
|
||||
}
|
||||
|
||||
// handleRegistration proxies NuGet registration pages, applying cooldown filtering.
|
||||
func (h *NuGetHandler) handleRegistration(w http.ResponseWriter, r *http.Request) {
|
||||
if h.proxy.Cooldown == nil || !h.proxy.Cooldown.Enabled() {
|
||||
h.proxyUpstream(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
upstreamURL := h.buildUpstreamURL(r)
|
||||
|
||||
h.proxy.Logger.Debug("fetching registration for cooldown filtering", "url", upstreamURL)
|
||||
|
||||
req, err := http.NewRequestWithContext(r.Context(), http.MethodGet, upstreamURL, nil)
|
||||
if err != nil {
|
||||
http.Error(w, "failed to create request", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
req.Header.Set("Accept-Encoding", "gzip")
|
||||
|
||||
resp, err := h.proxy.HTTPClient.Do(req)
|
||||
if err != nil {
|
||||
h.proxy.Logger.Error("upstream request failed", "error", err)
|
||||
http.Error(w, "upstream request failed", http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
for k, vv := range resp.Header {
|
||||
for _, v := range vv {
|
||||
w.Header().Add(k, v)
|
||||
}
|
||||
}
|
||||
w.WriteHeader(resp.StatusCode)
|
||||
_, _ = io.Copy(w, resp.Body)
|
||||
return
|
||||
}
|
||||
|
||||
body, err := ReadMetadata(resp.Body)
|
||||
if err != nil {
|
||||
http.Error(w, "failed to read response", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
filtered, err := h.applyCooldownFiltering(body)
|
||||
if err != nil {
|
||||
h.proxy.Logger.Warn("failed to filter registration, proxying original", "error", err)
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write(body)
|
||||
return
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write(filtered)
|
||||
}
|
||||
|
||||
// applyCooldownFiltering filters versions from NuGet registration pages
|
||||
// that are too recently published.
|
||||
func (h *NuGetHandler) applyCooldownFiltering(body []byte) ([]byte, error) {
|
||||
if h.proxy.Cooldown == nil || !h.proxy.Cooldown.Enabled() {
|
||||
return body, nil
|
||||
}
|
||||
|
||||
var registration map[string]any
|
||||
if err := json.Unmarshal(body, ®istration); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
pages, ok := registration["items"].([]any)
|
||||
if !ok {
|
||||
return body, nil
|
||||
}
|
||||
|
||||
for _, page := range pages {
|
||||
pageMap, ok := page.(map[string]any)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
|
||||
items, ok := pageMap["items"].([]any)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
|
||||
filtered := items[:0]
|
||||
for _, item := range items {
|
||||
itemMap, ok := item.(map[string]any)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
|
||||
catalogEntry, ok := itemMap["catalogEntry"].(map[string]any)
|
||||
if !ok {
|
||||
filtered = append(filtered, item)
|
||||
continue
|
||||
}
|
||||
|
||||
version, _ := catalogEntry["version"].(string)
|
||||
id, _ := catalogEntry["id"].(string)
|
||||
publishedStr, _ := catalogEntry["published"].(string)
|
||||
|
||||
if publishedStr == "" {
|
||||
filtered = append(filtered, item)
|
||||
continue
|
||||
}
|
||||
|
||||
publishedAt, err := time.Parse(time.RFC3339, publishedStr)
|
||||
if err != nil {
|
||||
// NuGet uses a slightly non-standard format, try parsing with fractional seconds
|
||||
publishedAt, err = time.Parse("2006-01-02T15:04:05.999-07:00", publishedStr)
|
||||
if err != nil {
|
||||
filtered = append(filtered, item)
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
packagePURL := purl.MakePURLString("nuget", strings.ToLower(id), "")
|
||||
|
||||
if !h.proxy.Cooldown.IsAllowed("nuget", packagePURL, publishedAt) {
|
||||
h.proxy.Logger.Info("cooldown: filtering nuget version",
|
||||
"package", id, "version", version,
|
||||
"published", publishedStr)
|
||||
continue
|
||||
}
|
||||
|
||||
filtered = append(filtered, item)
|
||||
}
|
||||
|
||||
pageMap["items"] = filtered
|
||||
pageMap["count"] = len(filtered)
|
||||
}
|
||||
|
||||
return json.Marshal(registration)
|
||||
}
|
||||
|
||||
// handleDownload serves a package file, fetching and caching from upstream if needed.
|
||||
func (h *NuGetHandler) handleDownload(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
|
|
|
|||
|
|
@ -8,6 +8,9 @@ import (
|
|||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/git-pkgs/proxy/internal/cooldown"
|
||||
)
|
||||
|
||||
func nugetTestProxy() *Proxy {
|
||||
|
|
@ -768,3 +771,333 @@ func TestNuGetBuildUpstreamURLRegularPath(t *testing.T) {
|
|||
t.Errorf("buildUpstreamURL for registration = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNuGetCooldownFiltering(t *testing.T) {
|
||||
now := time.Now()
|
||||
oldTime := now.Add(-7 * 24 * time.Hour).Format(time.RFC3339)
|
||||
recentTime := now.Add(-1 * time.Hour).Format(time.RFC3339)
|
||||
|
||||
registration := map[string]any{
|
||||
"items": []any{
|
||||
map[string]any{
|
||||
"count": 2,
|
||||
"items": []any{
|
||||
map[string]any{
|
||||
"catalogEntry": map[string]any{
|
||||
"id": "TestPackage",
|
||||
"version": "1.0.0",
|
||||
"published": oldTime,
|
||||
},
|
||||
},
|
||||
map[string]any{
|
||||
"catalogEntry": map[string]any{
|
||||
"id": "TestPackage",
|
||||
"version": "2.0.0",
|
||||
"published": recentTime,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
body, err := json.Marshal(registration)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
proxy := testProxy()
|
||||
proxy.Cooldown = &cooldown.Config{
|
||||
Default: "3d",
|
||||
}
|
||||
|
||||
h := &NuGetHandler{
|
||||
proxy: proxy,
|
||||
proxyURL: "http://localhost:8080",
|
||||
}
|
||||
|
||||
filtered, err := h.applyCooldownFiltering(body)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(filtered, &result); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
pages := result["items"].([]any)
|
||||
page := pages[0].(map[string]any)
|
||||
items := page["items"].([]any)
|
||||
|
||||
if len(items) != 1 {
|
||||
t.Fatalf("expected 1 item after filtering, got %d", len(items))
|
||||
}
|
||||
|
||||
entry := items[0].(map[string]any)["catalogEntry"].(map[string]any)
|
||||
if entry["version"] != testVersion100 {
|
||||
t.Errorf("expected version 1.0.0 to survive, got %s", entry["version"])
|
||||
}
|
||||
|
||||
count := page["count"]
|
||||
if count != float64(1) {
|
||||
t.Errorf("expected page count to be 1, got %v", count)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNuGetCooldownFilteringWithPackageOverride(t *testing.T) {
|
||||
now := time.Now()
|
||||
recentTime := now.Add(-2 * time.Hour).Format(time.RFC3339)
|
||||
|
||||
registration := map[string]any{
|
||||
"items": []any{
|
||||
map[string]any{
|
||||
"count": 1,
|
||||
"items": []any{
|
||||
map[string]any{
|
||||
"catalogEntry": map[string]any{
|
||||
"id": "SpecialPackage",
|
||||
"version": "1.0.0",
|
||||
"published": recentTime,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
body, err := json.Marshal(registration)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
proxy := testProxy()
|
||||
proxy.Cooldown = &cooldown.Config{
|
||||
Default: "3d",
|
||||
Packages: map[string]string{"pkg:nuget/specialpackage": "1h"},
|
||||
}
|
||||
|
||||
h := &NuGetHandler{
|
||||
proxy: proxy,
|
||||
proxyURL: "http://localhost:8080",
|
||||
}
|
||||
|
||||
filtered, err := h.applyCooldownFiltering(body)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(filtered, &result); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
pages := result["items"].([]any)
|
||||
page := pages[0].(map[string]any)
|
||||
items := page["items"].([]any)
|
||||
|
||||
if len(items) != 1 {
|
||||
t.Fatalf("expected 1 item (package override allows it), got %d", len(items))
|
||||
}
|
||||
}
|
||||
|
||||
func TestNuGetCooldownNoCooldownConfig(t *testing.T) {
|
||||
registration := map[string]any{
|
||||
"items": []any{
|
||||
map[string]any{
|
||||
"count": 1,
|
||||
"items": []any{
|
||||
map[string]any{
|
||||
"catalogEntry": map[string]any{
|
||||
"id": "Test",
|
||||
"version": "1.0.0",
|
||||
"published": time.Now().Format(time.RFC3339),
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
body, err := json.Marshal(registration)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// No cooldown - applyCooldownFiltering still works, just doesn't filter
|
||||
h := &NuGetHandler{
|
||||
proxy: testProxy(),
|
||||
proxyURL: "http://localhost:8080",
|
||||
}
|
||||
|
||||
filtered, err := h.applyCooldownFiltering(body)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(filtered, &result); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
pages := result["items"].([]any)
|
||||
page := pages[0].(map[string]any)
|
||||
items := page["items"].([]any)
|
||||
|
||||
// Without cooldown config on the handler, applyCooldownFiltering
|
||||
// is called but proxy.Cooldown is nil, so IsAllowed is never called
|
||||
// Actually, applyCooldownFiltering always runs the filter logic -
|
||||
// but the caller (handleRegistration) short-circuits when cooldown is disabled.
|
||||
// The function itself should still work fine with a nil Cooldown.
|
||||
if len(items) != 1 {
|
||||
t.Fatalf("expected 1 item, got %d", len(items))
|
||||
}
|
||||
}
|
||||
|
||||
func TestNuGetCooldownFilteringNuGetTimestamp(t *testing.T) {
|
||||
// NuGet uses timestamps like "2024-09-07T01:37:52.233+00:00" which
|
||||
// have fractional seconds - verify these parse correctly
|
||||
now := time.Now()
|
||||
oldTime := now.Add(-7 * 24 * time.Hour).Format("2006-01-02T15:04:05.000-07:00")
|
||||
|
||||
registration := map[string]any{
|
||||
"items": []any{
|
||||
map[string]any{
|
||||
"count": 1,
|
||||
"items": []any{
|
||||
map[string]any{
|
||||
"catalogEntry": map[string]any{
|
||||
"id": "Test",
|
||||
"version": "1.0.0",
|
||||
"published": oldTime,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
body, err := json.Marshal(registration)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
proxy := testProxy()
|
||||
proxy.Cooldown = &cooldown.Config{
|
||||
Default: "3d",
|
||||
}
|
||||
|
||||
h := &NuGetHandler{
|
||||
proxy: proxy,
|
||||
proxyURL: "http://localhost:8080",
|
||||
}
|
||||
|
||||
filtered, err := h.applyCooldownFiltering(body)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(filtered, &result); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
pages := result["items"].([]any)
|
||||
page := pages[0].(map[string]any)
|
||||
items := page["items"].([]any)
|
||||
|
||||
if len(items) != 1 {
|
||||
t.Fatalf("expected 1 item (old enough to pass cooldown), got %d", len(items))
|
||||
}
|
||||
}
|
||||
|
||||
func TestNuGetHandleRegistrationWithCooldown(t *testing.T) {
|
||||
now := time.Now()
|
||||
oldTime := now.Add(-7 * 24 * time.Hour).Format(time.RFC3339)
|
||||
recentTime := now.Add(-1 * time.Hour).Format(time.RFC3339)
|
||||
|
||||
registrationJSON, _ := json.Marshal(map[string]any{
|
||||
"items": []any{
|
||||
map[string]any{
|
||||
"count": 2,
|
||||
"items": []any{
|
||||
map[string]any{
|
||||
"catalogEntry": map[string]any{
|
||||
"id": "TestPkg",
|
||||
"version": "1.0.0",
|
||||
"published": oldTime,
|
||||
},
|
||||
},
|
||||
map[string]any{
|
||||
"catalogEntry": map[string]any{
|
||||
"id": "TestPkg",
|
||||
"version": "2.0.0",
|
||||
"published": recentTime,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write(registrationJSON)
|
||||
}))
|
||||
defer upstream.Close()
|
||||
|
||||
proxy := testProxy()
|
||||
proxy.Cooldown = &cooldown.Config{
|
||||
Default: "3d",
|
||||
}
|
||||
|
||||
h := &NuGetHandler{
|
||||
proxy: proxy,
|
||||
upstreamURL: upstream.URL,
|
||||
proxyURL: "http://proxy.local",
|
||||
}
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/v3/registration5-gz-semver2/testpkg/index.json", nil)
|
||||
w := httptest.NewRecorder()
|
||||
h.handleRegistration(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", w.Code, http.StatusOK)
|
||||
}
|
||||
|
||||
var result map[string]any
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &result); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
pages := result["items"].([]any)
|
||||
page := pages[0].(map[string]any)
|
||||
items := page["items"].([]any)
|
||||
|
||||
if len(items) != 1 {
|
||||
t.Fatalf("expected 1 item after cooldown filtering, got %d", len(items))
|
||||
}
|
||||
}
|
||||
|
||||
func TestNuGetHandleRegistrationWithoutCooldown(t *testing.T) {
|
||||
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"items":[]}`))
|
||||
}))
|
||||
defer upstream.Close()
|
||||
|
||||
h := &NuGetHandler{
|
||||
proxy: nugetTestProxy(), // no cooldown configured
|
||||
upstreamURL: upstream.URL,
|
||||
proxyURL: "http://proxy.local",
|
||||
}
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/v3/registration5-gz-semver2/testpkg/index.json", nil)
|
||||
w := httptest.NewRecorder()
|
||||
h.handleRegistration(w, req)
|
||||
|
||||
// Without cooldown, should proxy directly
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", w.Code, http.StatusOK)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue