Replicas sharing a database need one place to agree on the Last-Modified
of rewritten metadata, which process memory cannot give them.
- Add metadata_dates, keyed by ecosystem, proxy URL and package, holding the
ETag being dated and its date in Unix seconds, so no zone is involved
- Insert a package's first date only if no replica recorded one first
- Swap a date only if it is still the one read, so of two replicas
switching a package at once only one succeeds
- Create it in both schemas and in migration 011
Every fetch of an artifact wrote one path, so fetches with different URLs
or digests, which do not share a coalescing key, could overwrite each
other's bytes, serve the wrong ones, or delete them on a digest mismatch
while the other was about to open them.
storeArtifact now writes {ecosystem}/{name}/{version}/{fetch id}/{filename},
with a random id per fetch. Most ecosystems learn the digest only after
the fetch, and storage has no rename, so the id is the one rule that fits
all of them. Existing records keep their paths and stay readable. On a
file:// bucket, Delete now removes an emptied fetch directory, which
fileblob leaves behind, and no other directory, since another fetch may
be creating its own inside it.
A path a record stops pointing at is queued in pending_deletes rather
than deleted, since a request that read the record may still open it. A
loop deletes queued paths after max(1h, direct_serve_ttl), whether or not
max_size is set. UpsertArtifact and clears apply only if the record still
holds the path the caller read, so racing commits each queue the path
they replaced and a clear never orphans a newer commit. Eviction still
deletes inline, and counts space as freed only when the record still
pointed at what it deleted.
createTestPostgresDB dropped artifacts, versions, packages and
schema_info before calling CreateSchema, but not the migrations table.
On a database that has seen one test the migration records survive, so
the next CreateSchema fails while recording 001_add_packages_enrichment_
columns with a duplicate key on migrations_pkey. Running the package
against one Postgres therefore failed from the second Postgres-backed
test on.
Drop vulnerabilities, metadata_cache and migrations as well, so the
fixture clears every table CreateSchema creates. The package now passes
repeatedly against the same database.
* Use shared artifacts at cache boundaries
* Address artifact cache review
* Defer cached artifact validation to checkCache
Construct artifacts.Artifact from the row without validation so a
malformed content hash reaches newIntegrityChecks in checkCache, which
clears the row and treats the request as a miss. Erroring at the DB
boundary instead surfaced a 500 and left the bad row in place.
* Build stored Artifact after digest and scan checks
Construct the shared artifact struct from trusted storage output as a
literal, after the hash-mismatch and scanner paths that delete failed
downloads, so no path between Store and updateCacheDB can leave bytes
in storage without a database row.
- Consult versions.published_at before fetching the packument, and persist
the parsed time after the packument fallback, so each version's metadata
is fetched and parsed at most once
- Add DB.SetVersionPublishedAt, an upsert that writes only the publish time
- Preserve a stored published_at in UpsertVersion when the incoming value
is NULL, so the artifact-cache upsert cannot erase it
- Add handler tests for stored-time downloads and single-fetch behavior,
and a database test for preserve-on-NULL in both dialects
* Track applied migrations to skip column checks on startup
Add a migrations table that records which migrations have been applied.
On boot, load the set of applied names in one query and only run new ones.
A fully migrated database now does 1 query instead of ~12 HasColumn/HasTable
checks.
Fresh databases created via CreateSchema record all migrations as already
applied. Old databases get the migrations table on first MigrateSchema call
and each migration is recorded after it runs.
Closes#54
* Add benchmark for MigrateSchema on fully migrated database
* Optimize MigrateSchema to single query for fully migrated databases
Skip HasTable/HasColumn checks when the migrations table already exists.
A fully migrated database now does one SELECT instead of ~12 individual
column and table checks.
* Add migration docs and link from architecture
* Add test for upgrade from fully migrated database without migrations table
The proxy can now use an existing git-pkgs database as a starting point.
Packages and versions tables match git-pkgs schema, using PURL-based
references instead of integer IDs. The proxy adds its own artifacts
table for caching functionality.
Replace raw database/sql with jmoiron/sqlx for cleaner query handling.
Support both SQLite (default) and PostgreSQL as configurable backends.
Configuration via:
- CLI flags: -database-driver, -database-path, -database-url
- Environment: PROXY_DATABASE_DRIVER, PROXY_DATABASE_PATH, PROXY_DATABASE_URL
- Config file: database.driver, database.path, database.url
Tests run against both databases when PROXY_DATABASE_URL is set.
