mirror of
https://github.com/git-pkgs/proxy.git
synced 2026-11-09 15:49:16 -05:00
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Simon Chrzanowski
|
d02ad9b2ba |
Coordinate instances sharing a Postgres database
Instances started together on an empty database created the schema at the same time and all but one failed on duplicate types. Schema creation and migrations now run under a Postgres advisory lock. Eviction, reclaim, retention and the Gradle build cache sweep take a per-job advisory lock for each pass, and an instance that finds the job running elsewhere skips the pass. Before, every instance evicted the whole overshoot over max_size at once and emptied the cache. Mirror schedules record their last start in a new job_runs table, so each schedule runs once per interval instead of once per instance. Refs #543 |
||
|
Simon Chrzanowski
|
baeccd4408 |
Mirror gems and composer packages under the handler names
proxy mirror filed pkg:gem and pkg:composer packages under rubygems and packagist, the names purl gives their registries, while the handlers store them under gem and composer. Since the resolver accepts both spellings, mirrored gems were downloaded and filed under rubygems, so the packages list filtered by gem didn't show them. The mirror now uses the handler names. The package list, count and search match every spelling of the requested ecosystem, so rows already written as rubygems or packagist, by the mirror or by git-pkgs, show up too without a migration. Fixes #415 |
||
|
Simon Chrzanowski
|
0832c6142b |
Refuse pushes of a version that holds a pushed file under another name
Composer names are case-insensitive, so acme/private and acme/Private share the version record and its integrity but get different filenames. A second push under the other case was accepted and broke the integrity check of the first archive. For ecosystems with one file per version, any pushed file of the version now refuses the push, both before the upload and atomically when recording it. |
||
|
Simon Chrzanowski
|
48916bcb09 |
Never let a cache fill replace a pushed artifact
- Record a fetched artifact with its package and version in one transaction that refuses to replace a pushed artifact, so a fetch that finishes after a push keeps both the pushed file and its version's integrity. Pushes and cache fills share this write, including the compare-and-swap on the storage path that queues every replaced object. - Don't open the stored file for ecosystems without a manifest. - Derive multi-file versions from the file being named in the path. - Validate DELETE paths like PUT paths, answering 400 for malformed ones. - Parse ?sha256= with integrity.ParseHex. |
||
|
Simon Chrzanowski
|
5f431a3d89 |
Test concurrent pushes of one file at the database level
The handler-level test runs on SQLite only. This one also runs against Postgres when PROXY_DATABASE_URL is set. |
||
|
Simon Chrzanowski
|
5c0120f305 |
Record pushes atomically and address review findings
- Record the package, version and artifact of a push in one transaction that only inserts the artifact if no pushed copy exists. Of two concurrent pushes of the same file exactly one wins, and the loser's version integrity is rolled back with it. - Check for an existing file before reading the request body, and send Expect: 100-continue from proxy hosted push, so a refused re-push isn't uploaded first. - Leave an existing version record alone for multi-file ecosystems, so a generic push doesn't reset a version cached from upstream.generic. - Reserve the upstream.generic name "hosted" only while hosting is enabled, and only intercept /generic/hosted/ then. - Accept a file argument in proxy hosted delete for generic only. - Mark multi-file ecosystems explicitly, and give parse a min/max range. |
||
|
Simon Chrzanowski
|
d13b8be0ae |
Host generic files and share one push path across hosted ecosystems
Hosted ecosystems are now entries in one table: each defines its package
names, how the stored filename is chosen, how the manifest is read and which
namespaces are served from pushes alone. npm and Composer move there
unchanged, and generic is added as a third entry.
Generic files are pushed to /api/hosted/generic/{name}/{version}/{file}
and served without a token from /generic/hosted/{name}/{version}/{file},
with the sha256 as ETag. They have no manifest, a version can hold several
files, and they never come from an upstream. "hosted" is reserved as an
upstream.generic name. DELETE with a file removes only that file.
For all hosted ecosystems:
- A second push of the same file is refused with 409. Clients pin pushed
files by version and often by hash, so a published file is no longer
silently replaced. Delete it first to push it again.
- PUT takes an optional ?sha256= and refuses bytes that don't match.
proxy hosted push always sends it.
Versions of generic files carry no version-level integrity, since it could
only describe one of their files; each file is checked against its own
sha256.
Refs #133, #134, #535
|
||
Abhinav Gautam
|
a97b690858 |
Separate artifact and metadata cache metrics on /metrics and the analytics page (#463)
* Separate artifact and metadata cache metrics on /metrics and the analytics page * Serve metadata cache stats from a shared snapshot and split Grafana cache panels by kind |
||
Christian Heim
|
d56d4367c9 |
Count cached metadata against max_size and evict it LRU (#507)
* max_size now counts cached metadata as well as artifacts, and eviction drops whichever was used least recently, so metadata no longer piles up for good (#461). Metadata rows record when they were last served, an artifact not yet served counts from its download, and Arch stores a database blob again if eviction removed it under a refreshed row. * Shorter comments on the metadata last-access migration. * Show evicted metadata on the analytics page next to evicted artifacts, and cover the new counter in the metric surface test. * Evicting metadata now drops the row and queues its path in one transaction, so a crash before the blob is deleted leaves it to reclaim instead of orphaning it. * The last-access backfill now runs in the background in batches of 1000 instead of inside the migration, resuming after a restart, and until a row is backfilled eviction orders it by its write time. --------- Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com> |
||
|
simon chrzanowski
|
85b674f278 |
Keep the stored metadata on an upstream 304 (#495)
When upstream answered a revalidation with 304, the proxy read the cached document back from storage and then stored it again: a full upload and a SHA-256 over identical bytes, for every popular package once per TTL. A 304 now only moves the cache row's fetched_at on, keeping its digest. The update is conditional on the row still recording the digest of the document that was read, so a concurrent fetch that replaced it is not overwritten; in that case, and for rows written before digests were recorded, the document is stored as before. |
||
Christian Heim
|
c0320c7c87 | Clearing a record and queueing its file for deletion now happen in one transaction, so a failed queue write keeps the record instead of leaving a file nothing points at. Eviction drops the queue entry again once it has deleted the file itself. | ||
|
simon chrzanowski
|
3dae0fbae0 |
Stop UpsertPackage from clearing packages.source
The artifact source change added source = excluded.source to the packages upsert as well. source isn't in that INSERT's column list, so every package upsert set the column to NULL. The packages upsert is back to what it was, with a test that an existing source survives it. Refs #133 |
||
|
simon chrzanowski
|
e7449d1540 |
Read the package manifest of pushed archives
Resolvers take a version's dependencies from the registry metadata, not from the archive, so listing a pushed version needs its manifest. A push now reads package.json from the top directory of an npm tarball, or composer.json from the root or the single top directory of a Composer zip, and keeps it in the new artifacts.manifest column (migration 013_add_artifact_manifest). The manifest has to name the pushed package and version. A Composer manifest may leave the version out, and its name is compared case-insensitively. An archive that can't be read or whose manifest names something else is refused with 400 and removed from storage. The version's integrity is now the archive's sha512, which npm clients expect in dist.integrity. Refs #134 |
||
|
simon chrzanowski
|
5e66688873 |
Push, list and delete npm and Composer archives through /api/hosted
With hosted.enabled, PUT /api/hosted/{ecosystem}/{name}/{version} stores the request body as the archive for that version, marked local. It replaces any cached upstream copy and is served from the cache without contacting the upstream. GET /api/hosted/{ecosystem}/{name} lists what was pushed and DELETE removes a version again, after which clients get the upstream archive if there is one. Pushing and deleting need the bearer token; listing is open like the rest of the API.
Download handlers look archives up by the filename clients request, so a push is stored under the name its ecosystem derives from name and version: {name}-{version}.tgz for npm, without the scope, and {package}-{version}.zip for Composer. Other ecosystems are refused for now.
The Composer download handler resolves the upstream metadata before it looks at the cache and answers 404 for versions the upstream doesn't list. It now serves a pushed archive first. Only artifacts marked local take that path, so cached upstream archives still go through the metadata and commit checks.
The version's integrity is set to the pushed bytes, so the cache's integrity check verifies them rather than an upstream digest. Names and versions with path separators, dot segments or control characters are refused before anything is stored.
Refs #133
|
||
|
simon chrzanowski
|
87505c01c1 |
Mark pushed artifacts as local and keep them out of eviction
Artifacts get a source column: upstream for cached copies of a registry, local for archives pushed to the proxy. Existing rows become upstream through the new 012_add_artifact_source migration. Local artifacts aren't a cache, so the LRU size limit doesn't count or evict them and storage retention never selects them. They are only removed by deleting them. A local artifact has no upstream URL and stores it as empty. Refs #133 |
||
|
simon chrzanowski
|
9f42425646 |
Test upgrading cached metadata through the metadata_dates migration
- Start from the schema before migration 011 on SQLite and PostgreSQL, with npm and Composer metadata cached, and check the rows and stored bytes survive the migration unchanged - After the upgrade, check two replicas serve the cached document with matching ETag and Last-Modified and answer conditional requests with 304 - Change the upstream document and check neither replica answers the old validators with 304, and both agree on the new ones - Give the PostgreSQL handler test a database of its own, so it shares no tables with the database package tests on the same server |
||
Simon Chrzanowski
|
599b2e1473 |
Add a metadata_dates table for rewritten metadata dates
Replicas sharing a database need one place to agree on the Last-Modified of rewritten metadata, which process memory cannot give them. - Add metadata_dates, keyed by ecosystem, proxy URL and package, holding the ETag being dated and its date in Unix seconds, so no zone is involved - Insert a package's first date only if no replica recorded one first - Swap a date only if it is still the one read, so of two replicas switching a package at once only one succeeds - Create it in both schemas and in migration 011 |
||
|
Christian Heim
|
b664530f88 |
Add age-based retention for cached artifacts (#428)
* Add age-based retention for cached artifacts
Cached artifacts stayed forever unless storage.max_size pushed them out.
storage.retention adds a default, per-ecosystem and per-package duration
after which an artifact nobody downloaded gets evicted.
Ecosystems opt in one at a time: a handler registers with
retention.Default, and until it does, configuring that ecosystem (or one
of its packages) fails validation, and the default doesn't touch it. This
commit registers none, so nothing changes yet. Startup logs which
ecosystems the default covers and which it doesn't.
An artifact counts as expired when both its fetch time and its last
access are older than the cutoff. Checking only the last access would
evict a refetched artifact again right away, since clearing a record
keeps its old access time and a cache miss doesn't record a hit.
The sweep scans artifacts in id windows, so no single query holds the
SQLite connection for long, and checks each row against its own rule.
Expired records are cleared and their files go through pending_deletes
instead of being deleted directly, because a buffered hit can make an
artifact that's being served right now look expired. Buffered hits are
flushed before each sweep, and one sweep clears at most 10,000
artifacts.
Since a big first sweep can queue a lot, reclaim now keeps working
through due entries for up to 30 seconds per tick instead of stopping
after 100, and pending_deletes gets an index on queued_at.
Also:
- proxy_artifacts_evicted_total{reason, ecosystem} counts LRU and
retention evictions.
- gradle.build_cache.max_age accepts "7d", as its comment always said.
Refs #306
* Document storage retention
Describes the new storage.retention block, the order rules apply in,
that the default only covers ecosystems that support retention yet, and
what to expect from the first sweep over an old cache. Also lists the
new eviction counter and notes that gradle max_age takes days.
Refs #306
* Fix retention timing on Postgres and tighten the sweep
On Postgres, fetched_at and last_accessed_at are TIMESTAMP columns
without a zone. They hold the local time the proxy wrote, but lib/pq
reads them back labelled UTC. East of UTC that makes every artifact look
hours younger than it is, so the sweep evicted late: two hours in
Berlin, nine in Tokyo. The candidates' times are now read back as local
time, and the database tests run on both SQLite and Postgres, with one
that sets the local zone to Asia/Tokyo.
A few smaller things:
- When a delete fails, reclaim stops for the rest of the minute instead
of trying every queued path. Otherwise a storage outage turns into
thirty seconds of failed deletes and warnings every minute.
- If the buffered hits can't be written, the sweep skips that round.
Running on stale access times could clear something that is being
downloaded right now.
- DefaultCanonical only accepts a package key when rebuilding it gives
the same PURL. For alpine and deb it would otherwise produce keys
that never match.
- sweep_interval has to be at least a minute.
- New tests cover the eviction counter and an artifact that gets a hit
between the scan and the clear.
- The docs no longer say nothing changes without retention, and they
point out that the example's ecosystem and package rules only pass
validation once those ecosystems support retention.
Refs #306
* Reject qualifiers and subpaths in retention package keys
A rule names a whole package. Qualifiers like ?type=pom were dropped on
the way to the stored PURL and the rule quietly covered every file of the
package. Now such a key fails validation.
Refs #306
* Say when a package key doesn't match how packages are stored
A package rule for an ecosystem that supports retention, written in a
form its packages aren't stored under (pkg:npm/babel/core without the
@), failed with "retention is not supported for npm packages". Once the
ecosystem is enabled that is no longer true. Such a key now fails with
"does not match how npm packages are stored" and points to the package
rule forms in the docs. Keys for ecosystems without retention support
keep the old message.
Refs #306
* Note that re-running the mirror keeps packages cached
A mirror run over a package that is still cached goes through the normal
cache lookup and records a download, so the package's age starts over.
The docs only said that mirrored packages age from the time they were
mirrored.
Refs #306
* Say when an ecosystem takes no package rules
Some ecosystems will support retention without package rules, because nothing in their stored records names a package the way a PURL would (conan, helm). A package key for one of them used to fail with "does not match how conan packages are stored; see the package rule forms", which sends people looking for a form that doesn't exist. It now says package rules aren't supported there and points to the ecosystem rule.
Refs #306
* Use helm as the example of an ecosystem without package rules
Conan gets package rules once its handler stores archives under the recipe name, so the config test that shows the refusal now uses helm, whose cache records name the repository, not the chart.
Refs #306
* Say that each cached file ages on its own
The retention docs said each artifact, each version of a package, ages on its own. A version can have several cached files (a jar and its pom, several wheels, Conan sources and binaries), and each of them ages separately.
Refs #306
* Clear an expired artifact and queue its file in one transaction
The sweep cleared the record and queued the file for deletion as two separate writes. If queueing failed, or the process stopped in between, the file stayed in storage with no record pointing at it and no queue entry, and no later sweep would find it again. Both writes now share a transaction, so a failed queue rolls the clear back and the next sweep retries.
Refs #306
* Show artifact evictions on the analytics page
proxy_artifacts_evicted_total had no place on /ui/analytics. The Runtime card now lists evicted artifacts by reason and ecosystem, without the failure tint, and the coverage test records an eviction so it doesn't depend on test order.
Refs #306
* Store artifact times in UTC
fetched_at and last_accessed_at are now written in UTC, and the retention cutoffs are compared in UTC, the same convention #434 uses for the metadata cache. lib/pq sends a time with its offset and a Postgres TIMESTAMP keeps only the wall clock, so UTC on write is what reads back correctly. That replaces reinterpreting the times as local on read.
Rows written before this still hold local time. East of UTC they look younger by the offset until they are rewritten, so retention evicts them a few hours late once; west of UTC that much early.
Refs #306
---------
Co-authored-by: t <t@t>
|
||
|
Christian Heim
|
dcf90cfec3 |
Close the Composer download gaps left after #430 (#433)
* GitLab Composer archives are now cached per commit and fetched for the commit the lock file pins, and branch names with a slash are escaped in dist URLs so the download route matches them. * Composer branch archives that an older proxy cached from another commit's URL are now discarded and fetched again, and a refetched artifact records the URL it actually came from. * When a GitLab commit is fetched under its own name, the archive older proxies cached under the shared archive.zip name is dropped. A test now moves the branch from A to B and checks that A keeps coming from the cache. |
||
|
Christian Heim
|
16ebeaa975 |
On Postgres, metadata fetched_at is now stored in UTC, so metadata_ttl no longer expires hours late when the proxy runs east of UTC. Entries written before this expire late one last time and are stored in UTC on their next refresh. (#435) | ||
Ondrej Kokes
|
129e6c6c1b |
Filter cached packages by ecosystem using pills (#319)
* experimental filtering pills * align data in the dashboard table more nicely (#318) * Bump zizmorcore/zizmor-action from 0.6.2 to 0.6.3 (#333) Bumps [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) from 0.6.2 to 0.6.3. - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits]( |
||
Andrew Nesbitt
|
9dff45d36c |
Merge remote-tracking branch 'origin/main' into analytics-2-ecosystem-stats
# Conflicts: # README.md |
||
montehurd
|
55ed6b9a16 |
Keep hit timestamps from moving backwards
With proxies sharing a database, a batch can flush after a newer hit is already written, and overwriting last_accessed_at and updated_at with its older time could reorder LRU eviction. Both now keep the later of the stored and incoming times. The count still adds up. |
||
Wicliff Wolda
|
4ebd805ffc |
Per-ecosystem cache and download statistics
- Add GetEcosystemStats, aggregating packages, versions, artifacts, cache size, downloads and downloaded bytes per ecosystem - Publish six proxy_ecosystem_* gauges from it on the existing one-minute cache-stats tick - Set and selectively delete rather than Reset, so no scrape lands on a half-populated vector - Report artifacts with no package row under "unattributed" rather than dropping them, so the figures still add up Part 2 of 6 splitting #381 up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
montehurd
|
4827cb9d76 |
Batch cache hit writes
Every cached download wrote its hit in its own UPDATE. SQLite runs on one connection, so each download was a write transaction the rest queued behind, holding a warm npm cache near 1,000 requests a second with the CPU two thirds idle. Hits are now counted in memory and written in one transaction every database.hit_flush_interval, "1s" by default, "0" for a write per hit. A failed write keeps the hits for the next flush, and Close writes what is pending. The same test then served about 2,200 requests a second. Closes #322. |
||
|
montehurd
|
436155e50c |
Clear before deleting on eviction, and requeue failed reclaims
Eviction deleted an object before checking its record still pointed at it. If a refetch had moved the record, that object was queued and a request that read the record earlier could still be opening it. Eviction now clears first, skips the delete if the record moved, and queues the object when the delete fails after a clear. Reclaim queues a failed delete again, which moves it behind the rest, so objects the backend keeps refusing cannot fill every batch. Also drop the clearArtifactCache passthrough and ClearCachedArtifact's unused context, note that only tests call ArtifactPath, and document that max_size does not count objects waiting to be reclaimed. |
||
|
montehurd
|
4ce6c37bd9 |
Give each fetch its own storage path
Every fetch of an artifact wrote one path, so fetches with different URLs
or digests, which do not share a coalescing key, could overwrite each
other's bytes, serve the wrong ones, or delete them on a digest mismatch
while the other was about to open them.
storeArtifact now writes {ecosystem}/{name}/{version}/{fetch id}/{filename},
with a random id per fetch. Most ecosystems learn the digest only after
the fetch, and storage has no rename, so the id is the one rule that fits
all of them. Existing records keep their paths and stay readable. On a
file:// bucket, Delete now removes an emptied fetch directory, which
fileblob leaves behind, and no other directory, since another fetch may
be creating its own inside it.
A path a record stops pointing at is queued in pending_deletes rather
than deleted, since a request that read the record may still open it. A
loop deletes queued paths after max(1h, direct_serve_ttl), whether or not
max_size is set. UpsertArtifact and clears apply only if the record still
holds the path the caller read, so racing commits each queue the path
they replaced and a clear never orphans a newer commit. Eviction still
deletes inline, and counts space as freed only when the record still
pointed at what it deleted.
|
||
|
pinguinfuss
|
ba45227e03 |
test(database): drop every schema table in the Postgres fixture (#349)
createTestPostgresDB dropped artifacts, versions, packages and schema_info before calling CreateSchema, but not the migrations table. On a database that has seen one test the migration records survive, so the next CreateSchema fails while recording 001_add_packages_enrichment_ columns with a duplicate key on migrations_pkey. Running the package against one Postgres therefore failed from the second Postgres-backed test on. Drop vulnerabilities, metadata_cache and migrations as well, so the fixture clears every table CreateSchema creates. The package now passes repeatedly against the same database. |
||
|
pinguinfuss
|
3db2cd5c71 |
fix(database): set connection pool limits for Postgres (#350)
* fix(database): set connection pool limits for Postgres OpenPostgres returned sqlx.Open's handle with database/sql's defaults: no cap on open connections and two idle ones. Under load nearly every request opened a new Postgres session, ran its few statements and closed it again, paying a backend fork and SCRAM authentication each time. Set the pool limits the issue suggests: 32 open and 32 idle connections, idle connections closed after 5 minutes and every connection recycled after 30 minutes. The values live in named constants because the mnd linter rejects the literals inline. The test (skipped without PROXY_DATABASE_URL, like the other Postgres tests) takes 16 connections from the pool, releases them and checks that all 16 stay idle; with the default pool only two survive. Fixes #323 * test(database): pin the pool properties instead of the wiring Review pointed out that the pool test compared MaxOpenConnections to the constant it was set from, so the assertion followed the constant and would have accepted postgresMaxOpenConns = 0 (unlimited), and that its burst of 16 only proved MaxIdleConns >= 16. The burst now takes postgresMaxIdleConns connections, so every configured idle slot has to survive the release, and the open cap is checked to be finite and large enough for that burst before any connection is taken, so a cap below the idle count fails fast instead of blocking in db.Conn. The doc comment now says which settings the test covers; the idle-time and lifetime settings only show up in DBStats.MaxIdleTimeClosed and MaxLifetimeClosed after minutes of wall-clock time and stay unexercised. * test(database): guard the pool test against a vacuous burst Review showed that with the burst tied to postgresMaxIdleConns the test also passed for a constant of 2, database/sql's default, or of 0, where it took no connections at all. It now fails outright unless the configured idle count exceeds the default, and every connection it takes is released in a cleanup, so an assertion failure mid-burst no longer leaves sessions open for the rest of the test binary. |
||
|
Andrew Nesbitt
|
d950919608 |
Use shared artifacts at cache boundaries (#267)
* Use shared artifacts at cache boundaries * Address artifact cache review * Defer cached artifact validation to checkCache Construct artifacts.Artifact from the row without validation so a malformed content hash reaches newIntegrityChecks in checkCache, which clears the row and treats the request as a miss. Erroring at the DB boundary instead surfaced a 500 and left the bad row in place. * Build stored Artifact after digest and scan checks Construct the shared artifact struct from trusted storage output as a literal, after the hash-mismatch and scanner paths that delete failed downloads, so no path between Store and updateCacheDB can leave bytes in storage without a database row. |
||
|
pinguinfuss
|
c188a54ea4 |
fix(handler): preserve upstream content-encoding for cached metadata (#304)
* fix(handler): preserve upstream content-encoding for cached metadata Signed and hash-pinned index files (debian Release/Packages.gz, rpm repomd.xml, helm index.yaml, conda repodata.json, apk APKINDEX.tar.gz) were cached after Go's transport transparently decompressed any Content-Encoding: gzip response, so the proxy served bytes that differ from what the upstream signed and broke client verification. Request the identity encoding on the shared metadata fetch so Go no longer auto-decompresses, and persist the upstream Content-Encoding in a new metadata_cache column so both the cached and offline responses replay the exact bytes and header. The uncached streaming path now forwards Content-Encoding too. Fixes #300 * fix(handler): scope verbatim metadata fetch to the ProxyCached path The first cut forced Accept-Encoding: identity in the shared fetchUpstreamMetadata, which an adversarial review showed both missed the bug and regressed unrelated ecosystems: - proxyMetadataStream (the default path, since cache_metadata is off) never forced identity, so a client sending no Accept-Encoding still triggered Go's transparent gzip decompression and served altered bytes for apk/debian/rpm/conda indexes. - Direct FetchOrCacheMetadata callers that parse or rewrite the body (npm, pypi, cargo, composer, pub, nuget, swift, maven, helm) were forced to identity too, losing wire compression and 502-ing against upstreams that ignore identity and gzip anyway (helm rewriteIndex). - Rows cached before the fix kept serving decompressed bytes via ETag 304 revalidation. Scope the verbatim behavior to the ProxyCached code path, which serves upstream bytes through unchanged (apk, debian, rpm, go, hex, conda, cran, gem, conan, julia). That path now requests identity on both the cached fetch and the uncached stream branch and replays Content-Encoding; direct callers keep transparent compression, matching main. Migration 008 clears etag/fetched_at so legacy rows refetch once with identity. Tests now exercise the stream path with no client Accept-Encoding and pin that direct callers are not forced to identity. |
||
|
Abhinav Gautam
|
1e3369c959 |
fix(oci): cache tag lists and normalize manifest variants (#280)
* fix(oci): cache tag lists and normalize manifest variants * fix(oci): refine manifest cache variants * fix(oci): preserve manifest cache compatibility * fix(oci): dual-write manifest cache variants * fix(oci): preserve cached pagination links * fix(oci): rewrite named registry pagination links |
||
Victor Chacon Codesseira
|
76fcd07755 |
Read the stored publish time in the npm cooldown download check (#296)
- Consult versions.published_at before fetching the packument, and persist the parsed time after the packument fallback, so each version's metadata is fetched and parsed at most once - Add DB.SetVersionPublishedAt, an upsert that writes only the publish time - Preserve a stored published_at in UpsertVersion when the incoming value is NULL, so the artifact-cache upsert cannot erase it - Add handler tests for stored-time downloads and single-fetch behavior, and a database test for preserve-on-NULL in both dialects |
||
|
Andrew Nesbitt
|
1a814c7e1f |
Use shared integrity verification (#260)
* Use shared integrity verification * Finish integrity migration |
||
wickedOne
|
849500de1e |
fix: decode PURL percent-encoding in versions and package paths (#244)
* fix: decode PURL percent-encoding in versions and package paths * review fix |
||
|
Andrew Nesbitt
|
6fcc57c994 |
Optimize cached artifact serving (#245) | ||
|
Andrew Nesbitt
|
538a15d9f8 |
fix(container): serve cached images when upstream is unavailable (#199)
* container: cache manifests for offline pulls * Preserve direct-serve redirects for blob HEAD requests |
||
|
Andrew Nesbitt
|
992f5c68a7 |
Add .golangci.yml and clear gocognit/goconst findings (#113)
Bake the extended linter set into a project config so plain golangci-lint run matches what we check locally, with goconst tuned to ignore tests and bare lowercase words to drop ~200 ecosystem-name and test-literal false positives. Clear the remaining real findings: extract GradleBuildCacheConfig.Validate from Config.Validate, pull the eviction sort comparator into sortOldestFirst (zero time.Time already sorts first via Before so the switch was redundant), add headerAcceptEncoding and SQL column-type constants, and drop a dead empty-key recheck in the gradle handler. |
||
| c655399a07 | Apply 'go fmt' as suggested in CONTRIBUTING.md. | |||
|
Andrew Nesbitt
|
c01f0a5c05 |
Fix metadata caching, 404 propagation, mirror progress, and registry stubs
- ProxyCached now stores upstream Last-Modified in the cache and uses it (along with ETag) for conditional request handling, returning 304 when client validators match. Adds Content-Length to cached responses. - Handlers calling FetchOrCacheMetadata (pypi, composer, pub, nuget) now check for ErrUpstreamNotFound and return 404 instead of 502, matching the existing npm and cargo behavior. - Mirror jobs report live progress via a periodic callback while running, so API polls return real counts instead of zeroed progress. - Registry mirroring removed from CLI flags, API acceptance, README, and docs since every enumerator was a stub returning "not yet implemented". - Added tests for the conditional metadata path (ETag/If-None-Match, Last-Modified/If-Modified-Since, 304 responses, header omission). |
||
|
Andrew Nesbitt
|
d62c42b8d7 |
Add mirror command and API for selective package mirroring
Add a `proxy mirror` CLI command and `/api/mirror` API endpoints that pre-populate the cache from various input sources: individual PURLs, SBOM files (CycloneDX and SPDX), or full registry enumeration. The mirror reuses the existing handler.Proxy.GetOrFetchArtifact() pipeline so cached artifacts are identical to those fetched on demand. A bounded worker pool controls download parallelism. Metadata caching is opt-in via `cache_metadata: true` in config (or PROXY_CACHE_METADATA=true). The mirror command always enables it. When enabled, upstream metadata responses are stored for offline fallback with ETag-based conditional revalidation. New internal/mirror package with Source interface, PURLSource, SBOMSource, RegistrySource, and async JobStore. New metadata_cache database table for offline metadata serving. |
||
|
Andrew Nesbitt
|
e45706d808 |
Track applied migrations to skip column checks on startup (#60)
* Track applied migrations to skip column checks on startup Add a migrations table that records which migrations have been applied. On boot, load the set of applied names in one query and only run new ones. A fully migrated database now does 1 query instead of ~12 HasColumn/HasTable checks. Fresh databases created via CreateSchema record all migrations as already applied. Old databases get the migrations table on first MigrateSchema call and each migration is recorded after it runs. Closes #54 * Add benchmark for MigrateSchema on fully migrated database * Optimize MigrateSchema to single query for fully migrated databases Skip HasTable/HasColumn checks when the migrations table already exists. A fully migrated database now does one SELECT instead of ~12 individual column and table checks. * Add migration docs and link from architecture * Add test for upgrade from fully migrated database without migrations table |
||
|
Andrew Nesbitt
|
599fe9e254 |
Fix all golangci-lint issues across the codebase (#32)
* Fix all golangci-lint issues across the codebase Resolve 77 lint issues reported by golangci-lint with gocritic, gocognit, gocyclo, maintidx, dupl, mnd, unparam, ireturn, goconst, and errcheck enabled. Net reduction of ~175 lines through shared helpers and deduplication. * Suppress staticcheck SA1019 for intentional deprecated field usage The Storage.Path field is deprecated but still read for backwards compatibility with existing configs that haven't migrated to the URL field. |
||
|
Andrew Nesbitt
|
8ddf07587c |
Fix golangci-lint errors (errcheck, staticcheck, unused) | ||
|
Andrew Nesbitt
|
2d7cb8eae5 |
Refactoring and features | ||
|
Andrew Nesbitt
|
658e9621d8 |
Add Container, Debian, RPM handlers and enrichment API
Adds proxy support for Docker/OCI container registries, Debian/APT repositories, and RPM/Yum repositories. Includes a new enrichment API for package metadata, vulnerability scanning, and outdated detection. Updates the dashboard with Tailwind CSS, dark mode support, and a security overview section showing vulnerability counts. |
||
|
Andrew Nesbitt
|
1eb9d71bd7 |
Align database schema with git-pkgs for compatibility
The proxy can now use an existing git-pkgs database as a starting point. Packages and versions tables match git-pkgs schema, using PURL-based references instead of integer IDs. The proxy adds its own artifacts table for caching functionality. |
||
|
Andrew Nesbitt
|
41aa11ab66 |
Add sqlx with SQLite default and PostgreSQL option
Replace raw database/sql with jmoiron/sqlx for cleaner query handling. Support both SQLite (default) and PostgreSQL as configurable backends. Configuration via: - CLI flags: -database-driver, -database-path, -database-url - Environment: PROXY_DATABASE_DRIVER, PROXY_DATABASE_PATH, PROXY_DATABASE_URL - Config file: database.driver, database.path, database.url Tests run against both databases when PROXY_DATABASE_URL is set. |
||
|
Andrew Nesbitt
|
7b22638ef7 |
Hello world |
