- GetProjectsUsingPackage no longer counts each project's components
per row; the package page only needs names and versions. The ?purl=
listing leaves component_count out accordingly.
- The delete button sends to DELETE /ui/api/projects/{project}, the same
handler as the API and still behind the token, so it sits under /ui
like the cache deletes.
- The project page answers 500 when its components can't be read,
instead of showing an empty project.
- DELETE of a path ending in /sbom that names no project answers 400
with a hint, since it's most likely the upload path.
- The upload endpoint lists JSON, XML and plain text (SPDX tag-value)
as accepted.
- The docs say that versions are compared as the SBOM spells them.
FromPURL now names packages the way their handlers store them:
- NuGet IDs are lowercased, as the NuGet handler stores them.
- PyPI names are normalized as in PEP 503. The PyPI handler takes the
name from the file name, which spells "." as "_", so
pkg:pypi/zope.interface has to match the cached zope-interface.
- Conda package URLs without a channel qualifier are skipped: the conda
handler records packages as channel/name, so they can never match.
- The Go main-module check ignores case. syft keeps the module path's
case in its mainModule property, while golang package URLs are
lowercased.
A table test checks every mapped PURL type against the name its handler
passes when caching.
The vulnerabilities table is only written by tests: nothing in the proxy
calls UpsertVulnerability, so the dashboard card, the per-project count
and the per-component markers were always empty. Remove them along with
GetComponentVulnerabilities and inventory.Affected, and keep the
outdated marker, which uses the latest version the handlers record.
The page asks for the upload token and sends it to DELETE
/api/projects/{project}, so /ui gains no unauthenticated way to delete.
The token is kept in session storage for the next delete.
A syft SBOM of a repository lists the repository itself: the module in
go.mod and the root of package-lock.json, both without a version or with
the version from package.json, and the module its Go binaries were built
from. The project page showed them as dependencies.
Entries without a version are now skipped, as are lock-file roots (lock
entries that depend on others while nothing depends on them) and modules
syft marks as a binary's main module.
Repositories upload an SBOM of their default branch, for example from
syft in CI, to PUT /api/projects/{project}/sbom. The proxy records the
package versions it lists, keyed by the same unversioned PURL the
handlers record packages under, so the inventory joins with what the
proxy has cached and the vulnerabilities it holds.
That answers which repository uses which version of a package: the new
/ui/projects pages list each project's package versions with outdated
and vulnerability markers, package pages show the projects per version,
the dashboard lists vulnerable versions in use, and
GET /api/projects?purl= does the same for scripts.
The feature is off by default. Uploads and deletes need projects.token;
an upload replaces the project's components in one transaction.
projects.max_age removes projects that stopped uploading.
Within metadata_ttl cached npm and Composer metadata is served without
asking upstream, so a new release can take that long to show up. Both
registries publish a feed of the packages that changed.
upstream.npm_change_feed and upstream.composer_change_feed (off by
default, require cache_metadata) start a reader per feed that runs every
minute and clears fetched_at on the cached rows of listed packages that
were fetched before the read. The rows keep their document and ETag, so
the next request revalidates them upstream. Routed packages are cached
under other keys and never match. A Packagist resync expires all cached
Composer metadata. Failed reads back off up to 15 minutes. The feed
position is kept in memory, so after a restart the reader starts from the
feed's current position.
Alpine mirrors keep only the current build of each package, so once a
package is upgraded `apk add curl=8.5.0-r0` fails: the current index no
longer lists the old build, even when the proxy still has it cached.
With upstream.apk_snapshots set (e.g. "90d"), each version of an index or
detached signature the metadata cache stores is copied to
_snapshots/alpine/ and recorded in a new metadata_snapshots table with the
time it was first served. /apk/{name}@{YYYY-MM-DD}/ serves the version that
was current at the end of that day; packages below it are the plain path's
requests. The kept bytes are the signed upstream bytes, so apk verifies
them unchanged.
An hourly job deletes versions replaced longer ago than the configured
duration and queues their blobs on the pending-delete queue; the newest
version of each index stays. Packages listed by kept snapshots are not
protected from eviction or retention; the docs say so.
govulncheck reads vuln.go.dev directly, so on networks that only reach
the proxy it cannot run. upstream.go_vulndb (default https://vuln.go.dev)
is now served at /go/vulndb for `govulncheck -db`.
Only the files the database API defines are passed on (index/db,
index/modules, index/vulns and ID/{id}, as .json and .json.gz), with IDs
limited to letters, digits, '-' and '_'. They are stored and served
byte-for-byte; .json.gz files are gzip data without Content-Encoding.
All files, entries included, follow the metadata TTL since entries change
when their modified date does, and fall back to the cached copy when the
database is unreachable. Non-JSON responses are refused so a block page
does not replace the copy kept for an outage. HEAD is allowed under
/vulndb/, because govulncheck checks a non-default database with HEAD on
index/modules.json.gz.
The dashboard shows the default and the ecosystem overrides, and counts
package overrides and patterns without naming them, since the UI has no
auth. A package page shows the cooldown that applies to the package and
the kind of rule it comes from.
Also add FillVersionPublishedAt, which stores a publish time only where
none is stored, for recording the versions cooldown holds back.
Release lists and manifests record their sha256 digest and are read back
through storedMetadataEntry and openMetadataEntry, so a corrupted copy is
fetched again. pkg:swift PURLs naming a source repository are looked up
through the registry's /identifiers.
proxy mirror and mirror jobs now take pkg:swift PURLs whose namespace
is the registry scope, such as pkg:swift/apple/swift-nio@2.62.0 for
apple.swift-nio. The source archive is fetched from upstream.swift,
checked against the SHA-256 checksum in the release metadata and cached
under the key the archive route serves. PURLs whose namespace is a
source repository URL don't name a registry identity and are refused.
Unversioned PURLs list releases from upstream.swift.
SwiftPM reads the release list and Package.swift before it downloads
an archive, and the handler passed both through uncached, so a mirrored
release still couldn't be resolved with the upstream down. With
cache_metadata the handler now keeps them in the metadata cache along
with their Link header, which carries release list pages and the
version-specific manifests, and serves them when the upstream fails.
Manifests are keyed by tools version, so the 5.8 of an alternate link
and the 5.8.0 SwiftPM requests share a key. The mirror stores every
release list page and each manifest a release links.
On a cache miss the proxy stored the whole artifact before sending any of
it, so clients with an idle timeout gave up on large artifacts: git-lfs
after 30 seconds, pacman after 10. The caller that starts a shared fetch
now gets the bytes as they arrive while they are stored; requests joining
the fetch, HEAD and range requests still wait for the stored copy.
The client's copy is checked against the upstream's digest as it is read,
and a mismatch aborts the response before it completes. A client going
away leaves the copy to storage running. A storage error leaves the
client's copy running: the client gets the artifact, and the failure is
logged at error level and counted in proxy_artifacts_served_uncached_total.
A block-mode scanner has to see the whole artifact first, so with one
configured for the ecosystem a miss is handled as before.
proxy mirror and mirror jobs now take pkg:julia PURLs with the uuid
qualifier. The tree hash of the version comes from the General registry,
after checking the uuid is the one General lists for the name, and the
tarball is fetched from upstream.julia at /package/{uuid}/{tree-hash} and
cached under the name and tree hash the Julia handler uses.
The handler names a package from the registry's Registry.toml, which it
loads from the upstream. Without the upstream it fell back to caching and
looking up by uuid, so a package cached under its name, mirrored or
fetched before a restart, was not served offline. It now looks the
tarball up by its tree hash in that case.
Julia versions are listed from General whatever upstream.julia is, as
packages are identified by uuid and tree hash on every Pkg server serving
General.
Mirroring pkg:helm PURLs failed with an unsupported ecosystem error. The
namespace now names the upstream.helm repository, as the Helm handler
caches charts per repository: pkg:helm/{repository}/{chart}@{version}.
The chart is found in the repository's index through the handler's own
index lookup, fetched from the first URL the release lists, as helm does,
checked against the listed digest and cached under the repository,
digest and filename the handler serves it from. The index goes through
the metadata cache, so helm finds the chart there while the repository is
down. Unversioned PURLs list every version in the index.
Charts without a repository, from an unconfigured one, or listed in an
OCI registry are refused; the last are pulled through /v2, and the error
names the pkg:oci PURL that mirrors them.
git-lfs finds the LFS API under the rewritten remote, so clones through
/git/ already asked the proxy for LFS objects and got 404s. The proxy now
serves the batch API for downloads: it forwards the batch to the upstream,
which decides which objects the repository has, and points each download
at the proxy. Objects are served from the artifact cache. A miss asks the
upstream batch API for a fresh download URL, fetches it and checks it
against its sha256 object ID before caching it.
Download URLs often name another host than the Git server, such as object
storage. The proxy fetches from whatever host the upstream names, with the
usual address checks; upstream.auth credentials stay scoped to their
hosts. Objects are cached per repository. Uploads and locking are refused.
Closes#517.
Holds placed by the rate limit gate are written to a new upstream_holds
table, and each instance reads that table at most every two seconds and
refuses requests to a host any instance holds. Before, every replica had
to get its own 429 before it stopped sending.
Instances waiting on a fetch lease reread the holds when the fetch fails,
so when the holder was rate limited they answer 503 with its Retry-After
instead of each sending one more request to the limited host.
A HEAD on a cache hit opened the stored object and closed it unread.
On S3 and Azure that is a GET whose body is thrown away, which can also
drop the pooled connection; on GCS it is a GET as well.
The server now marks HEAD requests in the request context, and readStored
answers a marked cache hit from the database record. Size, content type
and ETag already came from the record. Accept-Ranges depended on the
opened reader being seekable, so backends now say whether their readers
are (Blob does, GCS does not) and the HEAD response matches the GET.
A HEAD still counts as a cache hit, as before. DirectServe redirects are
unchanged since they are decided before the object would be opened.
Mirroring pkg:composer failed with unsupported ecosystem. A tagged release
is now mirrored with the dist archive upstream.composer lists, under the
file name the Composer handler serves it as and checked against the
listed shasum. The release's publish time is stored as the handler stores
it, so the archive is served from the cache without the metadata, with
cooldown on as well.
Branches are refused, since the handler reads the metadata for every
branch download, and so are archive URLs with a query and packages in
upstream.composer_routes.
The mirror now finds the whole version rather than its files, for the
dist type and publish time.
Part of #414.
A conda version is mirrored with every build anaconda.org lists for it,
each cached under the channel, name, version and subdir/filename the conda
download handler reads from its path. The channel comes from the namespace
or the channel qualifier; a package without one is refused, as retention
keys without one are. The subdir, build and type qualifiers select builds.
Which files a version has, which qualifiers select them and where the
handler caches them is now described per ecosystem, so PyPI and conda share
mirrorFiles.
Part of #414.
Mirroring a PyPI version passed each file's sha256 as the integrity
GetOrFetchArtifactFromURLWithIntegrity records with the version. Wheels of
one version share the name and version they are cached under, so the last
file's checksum replaced the others', and serving the other wheels from
the cache failed the integrity check.
The mirror now passes the sha256 as a digest, which is checked against
that file's content hash alone.
proxy mirror and mirror jobs failed for pypi with "unsupported
ecosystem", since the resolver they use has no registry for it. A PyPI
version has several files, so the mirror now lists them from
upstream.pypi's JSON API, through the same client as version listing,
and fetches each one, verified against its sha256. The file_name
qualifier limits it to one file.
Files are cached under the name and version the download handler reads
from the filename, which can differ from the PURL (zope.interface's
wheels are zope_interface), so pip is served them from the cache.
internal/enrichment now adapts the shared module instead of calling
registries and OSV itself. License categories in the API and the web UI
come from enrichment.CategorizeLicense, which normalizes the license
first. Vulnerability fixed versions come from the affected range of the
queried package.
A generic upstream forwards any path below /generic/{name}/. The
documented mise setup puts a GITHUB_TOKEN on api.github.com to raise
the rate limit, so anyone who could reach the proxy could make any GET
request with that token, including for private repositories it can
read, and the answers went into the shared metadata cache. upstream.git
already refuses an upstream with credentials unless it lists its repos.
upstream.generic_paths now lists, per generic upstream, the paths it
may serve, with the same pattern syntax as git repos. Other paths
answer 404 without reaching the upstream, and so does a path holding
'%', '?', '#' or a backslash, which the upstream could read as a
different path than the one that matched. Config validation refuses a
generic upstream that upstream.auth has credentials for unless it has
generic_paths, which also covers a github.com token reaching the
generic github upstream. Upstreams without credentials keep serving
every path.
The docs and example config show the paths mise needs and recommend a
fine-grained token limited to public repositories.
Without X-Frame-Options or a frame-ancestors policy another site could
frame a version page and trick a signed-in user into clicking Delete.
Every /ui response now sends both; browsed archive files keep their
sandbox policy alongside frame-ancestors.
A page on another site could make a visitor's browser POST a text/plain
body that parses as JSON and start mirror jobs. Browsers only send
application/json cross-origin after a CORS preflight, which the proxy
never answers, so HandleCreate now refuses other content types with 415.
Shutdown cancelled the eviction, retention, reclaim, backfill and stats
loops but closed storage and the database without waiting for them, and
the stats loop never stopped. A pass still running kept the database
open, which on Windows left proxy.db locked after a server test. Shutdown
now waits up to 30 seconds for them, and for mirror jobs.
The job store tests left their jobs running after the test, and on
Windows the test's directories could not be removed while a job was
still writing to them. JobStore.Wait now waits for jobs to return, and
the tests cancel their jobs and wait for them.
A schedule's claim was kept when its run was cut off, so after a
restart every instance skipped it for most of an interval. The scheduler
now drops the claim when the run ends with its context cancelled.
Instances sharing a Postgres database each fetched a missed artifact from
upstream; only callers within one instance shared a fetch. The instance
that fetches now holds a lease on the miss in a new fetch_leases table and
renews it while it downloads. Another instance missing the same key polls
the cache and serves the stored copy, or takes the lease over when the
holder releases it without storing anything or stops renewing it.
Metadata misses use the same lease when metadata caching is on with a TTL
above zero; without a stored copy there is nothing to wait for.
proxy_fetch_lease_waits_total counts the misses that waited. Reclaim
deletes leases left by stopped instances.
Refs #543
Instances started together on an empty database created the schema at the
same time and all but one failed on duplicate types. Schema creation and
migrations now run under a Postgres advisory lock.
Eviction, reclaim, retention and the Gradle build cache sweep take a
per-job advisory lock for each pass, and an instance that finds the job
running elsewhere skips the pass. Before, every instance evicted the whole
overshoot over max_size at once and emptied the cache.
Mirror schedules record their last start in a new job_runs table, so each
schedule runs once per interval instead of once per instance.
Refs #543
