Holds placed by the rate limit gate are written to a new upstream_holds
table, and each instance reads that table at most every two seconds and
refuses requests to a host any instance holds. Before, every replica had
to get its own 429 before it stopped sending.
Instances waiting on a fetch lease reread the holds when the fetch fails,
so when the holder was rate limited they answer 503 with its Retry-After
instead of each sending one more request to the limited host.
The transport remembered a challenge for every /v2/<repository> it was
ever challenged on and never forgot one, so a long-running proxy, or a
metadata document naming many registries, grew the map without limit.
Tokens were only pruned of expired entries, and the expiry came from
the token service.
Challenges are now kept per registry host, holding only the realm and
service, and the pull scope for a request is built from its path, so
pulling many repositories from one registry keeps one entry. Both
caches have a size limit, evicting an arbitrary challenge or the token
that expires soonest, and a token is reused for at most an hour,
whatever lifetime the token service claims.
After a 401 the transport fetches a token from the challenge's realm,
using credentials configured for that realm, and sends it to the host
that issued the challenge. It trusted that host if it had any configured
credentials, even unrelated ones, or shared the realm's registrable
domain. The second rule makes every tenant on a shared domain such as
jfrog.io, azurecr.io or pkg.dev the same site, and the first hands a
token minted from one registry's credentials to another registry the
proxy happens to have a key for.
A credentialed token now goes only to a registry on the realm's own
host and port, to one configured with the same credentials as the
realm, or to Docker Hub and GitLab.com, whose registries use a token
service on another host. Anonymous tokens are unaffected.
The scope in the token request also came from the challenge, so a
registry could ask for push access or for other repositories. The proxy
only reads, so it now always asks for pull access to the repository in
the request path.
A response arriving during a hold was sent before it began, so a late
success no longer releases the host; only one after the hold has ended
does.
Artifact fetches check RateLimitGate.Held before calling the fetcher, so
held cache misses fail with the HeldError without counting as failures
in the host's circuit breaker.
Every handler that answers an upstream fetch error with 502 now answers
a held request with 503 and Retry-After instead.
OnHold and RecordUpstreamRateLimitHold drop the unused duration.
A 429, or a 503 with Retry-After, from an upstream host now stops the
proxy sending that host anything for as long as it asked (capped at an
hour), or for a minute doubling up to fifteen when no Retry-After came.
Held requests fail locally: cached metadata is served stale as before,
and anything uncached answers 503 with the remaining Retry-After so the
client paces itself too.
The gate is a RoundTripper under the auth transport, so metadata,
artifact and passthrough fetches all obey it. Metadata fetches had no
retry or circuit breaker, so a rate-limited upstream kept receiving one
request per cache miss for the whole block. /health lists held hosts
under upstream_holds and proxy_upstream_rate_limit_holds_total counts
holds.
Credentials configured for a token service were applied to any challenge
naming it. A host the proxy fetches from, such as one in a Composer dist
URL or a Helm index, could answer a /v2/ request with a Bearer challenge
naming that realm, and the proxy fetched a token with the configured
credentials and sent it to that host. A token cached from a legitimate
pull was handed over the same way.
- Follow a challenge to a realm with configured credentials only for a
registry that has configured credentials of its own, or is on the same
site (registrable domain) as the realm, as registry-1.docker.io is with
auth.docker.io; otherwise pass the 401 back
- Keep following anonymous challenges for any registry, as before
- Document the rule
