A detached fetch waiting on another instance's lease ran the wait under
the request context, so a client leaving cancelled every caller joined
to it. The wait now runs past the client, bounded by the holder's fetch
and the lease expiry.
When the holder's fetch failed, each waiting instance took the lease in
turn and repeated it, so against a hanging upstream the k-th instance
waited about k timeouts. The holder now marks the lease failed. The
instances waiting on it then serve their stale metadata copy if they
have one, and otherwise fetch at the same time.
A schedule's claim was kept when its run was cut off, so after a
restart every instance skipped it for most of an interval. The scheduler
now drops the claim when the run ends with its context cancelled.
Instances sharing a Postgres database each fetched a missed artifact from
upstream; only callers within one instance shared a fetch. The instance
that fetches now holds a lease on the miss in a new fetch_leases table and
renews it while it downloads. Another instance missing the same key polls
the cache and serves the stored copy, or takes the lease over when the
holder releases it without storing anything or stops renewing it.
Metadata misses use the same lease when metadata caching is on with a TTL
above zero; without a stored copy there is nothing to wait for.
proxy_fetch_lease_waits_total counts the misses that waited. Reclaim
deletes leases left by stopped instances.
Refs #543
Instances started together on an empty database created the schema at the
same time and all but one failed on duplicate types. Schema creation and
migrations now run under a Postgres advisory lock.
Eviction, reclaim, retention and the Gradle build cache sweep take a
per-job advisory lock for each pass, and an instance that finds the job
running elsewhere skips the pass. Before, every instance evicted the whole
overshoot over max_size at once and emptied the cache.
Mirror schedules record their last start in a new job_runs table, so each
schedule runs once per interval instead of once per instance.
Refs #543
