Serverless Cold-Start Warmup Ping: Hiding a 30-Second Model Boot Behind the Pipeline That Was About to Call It

The most frustrating class of production latency is the one that only exists the first time somebody uses your product in an afternoon. Runs that finish in twenty seconds when a colleague clicks through a demo take three and a half minutes when a real user opens the same page after lunch. Nothing has broken. No queue is backed up. The database is idle, the worker is idle, the logs are green. What is happening is that the serverless cold-start warmup ping you never sent is being paid, in full, inside the request the user is waiting on. This is the writeup of the tiny fire-and-forget ping that hid the boot on the judging path of Acurio — our citation-verification product for academic theses (the repo name is zoterohero) — and the two follow-up commits it took before the pattern was actually safe to leave running in production.

The Boot That Was Always Being Paid Inside The Judging Path

Acurio's judging path has an NLI classifier — a natural-language inference model that takes a citation's claim and a candidate source excerpt and decides whether the source supports, contradicts, or is unrelated to the claim. It is one of several signals we combine, but it is the signal that decides most edge cases, and every judged citation calls it. In production the model is deployed on serverless infrastructure that scales to zero when idle. Warm, one nli_batch call takes about 900 milliseconds at p50. After the endpoint has slept — which is anywhere from a few minutes of idle onwards — the same call takes 28 to 52 seconds at p50, with a measured worst case of 238 seconds, because the platform has to spin up a fresh container, download the model weights, load them into GPU memory, and only then serve the request.

The failure mode is not the cold start itself. Every serverless deployment has one, and 30-50 seconds for a large model is not unreasonable — a small model on CPU is smaller, a chunky classifier on GPU with weights on cold storage is what it is. The failure mode is where the cold start lands: inside the judging path, on a request the user is waiting on, after the pipeline has already spent perhaps a minute translating, embedding, and matching. A six-citation run — perfectly reasonable for a short paper or a chapter — finished in 203 seconds wall clock on a session's first click. The immediate re-run of the exact same document, with everything already warm, finished in 24. That is what the user experienced as "the tool is broken on the first click and mysteriously great after that."

The pipeline itself was fine. Everything in front of the NLI call — the source fetches, the translation to English, the embedding pass, the identity-matching that binds a citation to a source PDF — had been optimized to run concurrently, in a shape where the classifier was the last big I/O the run needed. The trouble was that "last big I/O" was also "the first serverless call this process had made in ten minutes," so the pipeline structure that made warm runs fast made cold runs pathological.

Warm Before You Need It: The Claim-Time Ping

The fix does not touch the judging path at all. It touches the moment a worker claims work. In acurio-core — the Rust judging service — the worker has four claim arms: run item, prepass, comparison, and embed backfill. All four are the earliest signals that judging is about to happen. The insight is that by the time any of them finishes its own work and reaches the nli_batch call, the classifier will need to be warm — and if we start the boot in parallel with the claim's own work, the boot happens alongside the translate/embed/match pass instead of after it.

So immediately after a successful claim, we spawn a fire-and-forget task that sends one minimal scoring request to the NLI endpoint. The response body is thrown away. Reaching the endpoint at all is what boots the container. The ping is deliberately outside the metered provider — no usage row, no per-job budget — because a process-level warmup ping does not belong to any one job and cannot be attributed to any one budget. If NLI is unconfigured or the provider's kill switch is on, the ping is a no-op.

Measured over a week of dev traffic, the warm-time p50 stays at 0.9 seconds. The cold-start p50 falls from 30-something seconds to roughly the difference between the boot ceiling and the time the rest of the pipeline needs to reach the NLI call — which in practice is zero, because translate and embed are slower than the boot. The user-visible latency of a six-citation cold run is now the same shape as a warm run. The pattern is a variant of an old serverless-latency playbook: keep something warm, but only when there is real evidence more work is coming.

What The Naive Version Got Wrong

The first draft of this pattern was the shape most engineers would write on a napkin: a mutex-guarded Option<Instant> for the last-fired timestamp, a 60-second cooldown, spawn a Tokio task if the cooldown had elapsed. Simple. It shipped. Then code review — a mechanical one, on the Rust side — pointed at the exact hole that the composition of two obvious ideas produces when neither is complete.

The 60-second cooldown throttles the start of a ping. It does not bound concurrency. A cold start can last up to the serverless platform's per-request ceiling — 330 seconds in our case, which outlives the cooldown almost six times over. In a busy claim loop that fires maybe_spawn_warmup after every successful claim, up to six detached warmup tasks could stack behind a single genuine cold start, each of them queueing duplicate serverless work at exactly the moment the platform is already struggling to boot the first container. The cure looks like the disease. What was meant to hide a cold start would, under a bad enough one, extend it.

The fix is to add an in_flight flag to the same lock the cooldown already lives inside. should_warm now arms both guards on true: it refuses if either the cooldown is unexpired or a ping is already in flight, and if it returns true it stamps last_started and sets in_flight = true in the same critical section. The cooldown still counts from the start instant, so genuine re-warming between waves still happens. The concurrency, previously unbounded during a cold start, is now bounded at one. And because a flip of the provider's kill switch between spawn and send matters — a worker that has been told to stop hitting the endpoint should stop hitting the endpoint — warmup_ping re-checks the kill switch inside the task, matching the per-attempt contract the metered adapter has always had.

The second review pass caught a smaller but nastier version of the same shape. The explicit finish_warmup call at the end of the spawned task cleared in_flight on the happy path. It was unreachable if the task panicked, was cancelled, or the runtime dropped it during a shutdown. in_flight would stay true until the process restarted, and no more warmups would ever fire from that worker. The fix is an RAII drop guard — a zero-size struct whose only job is to clear in_flight in its Drop impl:

struct InFlightReset;

impl Drop for InFlightReset {
    fn drop(&mut self) {
        finish_warmup(&mut lock_warmup_state());
    }
}

Bind one at the top of the spawned task with let _reset = InFlightReset; and the invariant now survives every exit path Rust has: normal completion, panic unwind, task cancellation, runtime shutdown mid-await. The pattern is the systems-programming twin of the debounced-autosave finalize-gate we wrote up in July: both are about making a promise-shaped piece of state (an in-flight thing) terminate correctly no matter how the underlying task ends. In the browser you reach for a promise you track in a ref. In Rust you reach for Drop.

Three Rules That Generalize

Three rules survived this rewrite and generalize to any warmup, keep-alive, or pre-fetch pattern that fires a request the user did not explicitly ask for:

Spend the boot before the user's request needs the thing. The point of a claim-time ping is not to eliminate the cold start — you cannot, the model still has to load — but to overlap it with work you were going to do anyway. Any pipeline stage that reliably precedes an expensive call by more time than the call's cold start is a candidate to warm from. Skip the warmup when the expensive call is the first thing the pipeline does; there is nothing to hide behind.

A cooldown is not a concurrency bound. A "one per 60 seconds" throttle does exactly what it says, and no more. If the thing being throttled can last longer than the throttle window, you need a separate in-flight guard on top of it, in the same critical section as the timestamp. Every serverless-adjacent pattern I have seen fail in production has failed on this: the naive version assumed the work fit inside the window, and the retry storm proved otherwise. This is a close cousin of the SKIP LOCKED discipline in the Postgres backstop-sweep writeup — different mechanism, same underlying discipline about who is allowed to be doing the work right now.

Fire-and-forget must terminate deterministically. A Tokio task, a Node .then() chain, a browser Promise — anything you spawn and stop tracking is one panic, cancellation, or shutdown away from leaking whatever state it was supposed to release. If the state is "another instance of this task may run once I finish," the leak is silent and permanent until the process restarts, which is the worst kind. RAII in Rust, try/finally in JavaScript, defer in Go: all three languages give you a way to make cleanup independent of the happy path, and none of them are more expensive than the bug they prevent.

The composition of these three rules is what made the second-day patch smaller than the first-day feature. The one-line "spawn a warmup" is the shape most people already know. The three lines around it — cooldown, in-flight flag, drop guard — are what makes the shape survive contact with real serverless latency. Skipping any of the three works fine on a synthetic benchmark and pathologically on a real cold start, which is a heuristic worth internalizing: patterns that hide latency are the easiest ones to break in ways that make the latency worse.

If you are wiring up a pipeline that calls a scale-to-zero model, an inference endpoint, or any serverless dependency where the first request of a session pays a boot cost your user notices — book a free AI Potenzial-Check, or read the Next.js worker split with pg_notify writeup for the process-boundary half of the same "who owns the slow work" theme.