RunPod FlashBoot: a four-request test

If you're shipping vLLM or any heavy ML model on RunPod Serverless, you've probably looked at FlashBoot, ticked the checkbox, and then watched your cold starts still take 60-120 seconds. RunPod's marketing says "1-second cold starts." Their docs describe FlashBoot as "pre-loading container images." Neither of those matches what most ML workloads actually see.

I ran four cold-start tests on a deployed RunPod endpoint serving a vLLM-backed PDF parser. The wall-clock numbers ranged from 7 seconds to 7 minutes. The point of this post is to explain why — what FlashBoot actually does at the systems level, when it kicks in, and how to set up your worker so it kicks in more often.

What does FlashBoot actually do?

The four measurements below are consistent with a process-state snapshot or functional equivalent. On the fast path, the Python process, warm GPU allocations, vLLM child process, and IPC state appeared to return together. RunPod does not publicly document the exact implementation, so I separate observed behavior from that mechanism inference below.

The operating model that fit every request was host + image SHA, not endpoint alone. When the next scale-from-zero appeared to land in a different host context, the worker booted fresh and paid the full warmup cost. This is measured behavior, not a published scheduling guarantee.

The TL;DR for an ML workload: set up an eager warmup at worker boot, then let FlashBoot do its thing. Each new host pays the warmup tax once. Subsequent scale-from-zeroes on that same host get the snapshot restore and finish a typical request in single-digit seconds.

Why do "cold" starts sometimes take 7 seconds and sometimes 110?

Because they're hitting different parts of the per-host model. Four consecutive requests against the same endpoint, single-page parse on each, with a deliberate scale-to-zero between every one:

First hit on a fresh host pays ~110 s for the warmup. Every subsequent restore on that same host is ~7-8 s. A new host, when RunPod's scheduler picks one, starts the cycle over.

The 456 s on Request 1 included a one-time image pull (the worker image is ~27 GB; this was the first time that physical host had ever seen it). Strip that off and you get ~110 s of actual boot work, which matches Request 3 exactly.

How can you tell if a request hit a snapshot restore?

By what's missing from the worker logs. A FlashBoot-restored worker skips its boot sequence entirely — no fitness checks, no Python import logs, no vLLM engine initialization, no model load. The first log line is Jobs in queue: 1, immediately followed by your handler's "starting job" entry.

Compare a fresh boot to a snapshot restore for the same request shape:

Fresh boot (Request 3):

04:45:45  Running 7 fitness check(s)...
04:45:46  All fitness checks passed. (1285.99ms)
04:45:46  [mineru-warmup] starting (backend=vlm-auto-engine ...)
04:45:51  Using vllm-async-engine as the inference engine for VLM.
04:46:23  Initializing a V1 LLM engine (v0.11.2) ...
04:46:47  Model loading took 2.1601 GiB memory and 18.41 seconds
04:47:14  torch.compile takes 22.81 s in total
04:47:17  init engine (profile, create kv cache, warmup model) took 30.66 seconds
04:47:18  get vllm-async-engine predictor cost: 87.26s
04:47:28  [mineru-warmup] done in 101.5s
04:47:28  Jobs in queue: 1
04:47:28  Started.
04:47:28  "starting job" {...}
04:47:34  "done" {...elapsed_seconds: 5.58...}

Snapshot restore (Request 4):

04:51:25  Jobs in queue: 1
04:51:25  Started.
04:51:25  "starting job" {...}
04:51:26  Using vllm-async-engine ...   (instant — engine handle restored from snapshot)
04:51:30  "done" {...elapsed_seconds: 4.58...}

No boot sequence. Three timestamps. The vLLM engine subprocess PID from the previous boot is reused — same EngineCore_DP0 pid=NNN from the snapshot. If you grep your own worker logs for the gap between Jobs in queue: 1 and the previous activity, you'll see whether RunPod did a fresh boot or a snapshot restore.

What state survived in this test?

The fast restore retained the state that mattered to this workload:

• Python interpreter state. Module imports stay loaded. Globals (job counters, contextvars, signal handlers) keep their values. The MinerU engine registry returns the same handles it returned before the snapshot.
• GPU VRAM. Model weights (~2.16 GiB for our VLM), vLLM's KV cache (~8.17 GiB on a 24 GB card), and captured CUDA graphs (~0.3 GiB) all survive. The first request after restore parses with the same allocations it had before.
• The subprocess tree. vLLM runs its engine in a child process for memory isolation. That subprocess gets captured along with the parent and restored with its IPC pipes intact. The engine PID persists.
• torch.compile cache. The JIT-compiled Dynamo / Inductor output stays valid across restore. No 22-second recompile.

What doesn't survive: snapshot lifetime is limited. RunPod doesn't publish the eviction policy, but obvious triggers include image rebuild (new SHA invalidates the snapshot), and presumably long enough idle on a busy host that the snapshot storage gets pushed out.

What broke before this worked? The asyncio gotcha

The "eager warmup at boot" idea is obvious in principle: run one throwaway parse during worker startup so the model is loaded and warm before any user request arrives. The implementation has one trap.

vLLM's AsyncLLMEngine binds its IPC primitives (transports, queues) to the asyncio event loop that initialized it. If you call asyncio.run(warmup()) followed by runpod.serverless.start(), your warmup creates loop A, runs the parse, then tears loop A down when asyncio.run returns. Then runpod.serverless.start() creates loop B for serving. When the first user request tries to talk to the vLLM engine through loop B, the engine handle is bound to the now-dead loop A. Result:

"error_type": "EngineDeadError",
"error_message": "EngineCore encountered an issue. See stack trace (above) for the root cause."

The engine subprocess itself is still alive. It's only the parent process's IPC reference that's broken.

The fix is to keep the warmup and the serve loop on the same asyncio event loop. RunPod's runpod.serverless.start() internally calls asyncio.run(JobScaler.run()), but JobScaler (in runpod.serverless.modules.rp_scale) is constructible directly and its run() is an awaitable coroutine. So you can compose:

import asyncio
from runpod.serverless.modules import rp_ping, rp_scale
from runpod.serverless.modules.rp_fitness import run_fitness_checks

config = {"handler": handler, "concurrency_modifier": _concurrency_modifier, "rp_args": {}}

async def _bootstrap():
    await run_fitness_checks()
    await warmup_async()          # <- engine binds to THIS loop
    rp_ping.Heartbeat().start_ping()
    await rp_scale.JobScaler(config).run()   # <- and stays here

asyncio.run(_bootstrap())

Now both phases share one event loop. The engine handle stays valid across the warmup → serve transition. In the measured fast path, that working loop, engine, and IPC state survived scale-to-zero together.

This does reach into runpod-python's internals (the runpod.serverless.modules.* submodules aren't part of the documented public API). Cheap to guard against drift: a unit test that asserts JobScaler exists with the expected constructor and an awaitable run() method. If RunPod refactors, CI catches it before production does.

When does the warmup pay off and when doesn't it?

Per host, not per endpoint. The math depends on your traffic pattern.

The case that stings is "quiet endpoint with sporadic traffic" — a few requests an hour, 10-minute idle gaps, RunPod bouncing between hosts. Without warmup, every cold start would be ~110-130 s. With warmup, you get a mix: some 7-second restores, some 110-second fresh boots. The mix tilts toward fast as the endpoint warms up across more hosts and RunPod's scheduler starts re-selecting them.

If your traffic is sustained enough that you can pin a worker (workers_min=1), you skip the entire question. You're paying for the GPU 24/7 but never paying a cold start. For workloads with even modest cost sensitivity, the warmup + FlashBoot path is the better trade.

What this means if you're shipping vLLM on RunPod

Three takeaways from the live measurements:

1. Always set up an eager warmup at worker boot. Loading the model on first request is silently worse than it sounds — you don't just pay 110 s once per cold start, you pay it every time a host doesn't have a snapshot, AND you forfeit the per-host amortization that makes the second-hit-on-a-host cheap. Without warmup, FlashBoot has nothing to snapshot.
2. Compose warmup and the serving loop under one asyncio.run(). If you asyncio.run() the warmup separately, the engine dies at the loop boundary. The fix is straightforward but the failure mode is opaque (EngineDeadError 75 ms into the first request) — easy to misdiagnose as a vLLM bug.
3. Don't market your cold start as "X seconds" without acknowledging the per-host mix. A snapshot-restore cold start is genuinely 7-8 seconds. A new-host cold start is ~110 s. Both are big improvements over the no-warmup baseline (~110-130 s per request, every request). But your users will see the mix, and a too-clean claim makes the bad days look broken.

The whole investigation was on a 24 GB A5000 / RTX 4090 class GPU running MinerU's 1.2B VLM via vLLM 0.11.2. Larger models and other runtimes may behave differently. If cold start dominates your workload, add boot-time warmup, watch for the same log pattern, and measure before choosing workers_min.

FAQ

Does FlashBoot snapshot the vLLM engine subprocess?

The test strongly indicates that it does: the engine PID persisted across the fast restore and its IPC connection still worked. That proves retained process behavior for this workload, not RunPod's undocumented internal implementation.

Why does my cold start take 60-120 seconds even with FlashBoot enabled?

Check whether the model loads lazily on the first request. This template warms it before runpod.serverless.start(), and the measured fast path later restored that warm state. Move model initialization to worker boot, then verify the result from logs and timings; RunPod does not publish enough detail to promise the same behavior for every runtime.

What's the difference between FlashBoot and a network volume?

A network volume is shared filesystem storage. FlashBoot restored warm process and GPU state in this test. They solve different problems: a volume persists files across workers, while the observed FlashBoot path avoided repeating process and model initialization.

Does FlashBoot work for non-GPU workloads?

Possibly, but this test does not establish it. CPU-bound workloads with heavy imports, in-memory indexes, or JIT compilation may benefit if RunPod restores equivalent process state. Measure the same fresh-host and reused-context sequence before depending on it.

How do I know if my worker is hitting a snapshot restore vs a fresh boot?

Check the worker logs in the RunPod dashboard. A fresh boot shows fitness checks, framework init logs, and any warmup output. A snapshot restore is silent until the first Jobs in queue: 1 line, then jumps straight to your handler's request-processing logs. The presence or absence of the boot sequence is the cleanest signal.

Is FlashBoot the same as RunPod's "Active Workers" tier?

No. Active Workers are a billing tier where you pre-commit to a number of workers that are always on, billed at a discount in exchange for the 24/7 commitment. FlashBoot is a free runtime optimization that applies to flex (scale-to-zero) workers. The two can be combined: an Active Worker on the same endpoint can also benefit from FlashBoot when it cycles, though for a worker that never goes idle there's nothing to snapshot.

Will FlashBoot survive a Docker image rebuild?

No. Each image gets its own SHA, and FlashBoot snapshots are scoped to (host, image SHA). When you push a new image, all existing snapshots are invalid. The first request after a rebuild on any host pays the full cold-start cost (image pull + warmup). Once each host has served the new image once, subsequent restores work normally.

What's next

The runpod-mineru repo wraps this into one Docker image: MinerU 3.4.x, the MinerU2.5-Pro-2605-1.2B VLM, boot-time warmup, structured logging, and the serverless handler. The wrapper code is MIT-licensed; MinerU and its model retain their own terms. The source is on GitHub.

If you want the deeper breakdown of which phases of a cold start cost what, the troubleshooting guide has the per-phase timing table from the same test runs. The scaling guide covers when to pair FlashBoot with workers_min ≥ 1 for predictable latency.