Troubleshooting

When something doesn't work, start with the top-level error, then inspect the debug block. Successful parse responses include the requested backend and detected input format. Failure responses still include GPU details, a best-effort cached model path, and completed phase timings, but fields that were not reached before the error can be absent.

How to read the debug block

{
  "debug": {
    "backend": "vlm-auto-engine",
    "input_format": "pdf",
    "model_dir": "/root/.cache/huggingface/hub/models--opendatalab--MinerU2.5-Pro-2605-1.2B/snapshots/<hash>",
    "gpu": {
      "available": true,
      "name": "NVIDIA RTX 4090",
      "compute_capability": "8.9",
      "total_memory_gb": 23.99
    },
    "phase_ms": {
      "fetch_input": 12,
      "mineru_parse": 18420,
      "package": 95
    }
  }
}

What to look at:

The worker also emits structured log lines visible in RunPod's worker log viewer — see reading worker logs below.

Hub build fails on the validator test pod

After every push, RunPod's Hub builds the image and then spins up a real GPU pod to execute .runpod/tests.json. The image is fine; the test pod fails. Three failure modes account for almost everything we've seen here:

"Pod could not be created"

Pod could not be created: This machine does not have the resources to deploy your pod. Please try a different machine.

Cause: RunPod can't allocate the gpuTypeId declared in .runpod/tests.json during the build window. The Docker image is fine — RunPod just couldn't find a free host of that type.

Fix: switch gpuTypeId to a higher-availability pool. The template currently uses "NVIDIA GeForce RTX 4090" because it has the best pool availability across RunPod's regions; "NVIDIA RTX A5000" works too but tends to be scarcer. Re-trigger the build after editing.

nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.9

Error response from daemon: failed to create task for container: ...
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.9,
please update your driver to a newer version, or use an earlier cuda container

Cause: the container's CUDA floor (12.9, inherited from vllm/vllm-openai:v0.11.2) is higher than the CUDA version the host driver exposes. RunPod scheduled the test pod on a host that satisfied allowedCudaVersions on paper but doesn't actually meet the container's prestart-hook requirement.

The trap: allowedCudaVersions tells RunPod "the worker accepts these driver CUDA versions." If older versions are listed there, RunPod is free to schedule on older-driver hosts, and the container's own requirement labels then reject the host at prestart. Result: intermittent failures (depends which host got picked).

Fix: keep allowedCudaVersions in both .runpod/tests.json and .runpod/hub.json aligned with the actual minimum the container needs. For the current vLLM v0.11.x base, that's ["13.0", "12.9"]. Don't pad the list with older versions just because they look harmless — every entry that the container can't actually run on is a future flake.

If you bump vLLM, re-check the CUDA floor from upstream's release notes (vLLM v0.11.0 was the bump to CUDA 13).

Build timeout (30 minutes)

Build exceeded maximum time limit of 1800 seconds (30.0 minutes). Build terminated.

Cause: RunPod's build pipeline has a hard 30-minute ceiling. The image bakes ~17.5 GB of model weights (MinerU2.5-Pro at 2.3 GB, plus all of PDF-Extract-Kit-1.0 at 15.1 GB — snapshot_download pulls the whole repo, with no allow_patterns) and installs vLLM + Torch on top; on a slow build-region day, those steps alone can blow past the cap.

Fix: the Dockerfile already uses hf-xet with HF_XET_HIGH_PERFORMANCE=1 for fast model bakes, and the two model downloads are split into separate RUN layers so a partial cache survives retries. If you still time out:

• Re-trigger the build (often a transient HF-egress slowdown)
• Pin a smaller VLM model via MINERU_VL_MODEL_NAME for http-client backends — doesn't help here since the bake is unconditional
• As a last resort, drop one of the baked models and rely on RunPod's per-endpoint model cache (one model only — see the model caching docs)

Escape hatch: skip the validator entirely

If a release is urgent and the Hub validator is the only thing blocking it, rename .runpod/tests.json to .runpod/tests_.json in the repo (underscore suffix). The Hub validator looks for the exact filename tests.json; the renamed file is invisible to it and no test pod is scheduled. Rename it back when the underlying issue is resolved.

This loses all CI signal — only use it as a temporary unblock, not a default.

"You don't have access to download the artifacts"

Symptom: creating an endpoint fails with an access or permission error about pulling the image, and nothing was ever built on your account.

Cause: there is no public prebuilt image for this template, so there is nothing to pull by name. This repo ships as a source template — RunPod clones it and builds the image into your own account's registry. An endpoint created by typing an image reference by hand (Resources → Serverless → New Endpoint) has nothing to point at and fails exactly this way.

Fix: deploy from the Hub listing instead, or via The Hub → Serverless repos, and let RunPod run the build. It takes ~5–10 minutes and the log is visible in the dashboard. See Deploy for all three paths.

You never build this image locally, so your workstation's architecture doesn't matter — the build runs on RunPod's x86 builders. If you do want to own the image, Option C on the Deploy page covers building and pushing to your own registry, which needs an x86 build host.

VLM backend crashes on Blackwell GPUs (cc=12.0)

Symptom: worker logs show successful model load followed by:

compute_capability: 12.0 >= 8.0
INFO Starting to load model .../MinerU2.5-Pro-2605-1.2B/...
INFO Model loading took 2.16 GiB and 0.36 seconds
CUDA error (...flash-attention/hopper/flash_fwd_launch_template.h:188): invalid argument

debug.gpu.compute_capability in the response is 12.0 (e.g. NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb).

Cause: xformers / flash-attention in vllm 0.11.2 (our base image) ships kernels for Ampere (8.x), Ada (8.9), and Hopper (9.0) — no Blackwell-SM120 code path. On Blackwell, xformers misroutes to the Hopper kernel and crashes during VLM model init.

Why the published image still fails: the Dockerfile currently selects vLLM 0.11.2. MinerU 3.4 allows newer vLLM releases, so the old <0.12 dependency ceiling is no longer the blocker. Changing the base image still requires a coordinated compatibility test across MinerU, CUDA, vLLM, xformers, the baked models, and RunPod's GPU pools; that upgrade has not been validated here yet.

Fix: keep the default gpuIds: "ADA_24,AMPERE_24,AMPERE_48" — all unambiguously pre-Blackwell. If you've manually opted into ADA_48_PRO, that pool can mix in Blackwell SKUs since RunPod groups them under the same pool name — remove it from your endpoint's GPU pool list. For workloads that need a 48 GB Ada-or-newer card, the pipeline backend doesn't use xformers/flash-attn and is unaffected, so it runs on Blackwell fine; only the VLM/hybrid backends crash. See Choosing a GPU for the GPU-pool background.

"Pod scaled to zero" but the next job has a noticeable cold start

Symptom: spiky workload. First job after a quiet period takes a long time; subsequent jobs in the same window are fast.

Cause: RunPod tears down the worker after idle_timeout seconds of inactivity. The next request spins a fresh worker — the cost is unpacking the image, loading the model into VRAM, and (for the VLM backend) JIT-compiling vLLM kernels.

Expected magnitudes (measured on RTX A5000):

Per-phase cold-start breakdown (VLM, A5000 24 GB)

If you're tracking why a cold start takes ~110 s, here's the live measurement we captured against the deployed template. Times are wall-clock between consecutive log entries from RunPod's worker log viewer, totals approximate ±2 s:

Headline observations from this run:

• vLLM engine init dominates (34 s of the 100 s MinerU wrapper time). Of that, torch.compile is 25 s — the single biggest cost.
• Model weight load is only 21 s despite being 2.16 GiB. The image bakes the model into /root/.cache/huggingface/, so this is a local FS read, not a network download.
• Available KV cache memory: 8.17 GiB on a 24 GB A5000. That's vLLM's KV budget after model + activations + reserve. Constraining factor for MINERU_MAX_CONCURRENCY if you try to raise it above 1.
• Maximum concurrency for 8,192 tokens per request: 87.13x — vLLM's in-engine batch ceiling on this hardware. Different from our per-worker concurrency knob; this is sequences per single vLLM forward pass.

Fix: not a bug, but levers if it's a problem:

• Bump idle_timeout (template default is 10 s) — workers stay warm longer, you pay for that time

• Set workers_min=1 — at least one worker is always warm, you pay 24/7 for it

• Enable RunPod's FlashBoot explicitly in the endpoint config (it's on by default for templates from the Hub)

• Eager warmup is now active by default. The worker runs one throwaway parse against the baked test fixture during boot, before runpod.serverless.start() claims the event loop. This loads the MinerU model into VRAM and JIT-compiles vLLM kernels, so the first real request lands on a warm engine. Look for [mineru-warmup] starting (backend=... lang=... fixture=/worker/test-fixture.pdf) then [mineru-warmup] done in Ns in the worker logs. To disable (e.g., for debugging cold-start ordering), set MINERU_SKIP_WARMUP=1 on the endpoint.

  Tune via env vars on the endpoint:

  • MINERU_WARMUP_BACKEND (default vlm-auto-engine) — which backend to warm. Must match the backend most callers will use; warming vlm-auto-engine but serving pipeline requests means the first pipeline call still pays cold-start.
  • MINERU_WARMUP_LANG (default en) — only meaningful for the pipeline backend; VLM ignores it.
  • MINERU_SKIP_WARMUP=1 — bypass entirely (worker falls back to lazy load on first request, ~100s tax).

  Observed snapshot-restore latency for the first request: ~7–8 s wall-clock on A5000 (measured 2026-05-26). The logs and persistent vLLM PID indicate that FlashBoot restored a warm process and its GPU state. See the four-request evidence below.

FlashBoot behavior observed in four requests

In a four-request test on 2026-05-26, FlashBoot behaved like a process snapshot scoped by host and image SHA. RunPod does not publicly document those internals, so this is an inference from timings, missing boot logs, persistent process identity, and host changes.

The four-request investigation that pinned this down. Same short single-page PDF, same parameters every time, worker scaled to zero between every request:

Worker identity is visible in the logs three ways: the EngineCore_DP0 pid=NNN line, the distributed_init_method=tcp://192.168.X.X pod-internal IP, and the request-id suffix. All three agreed: R1+R2 shared one worker context; R3+R4 shared another.

The per-host model:

FlashBoot lookup = (worker host, image SHA)
- match → restore snapshot in ~3 s, parse in ~5 s → ~7-8 s wall-clock
- no match → fresh boot, run fitness checks + warmup → ~110 s wall-clock

The observed restore retained the Python process, MinerU engine handle, vLLM child process, warm GPU allocations, and compiled state. The logs do not prove RunPod's underlying implementation.

Practical implications:

• The boot-time warmup pays off per host that the worker visits, not once per endpoint or once forever. Each new host pays the warmup tax once; every subsequent restore on that same host is fast.
• Snapshot invalidation: the obvious triggers are image rebuild (new SHA), MINERU_SKIP_WARMUP=1, and presumably eventual eviction after long idle. RunPod doesn't document the eviction policy.
• A request that lands on a fresh host still waits for the ~110 s worker boot. Requests restored in the observed warm context avoided that cost.

What controls which path you'll see:

CUDA out of memory

Symptom: handler errors with CUDA out of memory mid-parse. debug.gpu.total_memory_gb shows your card has fewer GB than the workload needs.

Cause: vLLM allocates KV cache up front based on gpu_memory_utilization (default 0.5). Concurrent jobs and unusually complex pages can exhaust the remaining VRAM. A longer document does not steadily increase VRAM when pages are processed sequentially.

Fix:

• Bump to a 48 GB pool for the affected workload (AMPERE_48)
• Switch to the pipeline backend which doesn't use vLLM and is documented at 4 GB minimum VRAM (per MinerU's hardware compatibility table), regardless of doc length
• Reduce MINERU_MAX_CONCURRENCY so fewer jobs share one worker
• Batch very long documents for shorter retries, bounded job time, and smaller outputs; batching is not a VLM memory fix

See Choosing a GPU for the VRAM math.

the job reported COMPLETED but carried no output

Symptom: the client raises MineruClientError: the job reported COMPLETED but carried no output, followed by the cap and the ways out. Callers using the RunPod SDK directly see a bare None instead, with nothing naming size — which is what this client's message exists to replace. Worker logs show the handler completed successfully — [mineru-worker] done: elapsed=Xs phase_ms={...} — immediately followed by:

"Failed to return job results. | 400, message='Bad Request',
 url='https://api.runpod.ai/v2/<endpoint>/job-done/<worker>/<request>?gpu=...&isStream=false'"

Cause: RunPod's /runsync gateway caps the response payload at ~20 MB. The worker built a valid result; when it tried to POST it back via /job-done, the gateway returned HTTP 400 and discarded it. The SDK then sees no output and hands the client None, which it reports as the message above rather than as a Python type name.

Triggers (measured on a real 82-page PDF):

• transport: "inline" — markdown + content_list + middle.json + base64 images add up fast; ~80 pages with embedded images was enough to exceed the cap. If you only need the markdown, narrow with formats: ["markdown"] first.
• transport: "tarball_b64" — gzip compresses the JSON, but the images inside the tarball are already raster bytes, so it often doesn't fit either. Confirmed same failure on the same doc.

Fix: use transport: "s3" for large outputs. The worker uploads the .tar.gz to an S3-compatible bucket and returns only a small presigned URL (~1 h TTL) — no gateway cap in the path.

{
  "input": {
    "file_url": "https://example.com/big.pdf",
    "transport": "s3"
  }
}

Configure the bucket via these env vars on the endpoint (not the template):

The Python client handles the rest via client.save_s3_tarball(result, dest_dir) — it follows result["results"][0]["tarball_url"], downloads the .tar.gz, and extracts. From curl, the entry inside results[] includes tarball_url, tarball_url_expires_in, and bucket_key; download within the TTL.

If you can't wire S3, the fallback is page chunking: split with start_page / end_page into segments small enough that each tarball fits under the cap, then concatenate the .md files client-side. Slower (two cold starts if workers go cold between calls) but no infra changes.

Worker returns ValueError: input bytes do not match any supported format

Symptom: the response has ok: false and the above error message.

Cause: the worker's _detect_format checked the first ~16 bytes against known magic numbers (%PDF, \x89PNG, PK\x03\x04 for OOXML, etc.) and didn't match anything.

Most common reasons:

• file_url returned an HTML error page instead of the file. The URL is wrong, expired, or behind auth. The response body starts with <!DOCT or <html.
• file_b64 was double-encoded or not base64. The decoded bytes are random.
• volume_path points at a file that exists but isn't a supported format (e.g. a .csv or .txt). MinerU doesn't accept plain text — convert to PDF first.

Fix: verify the bytes. Download from your file_url directly with curl, run file on the result, or check that base64 -d < input.b64 | xxd | head shows the right magic bytes.

Worker rejects an input source it used to accept

Symptom: ok: false with one of the messages below and no parse attempted. Each names the field and what about it didn't fit.

Job times out before the parse finishes

Symptom: MineruClientError: endpoint transport failed: timeout after some number of seconds (default 900 s in the client, configurable via timeout=).

Cause: large documents take longer than your client-side timeout, or longer than the endpoint's executionTimeoutMs.

Fix:

• Client-side: pass a larger timeout to parse_document(timeout=3600, ...)
• Endpoint-side: raise the endpoint's execution timeout — 3600 s for full books. deploy.py --execution-timeout 3600 applies it: the SDK's create_endpoint takes no such parameter, so the script sets it afterwards through the REST API's executionTimeoutMs field. If that call fails the summary says NOT SET with the reason, and only then do you need the console. For an endpoint that already exists, change it there or with a PATCH /v1/endpoints/{id}.
• Per-page math (warm worker, GPU- and content-dependent): MinerU upstream cites ~0.5 s/page for the VLM backend on an A100. We measured ~1–10 s/page on an A5000 24 GB depending on content density (uniform multi-page reports run fast; dense financial forms run slow). Pipeline ≈ 3–5 s/page across GPUs. A 1000-page book on pipeline = ~3000–5000 s; on VLM-on-A5000 ≈ ~1000–10000 s depending on content; on VLM-on-A100 ≈ ~500 s. Add 90–130 s for the first call on a cold worker if VLM (model load + vLLM warmup) — the cold-start tax is paid once per worker, not per page.

Reading worker logs

The worker emits one JSON object per line by default (set LOG_FORMAT=text for human-readable output during local development). RunPod's log viewer shows them as-is. To filter in CloudWatch, Loki, Axiom, or any other JSON log sink, key off the level, message, and any of the structured fields.

Typical lines on a successful job:

{"ts":"2026-05-25T18:30:42.103Z","level":"info","logger":"mineru-worker","message":"starting job","job_id":"queued-uuid-abc","backend":"vlm-auto-engine","lang":"en","start_page":0,"end_page":4,"gpu_name":"NVIDIA RTX 4090","compute_capability":"8.9"}
{"ts":"2026-05-25T18:30:48.612Z","level":"info","logger":"mineru-worker","message":"done","job_id":"queued-uuid-abc","elapsed_seconds":6.51,"phase_ms":{"fetch_input":12,"mineru_parse":6420,"package":79},"model_dir":"/root/.cache/huggingface/hub/.../snapshots/<hash>","refresh_worker":false}

On failure:

{"ts":"2026-05-25T18:30:42.789Z","level":"error","logger":"mineru-worker","message":"job failed","job_id":"queued-uuid-abc","error_type":"ValueError","error_message":"input bytes do not match any supported format","phase_ms":{"fetch_input":8}}

On a job that succeeded but lost part of its output — a file MinerU wrote and the worker then could not read:

{"ts":"2026-05-25T18:30:48.400Z","level":"warning","logger":"mineru-worker","message":"response degraded","job_id":"queued-uuid-abc","artifact":"content_list","file":"doc_content_list.json","reason":"unreadable","error_type":"JSONDecodeError"}

message:"response degraded" is one string for every such case, so a single alert rule catches all of them — as does the mineru.degraded.total counter when OpenTelemetry is configured. The same information rides in the response itself under degraded, which is what tells you which document to reprocess — see Incomplete responses. The job is still ok: true, so nothing else will flag it.

Key fields:

Cancellation and worker recycling

Symptom: a job appears in RunPod's queue, then disappears from the worker logs mid-parse with no done line.

Cause: RunPod sent SIGTERM to the worker — either because the endpoint scaled down due to idle timeout, you triggered a recycle from the dashboard, or the worker requested a refresh via the refresh_worker response flag.

When SIGTERM arrives, the worker logs:

{"ts":"...","level":"warning","logger":"mineru-worker","message":"sigterm received, draining current job"}

What happens to in-flight jobs: they drain to completion. After SIGTERM the RunPod SDK stops pulling new jobs, so anything still running was already accepted — the worker finishes it and returns the result normally. Between request phases (fetch_input → parse → package) the worker logs a sigterm received mid-job; continuing to drain breadcrumb so you can see where the job was when the signal landed.

Worker versions up to 1.8.0 instead failed in-flight jobs at the next phase boundary with RuntimeError: worker shutting down, refusing further work. RunPod treats a handler-returned error as terminal (FAILED, never retried), so routine scale-ins surfaced as permanent job failures to clients. If you see that error, upgrade the worker image.

What SIGTERM cannot do:

• Cancellation mid-aio_do_parse. The vLLM forward pass is a blocking GPU call from asyncio's point of view; interrupting it would corrupt the engine state. The worker finishes the current document even after SIGTERM, then exits cleanly.

If you need hard cancellation guarantees mid-parse, that's an upstream MinerU feature request, not a template-level fix.

Getting more help

If your symptom isn't here:

1. Pull the worker logs from RunPod's dashboard — look for [mineru-worker] lines and any tracebacks
2. Check the debug block in the response
3. Open a bug report — the issue template asks for the response and the GPU pool, both of which let you diagnose 90% of issues at a glance
4. Parsing accuracy issues (output is structurally fine but wrong content) belong upstream at opendatalab/MinerU — they're MinerU's responsibility, not this template's