OpenTelemetry

The worker already ships with optional OpenTelemetry export for logs, traces, and metrics. It is off by default. Until you set OTEL_EXPORTER_OTLP_ENDPOINT, the OTel SDK is not loaded and the worker continues to write JSON logs only to RunPod's dashboard.

When you turn it on, the OpenTelemetry SDK is configured at worker boot to ship to any OTLP/HTTP-compatible backend. The template itself is vendor-neutral; the only Axiom-specific recipe lives in the Axiom blog post.

Enable OpenTelemetry

Set OTEL_EXPORTER_OTLP_ENDPOINT on your RunPod endpoint to the base URL of an OTLP/HTTP collector. That one environment variable enables all three signals. The published worker sends OTLP/HTTP with protobuf and appends /v1/traces, /v1/logs, and /v1/metrics automatically.

OTEL_EXPORTER_OTLP_ENDPOINT=https://your-collector.example.com
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer your-token
OTEL_SERVICE_NAME=mineru-runpod
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,team=documents

Use the collector's base URL, not a per-signal path. If you include /v1/traces, for example, the exporter appends the path again and the collector returns 404.

The worker also reads the standard per-signal endpoint and header variables:

• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT / OTEL_EXPORTER_OTLP_TRACES_HEADERS
• OTEL_EXPORTER_OTLP_LOGS_ENDPOINT / OTEL_EXPORTER_OTLP_LOGS_HEADERS
• OTEL_EXPORTER_OTLP_METRICS_ENDPOINT / OTEL_EXPORTER_OTLP_METRICS_HEADERS

Use these when a backend stores metrics separately from traces and logs. Unlike the shared base endpoint, each per-signal endpoint is an exact URL and is not given a suffix by the Python exporter. It normally includes the matching /v1/traces, /v1/logs, or /v1/metrics path; follow your backend's setup. The RunPod Hub form exposes all three header overrides because some backends, including Axiom, require a separate dataset for each signal.

After the next worker boot, submit one parse and confirm that you see a mineru.job trace, the matching job_id in logs, and mineru.jobs.total in metrics. The first metric batch can take up to 10 seconds. For an exact Axiom setup and verification checklist, follow the Axiom walkthrough.

What gets emitted

Traces. One span per job (mineru.job) and one span per phase inside it: mineru.fetch_input, mineru.parse, and mineru.package. The boot-time warmup has its own mineru.warmup span. Fetch spans include input size and source kind; parse spans include input format, backend, and requested page range; package spans include the selected transport.

Logs. Every line emitted by the worker's structured logger is mirrored to OTLP, additive to the existing stdout JSON. RunPod's dashboard remains the source of truth; the mirror lets you query the same records from your downstream sink. Each record carries job_id as an attribute and (when a span is active) the OTel trace and span IDs so logs link back to traces in the UI.

Metrics. The full catalog below, exported via the OTLP/HTTP metric exporter on a 10-second batch interval. All histogram metrics use base-2 exponential bucket aggregation, not the SDK default of explicit (linear) buckets. Latency metrics span ms → minutes and byte-size metrics span KB → hundreds of MB; exponential buckets give uniform resolution across those ranges without per-metric bucket tuning. Any modern OTLP backend (Axiom, Honeycomb, Grafana Mimir, Datadog) accepts exponential histograms.

The mineru.source fetch-span attribute uses the same bounded source kind. Full URLs and volume paths remain in the job's results[].source response for caller-side correlation, but they are not exported as telemetry attributes.

Resource attributes are attached to every signal (logs, spans, metrics):

• service.name — mineru-runpod by default; override with OTEL_SERVICE_NAME
• mineru.version — the MinerU library version baked into the image
• runpod.endpoint_id, runpod.pod_id, runpod.gpu_type, runpod.gpu_count — read from the RUNPOD_* env vars RunPod sets on every worker

Compatible backends

Any OTLP/HTTP collector. Tested or known-compatible:

• Axiom — OTLP-native, no agent. See the dedicated walkthrough for endpoint URLs, header layout, and the metrics-dataset gotcha.
• Honeycomb — set OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io and OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=<key>.
• Grafana Tempo / Loki / Mimir — point at your Grafana Cloud OTLP gateway or self-hosted collector.
• Datadog — via their OTLP intake (requires the dedicated endpoint URL).
• Jaeger — via Jaeger's OTLP receiver.
• Any OTel Collector — point at its OTLP/HTTP receiver and let the collector fan out to whatever downstreams you have configured.

If you write up a recipe for a backend not listed here, send a PR adding it as a blog post (one post per vendor, please — this guide stays vendor-neutral).

Performance notes

On one RTX 4090 deployment, enabling OTel added roughly 200-500 ms to a fresh boot for SDK initialization and the first export. That work did not visibly repeat on the measured fast FlashBoot path, but RunPod does not document enough snapshot detail to make that a platform guarantee.

The batch span / log / metric processors flush every 500 ms (spans, logs) or every 10 s (metrics) with a small in-memory queue. Network export happens off the request path; latency-sensitive deployments should still benchmark with their own collector. If the collector is unreachable, the SDK retries with exponential backoff and drops batches that age out; the worker continues serving requests either way.

Failure modes

Bad endpoint URL or auth. The SDK logs Transient error HTTPConnectionPool(...) to stderr (visible in RunPod's logs) and retries. The worker stays healthy and continues serving — the stdout-JSON log channel is unaffected.

Collector goes down mid-job. Same behavior as above. In-flight batches retry until they age out of the queue; new batches keep flowing. Logs in RunPod's dashboard are unaffected (additive mirror, not replacement).

OTel SDK init crashes. The worker logs [mineru-telemetry] init failed, continuing without OTel: ... to stdout and proceeds to serve traffic without telemetry. This is deliberate — a misconfigured exporter must never block worker boot.

Missing dependencies. The container always ships with the OTel packages installed; forks that strip them will fall into the "init crashes" path above (telemetry simply stays disabled). The worker still serves requests.

Reading worker logs

Independent of OpenTelemetry, the worker emits structured JSON to stdout on every log line — see Troubleshooting → Reading worker logs for the schema and field reference. The OTel logs export above mirrors those records to your OTLP backend; the stdout JSON is always the primary channel and remains queryable through RunPod's log viewer regardless of whether OTel is enabled.

Set LOG_FORMAT=text on the endpoint for a human-readable single-line format instead of JSON — useful for local development, less useful once you're shipping to an indexed sink.