OpenTelemetry: send MinerU logs to Axiom

If you're running a serverless worker on RunPod, you can't ssh in to read logs and you can't run a sidecar agent — the worker scales to zero between jobs. You need to ship logs and metrics off the box during the request lifetime, on the worker's own time. The mineru-runpod template includes an OpenTelemetry exporter for exactly this, and Axiom is the sink I picked for my own deployment.

This post is the exact env-var layout. If you use a different OTLP backend (Honeycomb, Grafana, Datadog, Jaeger, your own OpenTelemetry Collector), the observability guide covers the vendor-neutral setup; come back here only for the Axiom-specific values.

Setup time from a fresh Axiom account to logs flowing: ~10 minutes, dominated by waiting for the worker's next cold start.

Why use Axiom for serverless worker observability?

Axiom ingests OTLP/HTTP directly, meters data usage rather than hosts, and has no agent to install. That fits workers which scale to zero between jobs. Check Axiom's current plan limits against your own log volume rather than estimating from worker count.

Three concrete reasons it fits this workload:

• OTLP-native ingest. No collector to run inside the container, no daemonset, no Fluent Bit config. The mineru-runpod worker calls Axiom's regional edge endpoint (https://eu-central-1.aws.edge.axiom.co/v1/logs or the US variant) directly through the OpenTelemetry Python SDK. That direct path fits a scale-to-zero worker without adding another long-running component.
• Usage-based limits instead of a host agent. Ephemeral workers can come and go without installing or licensing one agent per host. Axiom publishes current ingestion, storage, query, and retention limits in its limits reference.
• APL reads like SPL. Axiom Processing Language sits between Splunk SPL and KQL ergonomically. Filter by attribute, group by backend, drill into a span: the queries you actually run during an incident are easy in APL. No tutorial needed if you've used either.

I have no affiliation with Axiom. They're the backend I picked for my own deployment after looking at Honeycomb, Grafana Cloud, and self-hosted Jaeger.

How do I create Axiom datasets for OpenTelemetry?

Create three datasets in the Axiom UI. Axiom's current OpenTelemetry guidance calls for a dedicated dataset for each signal:

1. Sign in to Axiom and go to Datasets → New dataset.
2. Create mineru-logs with Events type.
3. Create mineru-traces with Events type.
4. Create mineru-metrics with Metrics type.

Names are arbitrary — substitute whatever fits your naming convention. I prefix everything with the service name so multiple endpoints (staging, prod, experiments) don't collide in queries.

How do I generate an Axiom API token for OTLP ingest?

In the Axiom UI, go to Settings → API tokens → Generate new token. Use an Advanced API token (prefix xaat-) and grant Ingest scope on all three datasets. Missing scope on one dataset surfaces as 403 Forbidden for that signal while the others may continue to work.

Treat the token like any production secret: configure it only in the RunPod endpoint environment, restrict dashboard access, never check it into git, and rotate it when access changes.

If your Axiom workspace is in the EU region, the management API lives at https://api.eu.axiom.co (US is https://api.axiom.co). This is the host you query for token CRUD and REST queries, not the OTLP ingest URL — that's a separate edge-deployment hostname documented in the env-var section below.

What environment variables ship RunPod worker telemetry to Axiom?

Set four environment variables on your RunPod endpoint: one base endpoint and one header value for each signal-specific dataset.

Enter these key-value pairs in your endpoint's Environment Variables section in the RunPod dashboard, substituting your token and dataset names. These are dashboard values, not shell commands:

OTEL_EXPORTER_OTLP_ENDPOINT=https://eu-central-1.aws.edge.axiom.co
OTEL_EXPORTER_OTLP_LOGS_HEADERS=Authorization=Bearer xaat-YOUR-TOKEN,x-axiom-dataset=mineru-logs
OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Bearer xaat-YOUR-TOKEN,x-axiom-dataset=mineru-traces
OTEL_EXPORTER_OTLP_METRICS_HEADERS=Authorization=Bearer xaat-YOUR-TOKEN,x-axiom-metrics-dataset=mineru-metrics

The endpoint URL is your Axiom edge deployment, NOT api.axiom.co / api.eu.axiom.co. This is the single biggest gotcha and the one that cost me hours when I first set this up. The api.* hosts are for management API (token creation, queries via REST). OTLP ingest goes to your workspace's edge deployment hostname. As of writing, the two are:

• US East 1: https://us-east-1.aws.edge.axiom.co
• EU Central 1: https://eu-central-1.aws.edge.axiom.co

Use the one that matches the region you picked when creating the Axiom workspace. The current full list lives at Axiom's edge deployments doc. If you send OTLP to api.{eu.}axiom.co, Axiom returns 400 mismatched region or 403 forbidden depending on path — the OTel SDK logs only the HTTP status code, so you'll see Failed to export ... code: 400 (or 403) with no clue why. Axiom support flagged this for me after I'd been chasing 403s for an hour against the wrong host.

How the worker routes each signal:

• Logs → OTEL_EXPORTER_OTLP_LOGS_HEADERS selects mineru-logs.
• Traces → OTEL_EXPORTER_OTLP_TRACES_HEADERS selects mineru-traces.
• Metrics → OTEL_EXPORTER_OTLP_METRICS_HEADERS overrides for metrics only, with the distinct x-axiom-metrics-dataset header that Axiom's metrics ingest requires.

The three dataset names must exactly match the names you created, and the token must have ingest scope on all three. The names above are examples, not magic strings. A wrong name returns 404; missing scope returns 403. The caller still receives its parse result, so inspect worker stdout for exporter errors.

Three details that trip people up:

Base URL, not the full path. Set the endpoint to the edge-deployment root, e.g. https://eu-central-1.aws.edge.axiom.co, NOT https://eu-central-1.aws.edge.axiom.co/v1/traces. The OpenTelemetry Python SDK appends /v1/traces, /v1/logs, /v1/metrics per signal automatically. If you set the full path, the SDK double-appends and Axiom returns 404.

Header case differs by signal. Events ingest uses x-axiom-dataset. Metrics ingest uses x-axiom-metrics-dataset (different header name, with -metrics- in it). Copying the events headers into OTEL_EXPORTER_OTLP_METRICS_HEADERS as-is sends metrics to the events dataset and Axiom's metrics view stays empty.

The worker uses HTTP/protobuf. Axiom's metrics endpoint requires protobuf. The published image constructs the Python HTTP/protobuf exporters directly; there is no protocol environment variable to set.

Save the variables, redeploy any active workers (or wait for the next cold start), and Axiom should see records within ~60 seconds of the first request that hits a warm worker. The metric reader flushes every 10 s; traces and logs flush every 500 ms.

How do I verify OpenTelemetry data is reaching Axiom?

Send one request to the worker, then check each signal's dataset.

In the Axiom UI:

• Stream → select mineru-logs. Each record carries service.name=mineru-runpod, runpod.endpoint_id=<your-endpoint>, and job_id for correlation.
• Traces → select mineru-traces. Each job has a mineru.job root span with fetch, parse, and package children; mineru.warmup appears at worker boot.
• Metrics view → set dataset = mineru-metrics → after the first 10-second flush you should see mineru.jobs.total, mineru.job.duration, the GPU memory gauges, and the rest of the metric catalog.

If nothing arrives in any view, open the worker's stdout (RunPod dashboard → Logs) and grep for Failed to export. The OTel SDK logs each retry with the HTTP status code:

• Failed to export ... code: 404, reason: Not Found — the dataset name in one of the *-dataset headers doesn't exist in your Axiom workspace. Rename the dataset or update the env var so they match.
• Failed to export ... code: 403, reason: Forbidden — first confirm that the endpoint is the workspace's regional edge deployment, not api.axiom.co or api.eu.axiom.co. If the host is correct, add Ingest scope for the affected dataset to the token and update the secret in RunPod.
• Failed to export ... code: 400, reason: Bad Request — the dataset resolves and auth works, but the payload is being rejected. Causes: region mismatch (workspace is EU but you're hitting us-east-1.aws.edge.axiom.co, or vice versa — Axiom returns mismatched region in the body), wrong header name on metrics (use x-axiom-metrics-dataset, not x-axiom-dataset), or the dataset was created of the wrong type for the signal.
• Failed to export ... code: 401, reason: Unauthorized — the API token is wrong or expired. Generate a fresh token in Settings → API tokens and update the env var.
• No Failed to export lines AND no [mineru-telemetry] init failed either — OTEL_EXPORTER_OTLP_ENDPOINT is empty or the worker hasn't cold-started since you set it. RunPod env-var changes only take effect on the next cold start; warm workers keep the previous values.

Which APL queries help debug a mineru-runpod worker?

APL queries can answer which errors fired, which parses are slowest, and how throughput changes by endpoint. Query mineru-logs for log records and mineru-traces for spans. Adjust attribute paths to match how your workspace expands OTLP attributes.

Recent errors, grouped by error type:

['mineru-logs']
| where ['service.name'] == "mineru-runpod"
| where ['severity_text'] == "ERROR"
| where _time > ago(1h)
| summarize count() by tostring(['attributes.error_type'])
| sort by count_ desc

Slowest parses in the last hour:

['mineru-traces']
| where ['name'] == "mineru.parse"
| where _time > ago(1h)
| project _time, duration = ['duration'], backend = ['attributes.mineru.backend'], input_format = ['attributes.mineru.input_format']
| sort by duration desc
| take 20

Throughput by endpoint over time:

['mineru-traces']
| where ['name'] == "mineru.job"
| where _time > ago(24h)
| summarize jobs = count() by endpoint_id = tostring(['resource.runpod.endpoint_id']), bin(_time, 5m)
| render timechart

Attribute paths in APL depend on how Axiom unrolls OTLP records — ['resource.runpod.endpoint_id'] works in my workspace but yours may need ['runpod.endpoint_id'] directly. Run a quick | take 5 | project * against the dataset first to see the actual field names your workspace produces.

Does enabling OpenTelemetry slow down cold starts?

In my deployment, enabling OTel added roughly 200–500 ms to a fresh boot for SDK initialization and DNS resolution. That work did not visibly repeat on the measured fast FlashBoot path. It is small beside a ~110-second fresh model boot, but measure it if startup latency matters to your workload.

Those numbers came from an RTX 4090 with vlm-auto-engine: the fresh boot was ~110 s (image pull, vLLM initialization, model load, and warmup), with 200–500 ms for OTel initialization. See the FlashBoot investigation for the observed fast-path evidence and its limits.

If you're cost-sensitive about cold starts and don't need observability on every deployment, leave OTEL_EXPORTER_OTLP_ENDPOINT unset. The mineru-runpod worker skips the OTel SDK import entirely when that variable is empty — zero overhead, zero behavior change. Flip it on for the endpoints where you actually want the visibility (production, staging) and leave it off for experimentation runs.

Where does this setup fall down?

Three real limitations: plan limits change, GPU metric cardinality adds up, and OTLP/HTTP export adds a small amount of cold-start work. None is a blocker for a small deployment, but each deserves measurement as volume grows.

Plan ceilings. Axiom currently publishes separate limits for monthly data loading, stored data, query compute, datasets, and retention. Those numbers have changed substantially since this integration first shipped. Check the live limits and monitor actual usage; keep worker logging at info unless you need a short diagnostic window.

Metric cardinality. The GPU gauges (mineru.gpu.memory_used_bytes, mineru.gpu.utilization_percent) emit one time series per device label per gauge per worker. A multi-GPU worker times multiple worker instances times four GPU metrics multiplies fast. Axiom's metrics pricing is per-event rather than per-series, so this is a "watch the bill" concern rather than a hard limit. If you scale to dozens of concurrent workers, drop the device label or sample less frequently.

Cold-start latency. OTel init adds 200–500 ms on first-ever boot. For most workloads this is dominated by the existing ~110 s of vLLM init, so it doesn't matter. If you're optimizing the cold start specifically (chatbot-style low-latency workloads, for example), benchmark with and without OTel before committing.

Logs are mirrored, not exclusive. The worker still writes stdout JSON to RunPod's dashboard regardless of OTel. That's deliberate: RunPod's UI remains a working fallback when the OTel pipeline misbehaves. The cost is paying twice for log storage if you care about long retention in both places. Most teams don't, and the duplication is the price of the dashboard fallback.

FAQ

Does Axiom support OpenTelemetry natively?

Yes. Axiom ingests OTLP/HTTP traces, logs, and metrics on /v1/traces, /v1/logs, /v1/metrics paths — but the base hostname is your region's edge deployment, not api.axiom.co. The two as of writing are https://us-east-1.aws.edge.axiom.co and https://eu-central-1.aws.edge.axiom.co (full list at Axiom's edge deployments doc). The OpenTelemetry Python SDK in the mineru-runpod worker speaks this directly with no Collector or agent in between. See Axiom's OpenTelemetry docs for the full list of supported signals and headers.

Why aren't my metrics showing up in Axiom?

Three common causes: OTEL_EXPORTER_OTLP_METRICS_HEADERS uses x-axiom-dataset instead of x-axiom-metrics-dataset; the token lacks ingest scope on the metrics dataset; or the dataset was created as Events instead of Metrics. Each failure appears in worker stdout while parsing continues.

What's the cheapest way to observe a RunPod serverless worker?

Axiom can be a low-setup option because the worker sends OTLP/HTTP directly and needs no collector. Whether it is cheapest depends on current plan limits and your data volume. RunPod's dashboard remains useful for immediate log triage; an OTLP backend adds cross-worker queries, metrics, traces, and alerts.

Can I send traces and logs to the same Axiom dataset?

Do not use that as the default. Axiom's current OpenTelemetry guide requires a dedicated dataset for each component. Use OTEL_EXPORTER_OTLP_LOGS_HEADERS, OTEL_EXPORTER_OTLP_TRACES_HEADERS, and OTEL_EXPORTER_OTLP_METRICS_HEADERS to route them separately.

How much does OTLP/HTTP export add to cold start time?

My deployment measured 200–500 ms on a fresh worker boot for SDK initialization and DNS resolution. It was not separately visible on the observed fast FlashBoot path. Treat those as workload-specific measurements, not a RunPod guarantee.

Why isn't api.axiom.co the OTLP endpoint?

Because Axiom splits two concerns onto two hostnames: api.{eu.}axiom.co is the management API (token CRUD, REST queries, dashboards), while OTLP ingest goes to the edge deployment hostname for your workspace's region. The split isn't obvious from the OpenTelemetry side because most other backends expose ingest on the same hostname as the management API. Axiom's own OpenTelemetry guide and edge deployments doc document the edge URLs, but it's easy to miss if you start from a generic OTel tutorial.

Does enabling OpenTelemetry require an OpenTelemetry Collector?

No. The mineru-runpod worker uses the OpenTelemetry Python SDK with the OTLP/HTTP exporter, talking directly to Axiom's ingest endpoint. A Collector is useful when you want to fan out to multiple backends, apply sampling rules, or buffer locally — none of which apply to a single-sink serverless worker shipping into Axiom.

Same template, different backend

The four environment variables above are the Axiom-specific part. Other OTLP/HTTP backends may accept one shared OTEL_EXPORTER_OTLP_HEADERS value for all signals. The OpenTelemetry guide covers the vendor-neutral layout and metric catalog.

If this saved you time, the easiest way to say thanks is signing up for RunPod through this link. Star the repo on GitHub for updates.