Inferno

What it is

Inferno is an async load tester for OpenAI-compatible inference endpoints: wrk, but for tokens. It measures the metrics LLM serving actually cares about: time-to-first-token, output tokens per second, and the concurrency level where p95 latency falls off a cliff.

It sweeps concurrency levels (say 1, 2, 4, 8, 16), sends N requests per level from a variable-length prompt set, and prints the table you paste into a capacity-planning doc. Warm-up requests run first and are discarded, so cold TLS handshakes and CUDA graph capture do not pollute the numbers. It works against anything that speaks the OpenAI API: vLLM, SGLang, TGI, llama.cpp, Ollama, or a hosted provider with rate limits.

Why it matters

Generic HTTP load tools answer the wrong questions for inference. Requests per second and byte throughput do not tell you how many concurrent requests a GPU can serve before p95 blows past your product's SLO, what TTFT users will feel at that concurrency, where aggregate throughput plateaus, or whether the endpoint starts shedding load before or after latency degrades.

The output reads like an answer. From the sample run against a local 70B model on vLLM: at concurrency 8, p95 is 1.714s with 330.4 tokens per second aggregate; at concurrency 16, per-request throughput halves while aggregate still climbs to 512.7 tokens per second and p95 hits 3.981s. That is the queueing regime, and roughly 4 seconds of p95 is the practical ceiling for an interactive SLO. The numbers are illustrative; the shape is the point.

The exit code is 1 if any level saw errors, so ramp checks slot into CI: a serving upgrade that moves the cliff fails the build.

Architecture decisions

Measure tokens, not bytes. Non-streaming runs count usage.completion_tokens with a word-count fallback; streaming runs request the terminal usage chunk automatically and otherwise count content-bearing SSE deltas, one per generated token on OpenAI- and vLLM-style servers. TTFT is timed from send to the first content-bearing chunk.

Latency percentiles cover successful requests only, linearly interpolated. Mixing failed requests into latency percentiles hides exactly the degradation you are looking for, and error rate gets its own column.

Round-robin prompt assignment, so every concurrency level sees the same mix of short, medium, and long prompts. Comparing levels against different prompt sets would be comparing different workloads. Custom sets come from JSONL, one request per line; the built-in set has six prompts.

The JSON report contains every per-request measurement, not just aggregates, so histograms can be re-derived offline instead of trusting the tool's buckets. Tests run against a deterministic in-process fake server, because a GPU benchmarking tool should not need a GPU to develop. Matplotlib charts are an optional extra that fails with an actionable message, since a load tester should not force a plotting dependency onto CI.