llm-rs vs vLLM on a 7B Model: What's the Same, What's Different
The last post ended with a promise: the 7B models, measured through AIPerf, where quantization and attention start to matter. Here it is, kept short.
The setup: Qwen2.5-7B-Instruct-AWQ (4-bit weights) on one RunPod RTX A5000 (24 GB, 768 GB/s). vLLM 0.34 against llm-rs, both capped at 64 concurrent sequences, both measured with AIPerf 0.13 using fixed lengths, fresh prompts every run (0% prefix-cache hits), and each model’s default sampling settings.
| Output tokens/s | llm-rs | vLLM | llm-rs / vLLM |
|---|---|---|---|
| One at a time, 256 in / 256 out | 100.6 | 117.9 | 85% |
| 256 at once, 256 in / 128 out | 1,184 | 1,395 | 85% |
| 16 at once, 2,048 in / 256 out | 343 | 435 | 79% |
| 64 at once, 2,048 in / 512 out | 673 | 767 | 88% |
A few days ago these ratios were 64%, 66%, 57% and 73%. The gains came from a faster sampler (described below) and faster 4-bit kernels at every batch size:
- Single-token gemvs: 1.15–1.6x faster.
- Batched decode GEMM: 42% less time at 16 tokens.
- Prompt GEMM: 1.19–1.32x faster.
What’s the same
On the serving side, llm-rs now makes the same choices vLLM does:
- Continuous batching with chunked prefill. New requests join as soon as there is room, and long prompts are fed in pieces so running requests keep decoding.
- A paged KV cache with prefix caching. Memory is handed out in 16-token blocks, and requests that share a prompt prefix share its blocks.
- Mixed steps with one variable-length attention launch. A single step carries prompt chunks and decode tokens together, and attention for all of them runs as one launch over a packed batch, in the style of FlashAttention’s
varlenkernels. - CUDA graphs for steady decoding. Repeated decode steps replay a captured graph instead of launching every kernel again from the CPU.
- Sampling on the GPU, with the model’s own defaults. For Qwen that means temperature 0.7, top-k 20, top-p 0.8 and a repetition penalty of 1.05, applied when a request doesn’t say otherwise. Both engines do this.
That last point turned out to matter. My own benchmark tool sends temperature: 0, which picks the most likely token and is nearly free. AIPerf sends no temperature, so the defaults apply. Under those defaults the llm-rs sampler took 2.74 ms of a 12.5 ms decode step: it found top-k, top-p and the random draw with three 30-step searches, each step a full pass over the 152,064-entry vocabulary. Now top-k uses a four-pass radix select, and top-p and the draw run over only the 20 tokens that survive top-k. The cost dropped to 0.49 ms, and the results still match the CPU sampler token for token. Benchmark with the sampling settings your users get, not the ones your tool happens to send.
What’s different
- Rust from the tokenizer to the kernels. Every GPU kernel is Rust, compiled to PTX with cuda-oxide and built into one server binary. Nothing in the request path is Python, PyTorch or Triton. vLLM builds on PyTorch, with CUDA kernels like Marlin and FlashAttention underneath.
- A different kernel for each batch size. llm-rs keeps one copy of the 4-bit weights and picks the kernel by how many tokens the step has:
- One token: a gemv that streams each weight once.
- 2–64 tokens: a split-K decode GEMM that still reads every weight once per step.
- More (prompts): a 128×128 tensor-core GEMM.
vLLM runs its Marlin kernel at every size.
- Integer weights in the multiply. Marlin turns each 4-bit weight into a scaled 16-bit float before the tensor-core multiply. llm-rs feeds the multiply the exact integer
code − zero, and applies each group’s scale once to the f32 partial sums afterwards. This week’s prompt GEMM in that format reaches 66–78 TFLOPS, 1.19–1.32x its predecessor, and brought time to first token on long prompts within 6–8% of vLLM. - Attention shaped for decodes. A decode has only 7 query heads per KV head on this model, so a 64-row tensor-core tile is mostly empty, and three of its four warps used to sit idle. llm-rs’s own varlen kernel instead gives each of the four warps a quarter of every key tile and merges their results at the end. Decode attention reads the KV cache at up to 656 GB/s, 85% of the card’s bandwidth. vLLM uses FlashAttention and FlashInfer kernels.
- A sampler defined by a reference. The GPU sampler must pick exactly the token llm-rs’s CPU sampler picks for the same logits and random draw, and the tests check that at the full 152k vocabulary. That rule is why it selects tokens without sorting: by radix select and running sums, in a fixed order.
Where vLLM is still ahead
Time between tokens, though the gap has narrowed:
| Mean inter-token latency | llm-rs | vLLM |
|---|---|---|
| One at a time | 9.5 ms | 8.15 ms |
| 256 at once, short prompts | 47.3 ms | 37.8 ms |
| 16 at once, 2,048-token prompts | 34.2 ms | 24.7 ms |
| 64 at once, 2,048-token prompts | 65.1 ms | 55.2 ms |
One at a time, the 4-bit gemvs now read the weights at 500–630 GB/s, up from 380–560 (of the card’s 768). What’s left in that step is mostly smaller pieces:
- the BF16
lm_head, 1.5 ms a token; - the RMS norms, 0.65 ms;
- the sampler, 0.49 ms.
Under load, Marlin’s batched GEMM is still ahead of llm-rs’s decode GEMM, which is the 1.2–1.4x gap in the table.
Next
The FP8 KV cache now has its own fast attention kernel too. It halves the cache traffic and runs 1.14–1.35x faster than BF16 attention at 16 or more decodes. At these 2.5k-token contexts, though, attention is too small a share of the step for that to show in serving. For now FP8 mainly buys cache capacity, and its speed should pay off at longer contexts. Next up are the smaller pieces of the single-request step and the batched decode GEMM. All of the numbers above, plus the runs behind them, are in the repo’s docs/PERF.md.