· 8 min read

vLLM vs TensorRT-LLM 2026: How to Choose and Benchmark

A dated, source-backed comparison of setup, hardware, prefix caching, and workload testing, plus an LLM system design interview framework.

TL;DR

Choose vLLM as the first candidate when models change often and TensorRT-LLM as the first candidate when one model stays frozen on NVIDIA GPUs. That is Lyceum’s 2026 selection rule, not a universal speed ranking.

The version date matters. Lyceum reports that TensorRT-LLM’s stable 1.2.1 release still has an engine-build step, while the 1.3 line removes it after release candidate 1.3.0rc20 of 30 June 2026. Northflank’s 15 September 2025 comparison calls TensorRT-LLM setup more complex; treat that as a dated snapshot rather than a permanent property.

Do not choose from a generic tokens-per-second table. Lyceum recommends testing on a recording of your own traffic and measuring how much prompt text repeats. Treat batch size and concurrency as experimental variables: report both latency and throughput for every point instead of selecting one flattering result.

This guide turns those findings into a decision table, a repeatable test procedure, and an LLM system design interview answer. It does not invent a head-to-head benchmark where the sources do not provide one.

What the three sources actually establish

SourcePublication contextWhat this guide takes from it
NorthflankPublished 15 September 2025Side-by-side architecture, model-support, setup, quantization, and ecosystem comparison
Lyceum2026 production guideVersion-specific engine-build status, model-change frequency, repeated prompt share, prefix caching, and testing on your own traffic
vLLM documentationCurrent first-party referenceAutomatic prefix caching behavior and its limits

Northflank describes vLLM’s architecture as “PagedAttention + async GPU scheduling” and TensorRT-LLM’s as “CUDA kernels + graph optimizations + Tensor Cores”. Lyceum explains the practical consequence for teams: vLLM serves a model without a separate build step, which suits teams that change models often, while TensorRT-LLM tunes a model for NVIDIA GPUs.

These sources answer different questions. Northflank supplies the broad comparison, Lyceum updates the deployment and workload-selection details, and the vLLM documentation provides the first-party prefix-caching reference. Combining them is more useful than pretending one article has a timeless leaderboard.

The 2026 decision table

Your constraintFirst candidate to testSource-backed reasonWhat remains to measure
Models change frequentlyvLLMLyceum says no separate build step suits teams that change models oftenCold-start time and correctness for each new model
One frozen model on NVIDIA GPUsTensorRT-LLMLyceum’s selection rule points this workload to TensorRT-LLMWhether tuning produces enough gain on your traffic
You already use NVIDIA’s serving stackTensorRT-LLMNorthflank places it in the “NVIDIA stack (Triton, NeMo, TensorRT)”Integration and operating effort
You need broad Hugging Face or open-source integrationvLLMNorthflank lists the Hugging Face ecosystem and easier open-source integrationYour exact model architecture and features
Large-batch throughput is the priorityBenchmark bothThe remaining sources do not establish a universal large-batch winnerThroughput at your actual latency ceiling
FP8 or INT4 is a requirementTest TensorRT-LLM and vLLM with the same schemeNorthflank lists “Cutting-edge support for quantization and FP8/INT4 inference” for TensorRT-LLMQuality, latency, and throughput for your model
Many requests share the same opening textMeasure prefix caching before switching enginesLyceum says repeated prompt share is a central cost and speed variable, and vLLM can reuse shared prompt startsPrefill savings; decode is a separate phase
Non-NVIDIA accelerators are in scopeDo not assume TensorRT-LLM supportNorthflank lists “NVIDIA-only, no support for AMD or non-CUDA environments” as a TensorRT-LLM convLLM support for the exact device and model

The table gives you a starting hypothesis, not a winner. For a production decision, run both candidates whenever the model and hardware support them.

Prefix caching: measure the prompt before the engine

Lyceum’s most useful contribution is to shift attention from engine branding to workload shape. It separates a request into prefill, when the model reads the prompt and builds its KV cache, and decode, when it writes the answer token by token.

Prefix caching retains the KV cache for a prompt’s opening text. A later request with exactly the same opening can reuse that work. Lyceum and the first-party vLLM automatic-prefix-caching documentation state the same boundary: prefix caching shortens prefill; it does not speed up decode.

That means a coding agent with a stable system prompt, tool definitions, and conversation history has a different optimization problem from a chatbot whose short prompts produce long answers. The former may benefit substantially from reused prefixes. The latter may see little benefit because decode dominates.

Use this measurement before benchmarking:

Repeated prompt share = repeated opening tokens divided by total input tokens.

Measure the distribution across real requests, not one hand-picked prompt. Record exact-prefix matches separately from merely similar text; Lyceum’s description requires the same opening text. Then keep the recorded traffic mix fixed while comparing engines.

A reproducible vLLM versus TensorRT-LLM benchmark

1. Pin and publish the versions

Record the exact vLLM and TensorRT-LLM versions, model revision, quantization, GPU type, and driver stack. The TensorRT-LLM build distinction is version-sensitive: Lyceum says stable 1.2.1 retains the engine build and the 1.3 line removes it after 1.3.0rc20. A comparison that omits these details cannot be reproduced.

2. Freeze one workload dataset

Use the same model, weights, quantization, prompts, output lengths, and request ordering for both engines. Preserve the real distribution of prompt lengths and repeated openings. Synthetic fixed-shape prompts are useful for controlled tests, but they do not replace the production traffic sample.

3. Change one variable at a time

Start with the same concurrency and batching policy. Then vary batch size or concurrency while holding the workload constant. Measure the latency and throughput curve instead of selecting one flattering point.

Run a separate prefix-caching experiment: one set with repeated openings and one without them. Keep decode lengths comparable so a prefill optimization is not mistaken for faster token generation.

4. Report latency and throughput together

Use this reporting template. It is a teaching template, not published benchmark data.

FieldWhat to report
ConfigurationEngine version, model revision, quantization, GPU, parallelism
WorkloadInput and output length distributions, concurrency, repeated prompt share
First-token latencyTTFT p50, p95, and p99
Generation latencyInter-token latency p50 and p95
ThroughputOutput tokens per second and completed requests per second
EfficiencyGPU utilization and peak memory, using one consistent measurement method
QualityTask-specific correctness checks for every quantization and engine

Mark missing cells not measured. Never fill a gap with a number from another GPU, model, engine version, or traffic mix.

5. Repeat after a release or model change

Lyceum’s selection advice explicitly favors vLLM when models change often. Re-running the same dataset makes upgrades comparable and reveals whether the operational cost of retuning is justified by the measured result.

LLM system design interview answer

Use this framework when asked to design an LLM inference service. It is an interview framework, not a claim about any company’s interview process.

Requirements first

State the model and size, traffic shape, prompt and output distributions, repeated prompt share, concurrency, latency targets, availability target, hardware constraints, and cost ceiling. Separate TTFT from inter-token latency; a user can experience them very differently.

Architecture second

A simple baseline is a client, an authenticated gateway with rate limiting and routing, an inference pool, a versioned model store, and metrics for latency, throughput, queue depth, and GPU utilization. Explain where prefix caching fits and why exact opening-text matches matter.

Engine choice third

Use the decision table. For frequent model changes, begin with vLLM and cite Lyceum’s no-separate-build-step rationale. For one frozen model on NVIDIA, begin with TensorRT-LLM and test whether its tuning pays on your traffic. Mention the version-specific engine-build change rather than repeating a timeless setup stereotype.

Scaling and reliability fourth

Scale replicas from measured utilization and queue depth. Keep the model revision and engine version pinned so rollback is a configuration change. Probe the full request path and alert on TTFT, inter-token latency, failures, and memory pressure. Re-run correctness and load tests after upgrades.

Cost last

Estimate cost from measured throughput and the actual GPU hourly rate, then include engineering and operating effort. Do not claim a universal cost-per-token winner without identical hardware, model, workload, and measurement boundaries. If prompt prefixes repeat, show the prefill and decode assumptions separately.

Claims to avoid in an interview or design document

  • Do not claim TensorRT-LLM is always faster. Lyceum says no engine wins every benchmark; model, GPU, version, and traffic move the result.
  • Do not claim vLLM is always easier. Northflank’s setup comparison is dated September 2025, and Lyceum documents a TensorRT-LLM build change in the 1.3 line.
  • Do not claim prefix caching makes every request faster. Lyceum’s cited boundary is prefill only, not decode, and reuse requires matching opening text.
  • Do not claim a public benchmark transfers to our workload. Transfer the method, not the number. Re-run your own model and traffic.
  • Do not equate higher throughput with a better user experience. Evaluate latency and throughput together at every batch-size and concurrency setting.

FAQ

Is vLLM or TensorRT-LLM faster?

Neither source set supports a universal winner. Lyceum says results move with model, GPU, engine version, and traffic. Compare both on your own recorded workload and report latency alongside throughput at every batch-size and concurrency setting.

Does TensorRT-LLM still require an engine build?

According to Lyceum, stable 1.2.1 does. The 1.3 line removes the build after release candidate 1.3.0rc20 of 30 June 2026. Pin and check the exact version you intend to run.

When is vLLM the better first test?

Lyceum points teams that change models often to vLLM because it serves without a separate build step. Northflank also highlights its Hugging Face ecosystem and easier open-source integration.

When is TensorRT-LLM the better first test?

Lyceum points one unchanged model running on NVIDIA GPUs to TensorRT-LLM. Northflank’s comparison also places it within the NVIDIA stack and lists FP8 and INT4 support among its strengths.

Does prefix caching accelerate decode?

No. Lyceum cites the vLLM documentation’s distinction: prefix caching shortens prefill and does not speed up decode. Measure repeated opening tokens and output lengths separately.

What is the smallest useful benchmark?

One fixed model and quantization, one recorded traffic sample, both engine versions pinned, and a concurrency sweep reporting TTFT, inter-token latency, throughput, correctness, and GPU utilization. Repeat with and without repeated prompt openings.

Further reading

Sources

https://northflank.com/blog/vllm-vs-tensorrt-llm-and-how-to-run-them

https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html

https://lyceum.technology/magazine/vllm-vs-tensorrt-llm-production-benchmark/


Interviewing for AI engineer roles? See the 0→1 AI Engineer Interview Playbook (2026 Edition) on Kindle.

The playbook

The 0→1 AI Engineer Interview Playbook

The interview loop that didn't exist two years ago, in one book: what AI engineer interviewers evaluate, the skill stack they probe, and LLM system design end to end.

  • The AI engineer interview taxonomy and how it differs from SWE and MLE
  • RAG, agents, fine-tuning and evals: what you must be able to defend
  • LLM system design walkthroughs for AI-native products
    Share:
    Back to Blog

    Related Posts

    View All Posts »