Putting llama.cpp, vLLM and SGLang on one leaderboard sounds reasonable, and it is mostly a mistake. All three load an open model and answer a prompt. But they were built to solve different problems.

llama.cpp is built to run a model almost anywhere. vLLM and SGLang are built to serve many people at once on GPUs. Between those two, the real difference is narrower than most comparisons suggest, and it mostly comes down to what your prompts look like.

This post sorts the three by the job each was designed for. It explains the handful of ideas that actually separate them, and it ends with a way to decide. It is written for a team that wants to run models on its own infrastructure, usually because the data cannot leave.

One caveat before starting. These projects ship every few weeks, so every version and number here is a snapshot from October 2026.

What an inference engine actually does

A trained model is just a large file of weights. An inference engine is the software that turns that file into answers.

It loads the weights into memory, turns your prompt into tokens, generates new tokens one at a time, and decides how to share the hardware between many requests at once. A good engine gets more answers per GPU, a faster first word, and less wasted memory.

Every request goes through two phases:

  • Prefill reads the whole prompt at once. It is compute-heavy, and it decides TTFT (time to first token), which is how long you wait before anything appears.
  • Decode generates the answer one token at a time. It is limited by memory bandwidth, and it decides ITL (inter-token latency), which is how fast the text appears after it starts.

Almost every optimization below is about keeping the GPU busy during decode without making prefill stall everyone else.

The KV cache, and why it is the whole game

While generating, the model needs the attention keys and values for every earlier token. Recomputing them for each new token would waste a lot of work, so engines store them. That store is the KV cache.

It grows with the length of every conversation, and it is multiplied by the number of users you serve at once. At high concurrency, the KV cache and not the model weights is usually what fills the GPU.

That is why most of the ideas in this post are really answers to one question: how do you manage the KV cache without wasting it?

Go deeper: Hugging Face’s caching explainer covers why and how KV caching works from first principles, and their continuous batching post builds on it.

Three engines, three jobs

llama.cppvLLMSGLang
CategoryLocal-first, run-anywhere runtimeGeneral-purpose GPU serving engineGPU serving engine tuned for shared context and structure
Problem it solvesRun models on almost any hardware with little memoryServe many users per GPU without wasting KV memoryReuse shared context across calls, produce structured output fast
Signature ideaGGUF files plus aggressive quantization, CPU/GPU hybridPagedAttention plus continuous batchingRadixAttention plus fast constrained decoding
Written inC/C++, no required dependenciesPython with CUDA/C++ kernelsPython with CUDA/C++ kernels
Typical userA laptop, a desktop app, an edge box, a small internal toolA platform team running an APITeams running agents, RAG, multi-turn chat, very large models
LicenseMITApache-2.0Apache-2.0
OriginGeorgi Gerganov, 2023UC Berkeley, 2023LMSYS (Berkeley, Stanford and others), January 2024

As of October 17, 2026, the latest tagged releases are vLLM v0.31.0 (Oct 5, 2026) and SGLang v0.5.21 (Oct 2, 2026); llama.cpp ships build tags several times a day, so no single “latest version” is meaningful for it. Check each project’s releases page before you plan around a specific version.

All three are actively maintained and commercially backed. The founding team behind ggml and llama.cpp joined Hugging Face in February 2026, with Georgi Gerganov’s team keeping full technical control and the projects staying fully open-source. vLLM’s core team has reportedly started a company, Inferact, and SGLang’s creators a company called RadixArk, each aiming to commercialize their respective open-source project. The licenses are permissive today, but company priorities can shift. That is a good reason to keep an OpenAI-compatible API between your apps and whichever engine you run, so switching engines stays a configuration change.

The ideas that actually separate them

Continuous batching

The old way to batch was to collect a group of requests, run them together, and wait until the slowest one finished. Short answers sat idle behind long ones.

Continuous batching lets new requests join the running batch at every step and lets finished ones leave immediately. The GPU stays busy. This is the main reason GPU engines scale with users in a way a simple local runtime does not. vLLM and SGLang are built around it. llama.cpp’s server supports it too, but across a fixed number of “slots” you set yourself.

Go deeper: Anyscale’s continuous batching write-up and the Hugging Face post.

PagedAttention (vLLM)

Older systems reserved one large contiguous block of memory per request, sized for the longest possible answer. Most of it went unused.

PagedAttention borrows an idea from operating systems. It splits the KV cache into small fixed-size blocks that do not have to sit next to each other, and it allocates them as needed. The result is almost no wasted KV memory, and blocks can be shared between requests. The vLLM paper reports 2 to 4 times higher throughput than the systems it compared against, at the same latency.

Go deeper: Efficient Memory Management for LLM Serving with PagedAttention (arXiv).

Prefix caching and RadixAttention

Many real workloads send the same start over and over: the same system prompt, the same policy document, the same conversation history plus one new message. Recomputing that shared start every time is waste.

Prefix caching reuses the KV cache for a prompt start the engine has seen before. vLLM does this automatically by hashing each KV block together with the tokens before it. It also supports an optional per-request “cache salt” so tenants in a shared setup cannot reuse each other’s cache.

RadixAttention is SGLang’s version. It keeps the cached sequences in a tree, so it can find and share overlapping prefixes across many calls, and it evicts old branches with a least-recently-used policy. In a multi-turn chat, only the newest message needs fresh computation.

The honest summary is that both engines can do prefix caching. SGLang was designed around it, so the gap is largest when prompts share long, varied prefixes. If your prompts barely overlap, expect little difference.

Go deeper: LMSYS on RadixAttention and SGLang, and the vLLM Automatic Prefix Caching design doc.

Chunked prefill

A long prompt in prefill can monopolize the GPU and freeze everyone else’s typing speed. Chunked prefill splits that long prompt into pieces and mixes them in with ongoing decode steps, so existing users keep getting tokens smoothly.

The tradeoff is a small increase in time to first token for the long prompt, in exchange for steadier latency for everyone else. The idea comes from the Sarathi-Serve work, and modern engines use it.

Go deeper: Sarathi-Serve (arXiv).

Speculative decoding

A small, cheap “draft” model guesses several tokens ahead. The large model then checks all of those guesses in one pass and keeps the ones it agrees with. The output is the same as the large model would have produced, only faster for each user.

Go deeper: Fast Inference from Transformers via Speculative Decoding (arXiv).

Structured output

For extraction work, you want valid JSON every time, not usually valid JSON. Constrained decoding does this by blocking, at each step, any token that would break the schema or grammar.

Both vLLM and SGLang support it, and both can use the XGrammar engine. llama.cpp supports JSON schemas and GBNF grammars natively.

Go deeper: XGrammar (arXiv).

Quantization and file formats

Quantization stores weights in fewer bits, which cuts memory and often speeds things up, at a small quality cost. I covered the mechanics in What Actually Happens When You Quantize a Language Model.

What matters for this comparison is which format lives where:

FormatHome engineBest for
GGUF (roughly 2 to 8 bit)llama.cppCPU, Apple Silicon, consumer GPUs, models bigger than VRAM
AWQ and GPTQ (4-bit, weights only)vLLM, SGLangFitting a bigger model on fewer GPUs
FP8 and newer FP4 variantsvLLM, SGLangRecent data-center GPUs with a small quality cost

A practical rule: GGUF for llama.cpp, FP8 or AWQ for the GPU engines. vLLM can load GGUF files, but its own documentation calls that path experimental, so do not plan production around it.

Go deeper: AWQ (arXiv), vLLM’s quantization docs, and llama.cpp’s own quantize tool README.

Splitting a model across GPUs

Three terms come up constantly. Tensor parallelism splits each layer’s math across GPUs, which lets bigger models fit and lowers latency, but it needs fast links between GPUs. Pipeline parallelism puts different layers on different GPUs or machines. Expert parallelism spreads the experts of a mixture-of-experts model across GPUs.

vLLM documents all three individually (tensor and pipeline parallelism, expert parallelism), plus separating prefill from decode onto different machines. SGLang supports all three too, though its pipeline and expert parallelism each have a dedicated docs page while tensor parallelism is only documented through its --tp-size server argument rather than a standalone guide. llama.cpp can split across GPUs and across hosts through an RPC backend, which is simpler and, as the security section below shows, needs care.

Hardware: where each one can run

Hardwarellama.cppvLLMSGLang
x86 and ARM CPUExcellentSupported, not its strengthLimited
Apple SiliconFirst-class (Metal)New plugin, earlyNot a target
NVIDIA GPUYesBest-in-classBest-in-class
AMD GPUYes (ROCm, Vulkan)Yes (ROCm)Yes (ROCm)
Intel GPU and other acceleratorsYes (SYCL, Vulkan and others)Widest list, mostly through pluginsSmaller list, growing
Model bigger than VRAM (CPU plus GPU)Signature featureLimitedLimited
Multi-nodeRPC backendMatureMature

The shortcut: if the machine has no NVIDIA or AMD data-center GPU, llama.cpp is almost always the answer. On a GPU cluster, vLLM and SGLang are both first-class, and vLLM covers the most accelerator types.

Benchmarks, honestly

Public benchmarks for these engines disagree, and the reasons are boring. Results swing with model size, dense versus mixture-of-experts, GPU, software version and how much your prompts overlap. A single “X% faster” headline is a hint, not a fact.

Three sources are worth reading with their conditions attached.

vLLM vs SGLang, out of the box. Modal’s LLM Engineer’s Almanac benchmarked Llama 3.1 8B on one H100, default settings, official containers. It measured the maximum requests per second while keeping the 95th-percentile time to first token under one second. The prompts were random pieces of a novel, so there was almost no shared prefix, which understates SGLang’s main advantage.

WorkloadvLLM (req/s)SGLang (req/s)
Input 1024 / Output 128 (RAG, summarization)17.0816.88
Input 128 / Output 1024 (code generation)5.346.50
Input 2048 / Output 2048 (long chat)1.231.25

Llama 3.1 8B, one H100, default settings, near-zero prefix sharing. Source: Modal’s LLM Engineer’s Almanac, retrieved via a Red Hat Developers article from Oct 14, 2026.

Roughly a tie. Neither engine wins every workload. Modal also noted that vLLM’s cold start was far longer (around five minutes for 8B models versus around one minute for SGLang), because vLLM turns graph compilation on by default. That matters if you scale to zero.

Dense versus mixture-of-experts. CCL-Bench (arXiv 2605.06544, A100s) held the communication backend and parallelism the same for both engines. On the mixture-of-experts model tested (DeepSeek-MoE-16B), SGLang exposed less communication time than vLLM. On the dense models tested, vLLM showed higher model FLOPs utilization than SGLang; see the paper for the exact figures.

ModelSGLang comm. % of step timevLLM comm. % of step time
DeepSeek-MoE-16B (mixture-of-experts)38.853.7

Lower is better. A100s, same communication backend and parallelism for both engines. Source: CCL-Bench, arXiv 2605.06544.

The winner flips with the model’s architecture.

Local runtime vs serving engine under load. A separate Red Hat benchmark from August 2025, not the October 2026 vLLM-vs-SGLang piece cited above, compared vLLM 0.9.1 with Ollama 0.9.2, which runs on llama.cpp, from one to 256 concurrent users on a single A100. vLLM is at v0.31 today, so treat this as a snapshot of two specific older versions, not a current comparison. Ollama defaults to 4 parallel requests and was also retuned to 32, the highest stable value the authors found, and still trailed at peak: about 793 tokens per second for vLLM against roughly 41 for Ollama. Worth knowing: the author is a Red Hat performance engineer who writes regularly about vLLM benchmarking, not an independent third party. Ollama is also a stand-in for llama.cpp here, not a direct test of llama-server with tuned parallel slots. Read the original piece rather than taking the headline number at face value.

Where llama.cpp competes. On a unified-memory AMD Strix Halo machine, llama.cpp won throughput on a large 4-bit mixture-of-experts model, while vLLM won time to first token. On non-data-center hardware, llama.cpp is very competitive.

Prefix-heavy workloads. The SGLang paper reports up to 6.4 times the throughput of other systems on agent, JSON, RAG and multi-turn workloads. That is the authors’ own result, so confirm it on your traffic before planning capacity around it.

Security: the part nobody writes about

All three engines have shipped critical, unauthenticated remote-code-execution advisories, and the real story is less about any single CVE and more about a recurring pattern: optional features, multimodal input handling, RPC backends, disaggregated serving, internal messaging over pickle, keep reopening the same class of hole. Details below are drawn from the GitHub Advisory Database rather than NVD. Each project’s own advisory list moves faster than this post can; check those directly for the current picture.

vLLM. CVE-2026-22778 (CVSS 9.8) is a two-stage chain: an invalid image sent to a multimodal endpoint triggers an error that leaks a heap address, chained with a heap overflow in video decoding to achieve code execution. It only affects deployments serving video models. The advisory also notes the exploit works even with an API key configured, naming POST /v1/invocations as an affected endpoint alongside /v1/chat/completions. Fixed in v0.14.1. See vLLM’s full advisory list for what’s shipped since.

SGLang. Two advisories, both CVSS 9.8, both unauthenticated pickle deserialization over an internal ZMQ message channel, fixed in 0.5.10:

CVEWhereAdvisory
CVE-2026-3059Multimodal generation pathGHSA-rgq9-fqf5-fv58
CVE-2026-3060Encoder-parallel disaggregation pathGHSA-jx93-g359-86wm

Neither was published through SGLang’s own GitHub security-advisories page, only ingested into the general Advisory Database from an external source. See SGLang’s full advisory list before treating this as a two-CVE problem. It isn’t.

llama.cpp. CVE-2026-34159 (CVSS 9.8) is a bounds-check failure in deserialize_tensor() in the RPC backend: an unauthenticated attacker with TCP access to the RPC port can read and write process memory. The project’s own advisory lists no official patched version.

What the RPC backend actually is, per the advisory and the project’s own docs:

  • Opt-in. It has to be explicitly compiled with -DGGML_RPC=ON.
  • Defaults to listening on localhost. Its intended use is distributed inference across machines, so someone has to deliberately bind it wider (the advisory calls out --host 0.0.0.0) to be exposed at all.
  • Discouraged regardless. The project’s own SECURITY.md is direct: never run it on an open network, and don’t use it at all unless your environment is fully trusted and isolated.

See llama.cpp’s full advisory list for what’s shipped since.

Notice where the risk sits: optional features such as multimodal fetchers, RPC, disaggregation and pickle-based messaging. A plain text-only deployment has a smaller attack surface.

A hardening checklist that applies to all three:

  • Private subnets only, and no public IPs on the engine.
  • An API gateway in front, handling authentication, rate limits, request size limits and audit logs. Treat the engine itself as unauthenticated unless you have configured otherwise.
  • Block outbound traffic so the engine cannot fetch URLs.
  • Pin container versions, subscribe to the project’s security advisories, and patch on a schedule.
  • Turn off features you do not use.
  • Load only model weights you trust, and keep trust_remote_code off.
  • In shared deployments, isolate prefix caches per tenant.
  • Check each engine’s usage-statistics settings and disable them if your policy requires it.

Cost and operations for a small team

The biggest cost is GPUs sitting idle, so the best engine is the one that lets you use fewer of them at your real load.

A rough memory formula: VRAM is about the model weights plus the KV cache per request times the number of concurrent requests. FP8 roughly halves weight memory compared with BF16, and 4-bit roughly quarters it. With long documents, the KV cache dominates, which is where prefix caching and KV quantization help.

For a small team:

  • One engine in production beats two. Default to the one with the broadest tooling.
  • Use the official container images rather than building from source.
  • Keep one OpenAI-compatible endpoint so applications do not care which engine sits behind it.
  • Budget for GPU nodes, a gateway, metrics and dashboards, and a monthly patching window.
  • Plan for cold starts. Either keep one warm replica or invest in snapshot and sleep-mode tuning.
  • Check the license of the model separately from the engine. Some model licenses add conditions.

Self-hosting keeps prompts inside your network, but it does not make privacy automatic. You still own request logging (prompts often contain personal data), shared caches across tenants, model provenance and outbound network access.

How to decide

Start with workload shape, not the leaderboard.

Data-center GPUs available?
 ├─ No  → CPU-only, Apple Silicon, or edge?  → llama.cpp
 └─ Yes → More than ~5-10 requests at once?
           ├─ No  → POC or single user → llama.cpp (or your production engine)
           └─ Yes → Do most requests share long prefixes?
                     ├─ Yes → Benchmark SGLang vs vLLM on your own traces
                     │        (tie → vLLM for ecosystem)
                     └─ No  → vLLM
Always: private network + auth gateway + pinned, patched versions
ScenarioLikely choiceWhy
Laptop, proof of concept, demollama.cppOne binary, GGUF, no ops
Production API for internal appsvLLMBroad model and hardware support, mature tooling, large community
Agents or RAG with heavy prefix sharingPilot SGLang against vLLMRadixAttention reuses shared context, faster cold start
CPU-only server or edge devicellama.cppBest CPU kernels and quantization, tiny footprint
Structured extraction at volumevLLM by default, SGLang if it wins on your dataBoth have fast constrained decoding, and batch jobs are close on throughput
Very large mixture-of-experts across many GPUsvLLM or SGLangBoth invest heavily in expert parallelism
Small unified-memory machine, few usersllama.cppFirst-class Metal, competitive on big quantized models

Test on your own traffic

Public numbers age fast and disagree, so measure.

  1. Capture real traces: prompt lengths, output lengths, how much prefixes overlap, and how many requests run at once.
  2. Set your targets for time to first token and inter-token latency before you test.
  3. Replay the same traces against each candidate engine on the same hardware, with prefix caching on for both.
  4. Compare cost per million tokens at your target latency, not peak throughput.
  5. Re-run it after major version bumps, because these engines change quickly.

If it is a tie, pick the one with the better tooling for your team.

Quick Recall

llama.cpp  = run anywhere. Local, CPU, Mac, edge, low concurrency. GGUF.
vLLM       = serve at scale, safely. Broadest models/hardware/tooling.
SGLang     = serve at scale, reuse context. Prefix-heavy, structured, big MoE.

Dividing lines
  concurrency     -> llama.cpp vs the GPU engines
  prefix overlap  -> vLLM vs SGLang (and only if you measure it)

Ideas to remember
  KV cache            memory of tokens already read; the real bottleneck
  continuous batching requests join and leave every step
  PagedAttention      KV cache in small pages, almost no waste
  RadixAttention      prefix tree, automatic reuse across calls
  chunked prefill     long prompts stop freezing everyone else
  quantization        GGUF for llama.cpp, FP8/AWQ for GPU engines

Always: private network, auth gateway, pinned and patched versions.
Benchmark your own traffic. Keep an OpenAI-compatible boundary.

Sources and further reading

Related

What Actually Happens When You Quantize a Language Model How shrinking a model's numeric precision cuts memory and speeds up inference, and exactly what you trade away to get there. Where AI Actually Speeds Up Engineering Work (And Where It Doesn't) A plain, honest look at this by task type, not another take on whether AI will replace engineers. Reviewing AI-Generated Code Like You Mean It The review habits that need to change when more of your codebase is written by AI.

Enjoyed this? Subscribe via RSS.