The practical answer is that all three projects let you keep your existing OpenAI client code and change only the base_url and the model name. The differences show up in which endpoints exist, which model file formats they load, how they handle concurrent requests, and how much of the surrounding server is actually protected by an API key. vLLM is the most explicit about its OpenAI surface in the documentation reviewed here; Ollama is the fastest to start but has a path quirk that breaks naive configurations; llama.cpp is the most portable option for GGUF weights.
This article is for a technically curious beginner or intermediate practitioner who already has a model running locally and now wants to wire it into an application, gateway, or automation script. It compares documented capabilities rather than benchmark scores, and it flags where the supplied sources do not cover a claim.
What “OpenAI-compatible” does and does not promise
An OpenAI-compatible server is an HTTP service that mimics the request and response shapes of OpenAI’s API closely enough that official and third-party SDKs work against it unchanged. In practice, that means a client library is configured with a base URL such as http://host:port/v1 and a model identifier, and the library then calls endpoints like /v1/chat/completions.
Compatibility is not all-or-nothing. The vLLM documentation lists its supported OpenAI APIs as Completions (/v1/completions), Chat Completions (/v1/chat/completions), a Chat Completions batch API (/v1/chat/completions/batch), the Responses API (/v1/responses and related routes), Embeddings (/v1/embeddings), Transcriptions (/v1/audio/transcriptions), and a Translation API. It also documents specific gaps: the suffix parameter is not supported on the Completions API, the user parameter is ignored on Chat Completions, and the parallel_tool_calls parameter controls whether more than one tool call may be returned in a single response. Those are exactly the kind of details that decide whether a migration is a one-line change or a rewrite.
Choosing by need, not by popularity
Before comparing servers, decide what you are optimizing for. The criteria that matter most in local deployments are:
- Model format. Whether your weights are GGUF files, Hugging Face-style repositories, or something else determines which servers can load them at all.
- Hardware. CPU-only, a single consumer GPU, and a multi-GPU box lead to very different conclusions.
- Concurrency. A personal assistant serving one request at a time is a different problem from an internal service handling many parallel calls.
- API surface. If you need embeddings, transcription, or the Responses API, check that the server implements them before you commit.
- Operational burden. Installation, upgrade, and observability cost matter as much as raw capability.
vLLM: a throughput-oriented OpenAI server
vLLM is described in its documentation as providing an HTTP server that implements OpenAI’s Completions API, Chat API, and more, so that models can be served and interacted with using an HTTP client. The canonical startup form shown in the GateLLM documentation is:
vllm serve Qwen/Qwen3-8B --port 8000
# OpenAI-compatible endpoint at http://localhost:8000/v1For a gateway or client, the upstream base URL is therefore http://localhost:8000/v1. Because vLLM is natively OpenAI-compatible, the same SDK code that talks to a hosted provider can talk to it, provided your model identifier matches.
The authentication caveat that catches people out
vLLM’s own documentation carries a blunt warning: the --api-key option, or the VLLM_API_KEY environment variable, only authenticates requests to endpoints under the /v1, /v2, and /inference path prefixes. Other endpoints on the same HTTP server are not authenticated — most notably /invocations, which the documentation says exposes the same inference capabilities as the /v1 endpoints. The documentation explicitly states: “Do not rely on --api-key alone to secure vLLM,” and recommends hardening such as deploying behind a reverse proxy.
This is a deployment decision, not a bug report. If the server is bound only to a trusted interface, the exposure is limited. If it is reachable from a shared network, the API key is not a complete access control boundary.
Ollama: the shortest path, with one path gotcha
Ollama is the least ceremonious way to get a local model answering requests. The documented startup and model pull are one line each:
ollama serve # default: http://localhost:11434
ollama pull qwen3:8bThe critical detail, called out as a gotcha in the GateLLM documentation, is that Ollama’s native /api/chat endpoint is not OpenAI-compatible. When configuring an OpenAI-compatible client or gateway upstream, the base URL must include the version prefix: http://localhost:11434/v1, not http://localhost:11434. Pointing a client at the bare host and port is one of the most common causes of a 404 when a first request is sent.
A minimal working client, adapted from the documented example, looks like this:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
resp = client.chat.completions.create(
model="qwen3:8b",
messages=[{"role": "user", "content": "Introduce yourself in one sentence"}],
)
print(resp.choices[0].message.content)Note the model name. In a gateway setup, the model name configured in the gateway may differ from the upstream model ID that Ollama itself expects, such as qwen3:8b. Model-name mismatches produce errors that look like server faults but are really configuration faults.
llama.cpp: the portable GGUF server
The supplied sources list llama.cpp server alongside Ollama, vLLM, LM Studio, SGLang, and TGI as local or self-hosted inference services that are not client SDKs but upstreams: they run models and expose a, usually OpenAI-compatible, HTTP endpoint, which a gateway or client then targets. The same documentation notes that these services can be configured as an openai protocol upstream in a gateway, with the client side remaining unchanged.
Beyond that classification, the retrieved material does not document llama.cpp’s specific flags, default ports, or endpoint coverage, so this article does not assert them. What can be said safely is that llama.cpp is generally chosen when you want to run quantized GGUF weights, when you want CPU inference or partial GPU offload, and when you want a single self-contained binary rather than a Python service stack. If you are evaluating it, verify its OpenAI-compatible route prefix and default port against its own current documentation rather than assuming they match Ollama’s or vLLM’s.
Documented attributes side by side
The table below contains only attributes supported by the sources used here. It is not a performance comparison, and the absence of a value means the source did not state it — not that the feature is missing.
| Server | Documented default address | OpenAI-compatible base URL | Documented note |
|---|---|---|---|
| vLLM | http://localhost:8000 | http://localhost:8000/v1 | Natively OpenAI-compatible; broad documented API surface; --api-key protects only /v1, /v2, and /inference prefixes |
| Ollama | http://localhost:11434 | http://localhost:11434/v1 | Native /api/chat is not OpenAI-compatible; the /v1 suffix is required |
| llama.cpp | Not stated in the sources | Not stated in the sources | Listed as a self-hosted upstream that usually exposes an OpenAI-compatible HTTP endpoint |
A reproducible check before you build on top
Whatever server you choose, confirm the protocol before writing application code. Two requests are usually enough. First, list models:
curl http://localhost:8000/v1/modelsThen send a minimal chat completion and inspect the JSON shape:
curl http://localhost:8000/v1/chat/completions
-H "Content-Type: application/json"
-d '{"model": "", "messages": [{"role": "user", "content": "Say hello"}]}'If the first call returns a model list and the second returns a choices array with a message, the endpoint is behaving as an OpenAI-compatible chat server. This is a protocol check, not a quality or speed evaluation.
Troubleshooting the failures that actually occur
Most first-run problems fall into a small set of categories.
- 404 on
/v1/chat/completions. Either the base URL is missing its version prefix — the documented Ollama case — or the server does not implement that particular endpoint. Check the server’s own list of supported APIs. - Connection refused. The service is not running, is on a different port, or is bound only to loopback while the client is remote. Confirm the port and bind address.
- 401 or 403. An API key is expected, or the key is wrong. For vLLM, remember that authentication does not cover every endpoint on the server.
- Model not found. The identifier sent by the client does not match what the server or gateway expects. In gateway setups, the gateway model name and the upstream model ID are separate values.
- Unexpected parameter behavior. Some parameters are accepted but ignored, or rejected outright. vLLM documents
suffixas unsupported anduseras ignored on the relevant endpoints.
Security and exposure
An OpenAI-compatible endpoint is an inference endpoint. Treat it like any other service that consumes expensive resources. Bind to the interfaces you intend to serve, put a reverse proxy in front when the service is reachable beyond a trusted host, and do not assume a single API key flag secures every route. vLLM’s documentation is unusually direct about this limitation, and the same reasoning applies to any local server whose API-key support was designed for convenience rather than as a full authorization layer.
What cannot be settled without measurements
No benchmark data was supplied for this comparison, so no throughput, latency, or memory figures are reported here. Claims that one server is faster than another depend heavily on model, quantization, context length, batch size, GPU, and driver versions. If you need to decide on performance grounds, run the comparison yourself under controlled conditions: hold the model and quantization constant, fix the prompt set and sampling parameters, record time to first token and tokens per second at several concurrency levels, and repeat each measurement enough times to report a median and a spread rather than a single number. Until then, choose on format support, API surface, and operational fit.
A practical next step
Start with the server that matches your weights and your hardware, then verify the protocol with the two curl calls above before integrating anything. If your model is a GGUF file and you want minimal dependencies, llama.cpp is the natural candidate. If you want the shortest route to a working chat endpoint, Ollama with a /v1 base URL is hard to beat. If you need a broad OpenAI API surface and expect concurrent traffic, vLLM is the documented choice — provided you handle its authentication scope deliberately. Readers working through a first setup are invited to share their environment, the exact base URL used, and any error text in the community, since base-URL and model-name mismatches account for a large share of first-run failures.
FAQ
Can I use the official OpenAI SDK with Ollama?
Yes, if the base URL includes the version prefix. Ollama's native /api/chat endpoint is not OpenAI-compatible, so the client must target http://localhost:11434/v1 rather than the bare host and port.
Does vLLM's --api-key secure the whole server?
No. vLLM's documentation states that --api-key and VLLM_API_KEY only authenticate requests under the /v1, /v2, and /inference path prefixes. Other endpoints, notably /invocations, are not authenticated, and the documentation recommends hardening such as a reverse proxy.
Which of the three is fastest?
That cannot be answered from the sources used here, because no benchmark measurements were supplied. Speed depends on model, quantization, context length, concurrency, and hardware. Run a controlled comparison holding those variables constant and record time to first token and tokens per second.
Why does my request return 404 even though the server is running?
The most common cause is a base URL missing its version prefix, such as pointing at http://localhost:11434 instead of http://localhost:11434/v1. The second possibility is that the server does not implement that specific endpoint, so check its documented API list.