AIForum.

How to Run a Local Model as an OpenAI-Compatible API

On this page

THE SHORT ANSWER

Running a local model as an OpenAI-compatible API means placing a model behind an HTTP server that implements the same /v1 routes as OpenAI. Existing SDKs and clients can then point at a local base URL such as http://localhost:1234/v1. vLLM and mistral.rs are two documented options; each has different API coverage, authentication behavior, and operational tradeoffs.
Quick facts

Running a local model as an OpenAI-compatible API means starting an HTTP server that speaks the same request and response shapes as OpenAI’s API, then pointing your existing client at the local base URL. The model stays on your hardware, but the integration code can often stay almost unchanged.

This guide is for technically curious beginners and intermediate practitioners who want a practical path from model files to a working /v1/chat/completions endpoint. It uses two documented implementations as examples: vLLM, which exposes a broad set of OpenAI-compatible APIs, and mistral.rs, which provides a lightweight local server with a default port of 1234. Exact flags and supported features change between versions, so verify against the current documentation for your chosen runtime.

What "OpenAI-compatible" actually covers

An OpenAI-compatible server is not a single product. It is a server that implements some or all of OpenAI’s endpoint contract. vLLM’s OpenAI-compatible server documentation lists support for Completions (/v1/completions), Chat Completions (/v1/chat/completions), a Chat Completions batch API (/v1/chat/completions/batch), the Responses API (/v1/responses, /v1/responses/{response_id}, and cancel), Embeddings (/v1/embeddings), Transcriptions (/v1/audio/transcriptions), and Translation. Some endpoints are model-type specific: Completions and Chat Completions apply to text generation models, Embeddings to embedding models, and Transcriptions to ASR models. Chat Completions also requires a chat template for the model.

Compatibility is not identical to OpenAI. vLLM notes that the suffix parameter is not supported for Completions and the user parameter is ignored for Chat Completions. The parallel_tool_calls parameter is supported, but setting it to true does not guarantee more than one tool call; that depends on the model. mistral.rs places its OpenAI-compatible endpoints under /v1 and documents Chat Completions, Completions, and Responses as accepting an adapter parameter for loaded LoRA adapters.

Choose a runtime before you install

Start by matching the runtime to the job. vLLM is the broader API surface in the supplied documentation: it covers text generation, embeddings, audio transcription, translation, batch requests, and the Responses API. That makes it a reasonable candidate when you need more than chat completions or when you want to test OpenAI features beyond the basic chat loop. mistral.rs is presented as a local server that works with OpenAI SDKs and compatible clients using http://localhost:1234/v1 as the base URL. Its documentation emphasizes single-model and multi-model serving, plus dynamic LoRA adapters that appear in GET /v1/models.

Neither choice removes the need to check model licensing, hardware memory, context length, and runtime overhead. Model weights are only part of the memory picture; the KV cache and runtime overhead also consume memory, especially at long context lengths. The supplied sources do not provide memory tables or benchmark numbers, so treat any capacity estimate as a test you must run on your own hardware rather than a guarantee.

Prerequisites

  • A supported model that fits your hardware and license requirements.
  • A runtime installation: vLLM or mistral.rs, installed according to its current documentation.
  • Enough system memory or GPU memory for weights, KV cache, and runtime overhead.
  • A local port you can bind, such as 1234 for mistral.rs’s documented default.
  • An OpenAI-compatible client or curl for verification.

If you plan to expose the server beyond localhost, decide on authentication and network controls before the first request leaves your machine. That is not optional hardening; it changes how you start the server.

Start the server and confirm the base URL

The mistral.rs documentation gives a concrete startup command:

mistralrs serve -m Qwen/Qwen3-4B

After startup, the documented base URL is http://localhost:1234/v1. With a single -m model, the request model is "default" or can be omitted. In multi-model serving, use a model id exactly as it appears in GET /v1/models.

For vLLM, the server exposes the OpenAI-compatible routes described in its documentation. Start it with your chosen model and any authentication flags your deployment requires. If you use --api-key or the VLLM_API_KEY environment variable, remember the limitation described below: it authenticates only requests under /v1, /v2, and /inference path prefixes. Other endpoints on the same HTTP server are not authenticated, most notably /invocations.

Verify the model list

Before sending a chat request, call the models endpoint:

curl http://localhost:1234/v1/models

For mistral.rs, loaded dynamic LoRA adapters also appear in GET /v1/models, so vLLM-compatible clients can put the adapter alias in the model field. If you use a model name that is not in this list, expect a model-not-found error. The fix is to copy the exact id from the response or use "default" for a single-model mistral.rs setup.

Send a chat completion request

The mistral.rs documentation provides this curl example:

curl http://localhost:1234/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "default", "messages": [{"role": "user", "content": "Write a haiku about local inference."}], "max_tokens": 128}'

A successful response follows the OpenAI chat completion shape: choices contain a message with content. The exact text is model-dependent and not predetermined. If the request fails, check the base URL first: it must include /v1 when the client expects that prefix. Then check the model id, the JSON content type, and whether the server process is still running.

You can also use the OpenAI Python client. The mistral.rs documentation shows:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:1234/v1", api_key="not-used")
response = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Say hello from mistral.rs."}],
)
print(response.choices[0].message.content)

In that example, the api_key is required by the client but not validated by the server. That is a client-library convention, not a security feature. For vLLM, the --api-key option does perform authentication for the protected path prefixes, but it is not a complete security boundary for the whole server.

Use LoRA adapters when your runtime supports them

mistral.rs documents dynamic LoRA adapters as first-class model entries. After loading an adapter, it appears in GET /v1/models. You can select it in one of two ways: set model to the adapter alias, similar to vLLM-style usage, or keep the base model explicit and pass extra_body={"adapter": "code"}. The adapter field also accepts an exact generation object. Responses expose the resolved generation as adapter_generation for audit and retry routing. If you omit adapter and select the base model, the request runs without LoRA.

This matters for reproducibility. When an adapter is involved, record the base model, adapter alias, and resolved generation from the response. That lets you retry or compare results without guessing which weights served the request.

Authentication and exposure: the part most local setups get wrong

Local does not automatically mean private or safe. A server bound to 0.0.0.0 on a shared network is reachable by other machines unless a firewall or reverse proxy blocks it. vLLM’s documentation is explicit: the --api-key option or VLLM_API_KEY environment variable only authenticates requests to endpoints under /v1, /v2, and /inference path prefixes. Other endpoints on the same HTTP server are not authenticated, most notably /invocations, which exposes the same inference capabilities as the /v1 endpoints. The documentation warns not to rely on --api-key alone to secure vLLM and recommends hardening such as deploying behind a reverse proxy.

For any local OpenAI-compatible server, a practical baseline is: bind to localhost during development, use a reverse proxy with TLS and authentication for remote access, restrict network exposure with firewall rules, and log requests without logging secrets. If the server will be used by multiple people or automation, add rate limits and per-user credentials at the proxy layer. The exact proxy configuration is deployment-specific; the important part is that the model server itself is not treated as the only gate.

Troubleshooting common failures

The client gets a 404 or connection error

Check the base URL and port. mistral.rs documents http://localhost:1234/v1; a client that appends /chat/completions to a base URL without /v1 will hit the wrong path. Confirm the server is listening and that no other process owns the port.

The server returns model not found

Use the exact id from GET /v1/models. For a single-model mistral.rs server, "default" or an omitted model field is documented. In multi-model serving, aliases must match the model list.

Chat completions fail but completions work

vLLM documents Chat Completions as applicable only to text generation models with a chat template. If the model lacks a suitable chat template, the chat endpoint may not behave as expected. Test with the Completions endpoint or choose a model with a documented chat template.

A parameter is ignored or rejected

Compatibility is partial. vLLM does not support suffix for Completions and ignores user for Chat Completions. parallel_tool_calls is supported, but setting it to true does not guarantee parallel tool calls; that is model-dependent. Remove or adapt unsupported parameters before assuming the server is broken.

Authentication works for /v1 but another endpoint is open

This is the documented vLLM behavior, not a misconfiguration. If /invocations or another non-protected endpoint is reachable, put the server behind a reverse proxy or block that endpoint at the network layer.

Practical limits and when this approach fits

An OpenAI-compatible local server is a good fit when you want existing SDKs, scripts, or automation to talk to a model you control, or when you need to test prompts and workflows without sending data to a hosted API. It is less suitable when you need exact parity with every OpenAI feature, when you cannot operate the hardware, or when your security model requires centralized identity and audit controls that the model server does not provide.

The two documented options illustrate the tradeoff. vLLM offers a wider API surface, including Embeddings, Transcriptions, Translation, batch, and Responses endpoints, but its API key behavior requires additional hardening. mistral.rs offers a simple local base URL and LoRA adapter routing, with an API key that its documentation describes as required by the client but not validated by the server. Neither fact is a defect by itself; both are deployment constraints you must design around.

Next steps

Pick one runtime, start with a small model, and verify /v1/models before writing application code. Then send a chat completion with curl, connect your OpenAI client, and only after that add remote access, authentication, and logging. If you are comparing runtimes, test the same model, prompt, context length, and sampling settings on both, and record the exact versions. The supplied documentation does not provide benchmark numbers for either runtime, so publish only measurements you actually collect.

If you get stuck, share your operating system, runtime version, model id, base URL, the exact request, and the exact error in the Ai-Forum.org community. That level of detail is usually enough for another practitioner to reproduce the problem and suggest a targeted fix.

FAQ

Can I use the OpenAI Python SDK with a local model?

Yes. Point the client's base_url at the local /v1 endpoint. In the mistral.rs example, the client requires an api_key but the server does not validate it. For vLLM, --api-key authenticates only the protected path prefixes, not every endpoint.

Why does GET /v1/models matter?

It lists available model ids, and in mistral.rs it also lists loaded dynamic LoRA adapters. Requests must use an exact id from that list, or "default" for a single-model mistral.rs setup.

Is --api-key enough to secure vLLM?

No. vLLM documents that --api-key or VLLM_API_KEY authenticates only requests under /v1, /v2, and /inference. Other endpoints on the same server, most notably /invocations, are not authenticated. Use a reverse proxy and network controls.

Does vLLM support all OpenAI parameters?

No. vLLM notes that suffix is not supported for Completions, user is ignored for Chat Completions, and parallel_tool_calls behavior is model-dependent even when set to true.

References

  1. https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/
  2. https://docs.mistralrs.dev/guides/serve/openai-compatible-apis/

Related terms

OpenAI-compatible API
An HTTP API that follows OpenAI's request and response conventions closely enough for existing OpenAI SDKs and clients to work with minimal changes.
Base URL
The root address a client uses for API calls, such as http://localhost:1234/v1; clients typically append endpoint paths like /chat/completions.
Chat template
The model-specific formatting that turns a list of chat messages into the token sequence the model expects; vLLM notes Chat Completions requires a chat template.
LoRA adapter
A small set of additional weights that modifies a base model's behavior; mistral.rs exposes loaded adapters through GET /v1/models and an adapter parameter.
API key authentication limitation
In vLLM, --api-key or VLLM_API_KEY authenticates only /v1, /v2, and /inference paths; other endpoints such as /invocations remain unauthenticated.

Leave a Reply

Your email address will not be published. Required fields are marked *