Ollama OpenAI-Compatible API: How to Use It with Existing SDKs

Ollama ships an OpenAI-compatible API, so most applications built around the OpenAI SDK can be pointed at a local or cloud Ollama server by changing the base URL and supplying any non-empty API key. Ollama’s documentation describes support for the /v1/chat/completions endpoint and stateless /v1/responses requests, which is enough for a large share of existing chat clients, scripts, and frameworks.

This article explains how the compatibility layer works, shows working examples for Python, JavaScript, and cURL, covers the difference between the local server and Ollama’s cloud endpoint, and documents the limitations you should know before migrating an existing OpenAI integration. It assumes you are comfortable running commands and editing a few lines of code, but it does not assume prior Ollama experience.

What Ollama’s OpenAI compatibility actually provides

Ollama added built-in compatibility with the OpenAI Chat Completions API on February 8, 2024, according to the Ollama blog. The stated goal was to let existing tooling and applications use Ollama locally without custom adapters. In practice, that means you keep your OpenAI client library and change two connection settings: the base URL and the API key.

The compatibility is a subset of the OpenAI API rather than a full clone. Ollama’s official documentation lists /v1/chat/completions as the primary supported endpoint, with stateless /v1/responses requests available through the cloud API. Features that depend on server-side conversation state, built-in web search through /v1/responses, and custom or freeform tool-call replay are explicitly not supported by the cloud API.

Prerequisites

• Ollama installed and running, for local use.
• At least one model pulled, for example ollama pull llama2 or a newer model such as gpt-oss:20b.
• An OpenAI-compatible client library, such as pip install openai for Python or the openai npm package for JavaScript.
• For cloud use: an OLLAMA_API_KEY environment variable and a signed-in Ollama account. Ollama notes that using cloud models through the local server also requires signing in to Ollama.

Pointing a local OpenAI SDK at Ollama

Python

Install the client and create it with Ollama’s local base URL. The client requires an API key value, but Ollama ignores it.

from openai import OpenAI

client = OpenAI(
    base_url='http://localhost:11434/v1/',
    api_key='ollama',  # required but ignored
)

chat_completion = client.chat.completions.create(
    messages=[
        {'role': 'user', 'content': 'Say this is a test'},
    ],
    model='gpt-oss:20b',
)

print(chat_completion.choices[0].message.content)

Expected result: the model’s reply is printed to your terminal. If you see a connection error instead, confirm the Ollama server is running and listening on port 11434.

JavaScript

import OpenAI from 'openai';

const openai = new OpenAI({
  baseURL: 'http://localhost:11434/v1/',
  apiKey: 'ollama', // required but ignored
});

const chatCompletion = await openai.chat.completions.create({
  messages: [{ role: 'user', content: 'Say this is a test' }],
  model: 'gpt-oss:20b',
});

console.log(chatCompletion.choices[0].message.content);

cURL

A single cURL request is the fastest way to confirm the endpoint works before touching any SDK.

curl -X POST http://localhost:11434/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{
    "model": "gpt-oss:20b",
    "messages": [{"role": "user", "content": "Say this is a test"}]
  }'

Because the request format matches OpenAI’s, you can reuse existing payloads, including multi-turn conversations with system, user, and assistant roles. The Ollama blog example shows exactly that pattern, sending a system message, a user question, a prior assistant answer, and a follow-up user question in the same messages array.

Using Ollama’s cloud endpoint

Ollama also exposes a hosted endpoint at https://ollama.com/v1. The same OpenAI client works, but you must set a real API key in OLLAMA_API_KEY, and no local Ollama installation is required.

import os
from openai import OpenAI

client = OpenAI(
    base_url='https://ollama.com/v1',
    api_key=os.environ['OLLAMA_API_KEY'],
)

response = client.chat.completions.create(
    model='gemma4:31b',
    messages=[{'role': 'user', 'content': 'Say hello in one sentence.'}],
)

print(response.choices[0].message.content)

The cloud client also supports stateless Responses requests, so you can use the newer responses.create surface:

response = client.responses.create(
    model='gemma4:31b',
    input='Say hello in one sentence.',
)

print(response.output_text)

Note that the cloud API does not support stateful Responses, built-in web search through /v1/responses, or custom and freeform tool-call replay. If your workflow depends on those features, you will need to keep using OpenAI’s own endpoint or restructure the request so the state lives in your application.

Switching between local and cloud without rewriting code

The practical pattern is to keep one client and drive the base URL and key from configuration. Locally, any non-empty string works as the key; in the cloud, the key must be valid. Because the request shapes are the same, a single code path can serve both, provided you avoid the unsupported cloud features listed above.

Using Ollama behind existing frameworks

The OpenAI compatibility layer is not limited to the official SDKs. The Ollama blog demonstrates the Vercel AI SDK, an open-source library for conversational streaming applications, by cloning the next-openai example and editing the route handler to point at Ollama’s endpoint. Any framework that lets you override the OpenAI base URL should work the same way, though you should verify each framework’s own feature expectations, since some assume OpenAI-only capabilities such as stateful responses or built-in search.

Streaming and multi-turn conversations

The compatibility layer follows the OpenAI request shape, so multi-turn patterns carry over directly. A multi-turn request simply sends the full message history each time, with alternating user and assistant roles and an optional system message at the start. Ollama’s blog example does exactly this with Llama 2, asking about the 2020 World Series and then following up with a question about where it was played, while including the prior assistant answer in the payload.

Streaming behavior depends on the client library and the model, so verify it with a small test before relying on it in production. If streaming fails while non-streaming works, the practical fallback is to disable streaming in the client while you investigate.

Choosing between local and cloud

Local Ollama keeps prompts and responses on your own machine, which matters for private data, and it does not require an API key beyond the placeholder value. The tradeoff is hardware: local model speed and quality depend on your GPU or CPU and the model size you can run. The cloud endpoint removes the hardware constraint and adds access to larger models such as gemma4:31b, but it requires an API key and is subject to the feature limits described above. Neither path is universally better; the right choice depends on privacy requirements, hardware, and which API features your application actually uses.

Common errors and how to diagnose them

Connection refused or timeout

Symptom: the client cannot reach the server. Likely cause: Ollama is not running, or it is listening on a different port. Run the cURL example against http://localhost:11434/v1/chat/completions to isolate the problem. If cURL also fails, the issue is the server, not the SDK.

Authentication errors against the local server

Because Ollama ignores the key locally, an auth error against localhost usually means the client is actually pointed at the cloud base URL. Confirm the base URL ends in /v1/ and uses localhost for local work.

Model not found

If the model name in your request has not been pulled locally, the request fails. Run ollama pull <model> first, then repeat the request with the exact same name. Match the name against your local model list, since small spelling differences are a common cause of this error.

Unsupported feature errors in the cloud

If a Responses request fails with an unsupported-feature message, you have probably used a stateful feature or a built-in tool. Re-issue the request as a stateless call and keep conversation history in your own application.

Verification checklist

1. Confirm the server responds: run the cURL request and check for a JSON completion.
2. Confirm the SDK path: run the Python or JavaScript snippet and print the message content.
3. Confirm the model: list local models and match the name exactly.
4. Confirm the environment: for cloud, verify OLLAMA_API_KEY is set and exported in the same shell that runs your code.

What to do next

If your goal is a drop-in local replacement for an OpenAI chat integration, start with the cURL check, then move to your SDK of choice. If your application relies on stateful Responses, built-in web search, or freeform tool-call replay, plan around those gaps rather than assuming full parity. Readers are welcome to share their environment, the exact command they ran, and the error text in the community if a request behaves differently from the documentation.