Running Ollama in Docker gives you a repeatable local LLM service that other containers can call over a Docker network. The short version: start the official image with a named volume and port 11434, then connect other containers to the service name ollama on port 11434. Do not use localhost from inside another container.
This guide is for developers and local-AI practitioners who already have Docker installed and want a container-to-container setup rather than a host-only install. It covers CPU and GPU startup, Compose wiring, verification, and the most common connection failures.
Prerequisites and the network model
You need Docker Engine or Docker Desktop, enough RAM or VRAM for your chosen model, and network access to pull images. The official Ollama Docker image is ollama/ollama. The CPU-only command in the Ollama Docker documentation is:
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollamaThat command does three important things: it persists models in the named volume ollama, publishes container port 11434 to host port 11434, and names the container ollama. The published port is useful for host tools and browser-based clients; other containers should normally use a user-defined Docker network instead.
Docker Compose creates a default network for the services in a project. On that network, Docker’s embedded DNS lets one service reach another by its service name. So an app container can call http://ollama:11434, while the host can call http://localhost:11434.
Start Ollama with CPU only
For a first run, use the CPU-only command above. Then confirm the container is running:
docker ps --filter name=ollamaCheck logs if it exits immediately:
docker logs ollamaPull a model into the container. Replace llama3.2 with the model you want:
docker exec -it ollama ollama pull llama3.2From the host, query the API to verify the service responds:
curl http://localhost:11434/api/tagsA JSON response listing models indicates the API is reachable. If the command fails, check that the container is running and that host port 11434 is not already in use.
Enable GPU acceleration
CPU-only works, but inference is usually slower. The official docs cover NVIDIA, AMD, and JetPack paths.
NVIDIA GPUs
Install the NVIDIA Container Toolkit, configure Docker to use the NVIDIA runtime, and restart Docker. The Ollama docs give the repository and package commands for APT and Yum or DNF. After configuration, run:
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollamaThe --gpus=all flag is the key difference from the CPU command. If the container starts but does not use the GPU, verify the toolkit installation and Docker runtime configuration before changing the model or image.
AMD GPUs
For AMD GPUs, the docs specify the rocm tag and device access:
docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocmThis passes the kernel fusion driver and Direct Rendering Infrastructure devices into the container. Host driver compatibility matters; a working ROCm setup on the host is a prerequisite.
JetPack and Vulkan notes
On NVIDIA JetPack systems, Ollama cannot automatically discover the correct JetPack version. Pass JETSON_JETPACK=5 or JETSON_JETPACK=6 to select version 5 or 6. Vulkan support is bundled into the ollama/ollama image according to the same documentation. These are documented options, not universal fixes; use them when your hardware matches the documented case.
Connect another container to Ollama
The most common mistake is using localhost from inside a second container. Inside that container, localhost refers to itself, not to the Ollama container. Use the Ollama service name or container name on a shared Docker network.
With plain Docker, create a user-defined network and attach both containers:
docker network create ollama-net
docker run -d --network ollama-net -v ollama:/root/.ollama --name ollama ollama/ollama
docker run --rm --network ollama-net curlimages/curl:latest curl -s http://ollama:11434/api/tagsThe second container resolves ollama through Docker DNS and reaches port 11434. The -p 11434:11434 flag is not required for container-to-container traffic on the same network; it is only needed when the host or external clients must reach the API.
With Docker Compose, define both services in the same project. The following is an illustrative wiring pattern, not a tested deployment. Adapt image tags, volumes, and application paths to your environment:
services:
ollama:
image: ollama/ollama
volumes:
- ollama:/root/.ollama
ports:
- 11434:11434
app:
image: your-app-image
environment:
OLLAMA_BASE_URL: http://ollama:11434
depends_on:
- ollama
volumes:
ollama:In this pattern, the app container uses http://ollama:11434. The depends_on key controls startup order, but it does not wait for Ollama to be ready to serve requests. Your app should retry the first connection or include a health check.
The Docker RAG guide demonstrates the same three-component shape: an LLM served via Ollama, Qdrant as a vector database, and Streamlit as the web application. It clones a sample repository and containerizes the stack. That guide is a useful reference when you want to connect Ollama to a vector database and a UI rather than only test the API.
Verify the connection from the application container
After starting the stack, run a request from inside the app container. If the image includes curl:
docker compose exec app curl -s http://ollama:11434/api/tagsIf it does not, use a temporary container on the same Compose network:
docker compose run --rm app sh -c 'wget -qO- http://ollama:11434/api/tags'Successful output is a JSON model list. A successful TCP connection but an application error usually points to the model name, request format, or context settings rather than Docker networking.
Troubleshooting common failures
Connection refused
If the app reports Connection refused to ollama:11434, check that the Ollama container is running and attached to the same network. Confirm the port inside the container is 11434. The published host port does not matter for container-to-container traffic.
Name resolution fails
Errors such as Could not resolve host: ollama or no such host mean the app container is not on the same Docker network or the service name is wrong. In Compose, service names are the DNS names. In plain Docker, use --network on both containers.
Works on the host but not in a container
Host access uses http://localhost:11434 because of the published port. Container access uses http://ollama:11434 because of Docker DNS. Mixing these two addresses is the usual cause.
GPU not used
For NVIDIA, missing --gpus=all or an incomplete NVIDIA Container Toolkit setup will fall back to CPU. For AMD, missing the rocm tag or device flags will do the same. Check container logs for runtime or driver messages before assuming the model is the problem.
Port already allocated
If Docker reports that port 11434 is already in use, another Ollama instance or service is bound to it. Stop the conflicting process, change the host-side port, or omit the published port when only containers need access.
A concrete RAG wiring example
The Docker RAG guide uses Ollama, Qdrant, and Streamlit as separate components. In a Compose project, the Ollama service stays the same, Qdrant provides vector storage, and the application service receives two internal URLs. This is an illustrative extension of the documented pattern:
services:
ollama:
image: ollama/ollama
volumes:
- ollama:/root/.ollama
qdrant:
image: qdrant/qdrant
volumes:
- qdrant:/qdrant/storage
app:
image: your-rag-app
environment:
OLLAMA_BASE_URL: http://ollama:11434
QDRANT_URL: http://qdrant:6333
depends_on:
- ollama
- qdrant
volumes:
ollama:
qdrant:Again, this is a wiring pattern rather than a tested deployment. The important detail is that every service-to-service URL uses the Compose service name and the container port, not localhost and not the published host port.
Practical decisions and limits
Publishing 11434:11434 makes the API reachable from the host. If you only need container-to-container access, omit the port mapping and keep the service on an internal network. If you do publish it, bind to a specific interface or firewall the port rather than exposing it broadly.
Model storage should use a named volume or bind mount. Without it, pulling a model again after recreating the container wastes time and bandwidth. The official CPU and GPU commands both use -v ollama:/root/.ollama for this reason.
GPU support is not universal across Docker hosts. The official documentation covers NVIDIA Container Toolkit, AMD ROCm, and JetPack, but host driver versions and Docker runtime configuration still determine whether acceleration works. Treat GPU passthrough as a host-level prerequisite, not a container-only setting.
Next step
Start with the CPU command, pull one model, and verify http://localhost:11434/api/tags from the host. Then move the same service into a Compose project and test http://ollama:11434/api/tags from another container. Once that works, add your application or vector database and use the Docker RAG guide as a reference for the Ollama, Qdrant, and Streamlit pattern.
If you get stuck, share your Docker Compose file, the exact error message, and whether the failing request comes from the host or another container in the Ai-Forum community. Those three details usually identify the network layer that needs attention.
FAQ
Can another container reach Ollama at localhost:11434?
No. Inside a container, localhost refers to that container. Use the Ollama service name, such as http://ollama:11434, on a shared Docker network.
Do I need to publish port 11434?
Only if the host or external clients must reach Ollama. Containers on the same Docker network can use the container port directly.
How do I pull a model into the Ollama container?
Run docker exec -it ollama ollama pull llama3.2, replacing llama3.2 with the model you want.
Does Ollama in Docker support GPU acceleration?
Yes, with host setup. NVIDIA uses the NVIDIA Container Toolkit and --gpus=all; AMD uses the rocm tag and device flags. JetPack systems may need JETSON_JETPACK=5 or 6.
Why does my app container fail to connect to Ollama?
Check that both containers are on the same network, that the app uses the service name rather than localhost, that Ollama is running, and that the request targets port 11434.