AIForum.

How to Use Multiple GPUs for Local AI: Tensor Parallelism vs Layer Offloading

On this page

THE SHORT ANSWER

Spreading a local model across two or more GPUs is a runtime flag decision, not a rebuild. llama.cpp defaults to layer splitting (pipeline parallelism), which pools VRAM and tolerates mismatched cards, while the experimental tensor mode splits weights and KV cache across GPUs and wants identical hardware. This guide explains what each mode does, which flags control it, and how to tell which one your machine actually needs.
Quick facts

If a model does not fit in one GPU’s VRAM, you have two practical ways to spread it: layer splitting, also called pipeline parallelism, and tensor parallelism. In llama.cpp, layer splitting is the default and the most compatible choice; tensor parallelism is an experimental mode that splits both weights and the KV cache across GPUs and generally wants identical cards. Both are selected at runtime with command-line flags rather than by recompiling anything, and the same arguments apply across most llama.cpp binaries, including llama-cli and llama-server.

This article is for practitioners who already run local models and are deciding whether a second GPU is worth adding, or who have added one and are trying to work out why performance did not improve. The short version: layer splitting buys you capacity, tensor parallelism aims at speed, and the two behave very differently on mismatched hardware.

Two ways to spread a model across GPUs

Layer splitting cuts the model by transformer layer. Each GPU holds a contiguous slice of layers, and the KV cache for layer l lives on whichever GPU owns layer l. Tokens pass through the cards in order, which is why the whole arrangement behaves like one long pipeline rather than a pool of independent workers.

Tensor parallelism cuts differently. Instead of dividing whole layers, it divides the weights within layers, and it splits the KV cache as well. That means every GPU participates in the same layer at the same time, which is the property that makes it attractive for speed — and also the property that makes it sensitive to how fast the GPUs can talk to each other.

The distinction matters because it changes what you are buying. Layer splitting is fundamentally a capacity strategy: it lets a model too large for one card stay entirely on accelerators instead of spilling into comparatively slower system RAM. Tensor parallelism is a throughput strategy: each GPU does less work per token, and whether that actually helps depends on the split mode and the interconnect speed relative to a single GPU’s own speed.

Layer splitting: the default, and usually the right first move

The llama.cpp multi-GPU documentation lists layer as the default split mode and describes it as the most compatible multi-GPU choice. It is recommended when you want more memory than a single GPU provides and your priority is fast prefill, and it can tolerate slow interconnect speeds between GPUs.

There is a common expectation to correct here. Adding a second card under layer splitting does not automatically make generation faster, because the cards take turns rather than working simultaneously. As the localai.computer guide to splitting models across cards puts it, layer splitting means capacity pools and speed does not. If your model already fits comfortably on one GPU and you were hoping for a speedup, layer splitting is unlikely to be the answer.

The other advantage is hardware tolerance. Layer splitting does not care whether the cards match. That is significant for home servers, where the cheapest route to more memory is often a second card of a different model or size than the first.

Tensor parallelism: experimental, and picky about hardware

The tensor mode is marked experimental in the llama.cpp documentation. It splits both weights and KV across GPUs, in contrast to the older row mode, which is now deprecated and described as a comparatively poor-performing path that only split dense weights. The documentation states that row is superseded by tensor and should be avoided in new deployments.

Because tensor parallelism requires the GPUs to cooperate on every layer, identical cards are the sensible target configuration. The localai.computer guide states this plainly: tensor parallelism wants identical cards, while layer splitting does not care.

Treat tensor as something to test rather than something to assume. The mode is experimental, and the benefit depends on interconnect speed versus single-GPU speed, so the result on your specific machine is an empirical question.

Comparing the split modes

The table below summarises the modes as documented, including the deprecated one, so you can recognise it if you encounter it in an older configuration.

ModeWhat it doesWhen to use it
noneUses a single GPU only; choose which one with --main-gpu.You want to confine the model to one GPU even though more are visible.
layer (default)Pipeline parallelism. Each GPU holds a contiguous slice of layers; the KV cache for layer l lives on the GPU that owns layer l.The default and most compatible multi-GPU choice. Use it when you need more memory than one GPU provides and want fast prefill. Tolerates slow interconnects.
rowDeprecated older row-split tensor-parallel path; splits only dense weights.Avoid in new deployments; superseded by tensor.
tensorExperimental tensor parallelism that splits both weights and KV across GPUs.Test it when you have matched cards and want to see whether throughput improves.

The flags that control the split

Three arguments cover most home setups: CUDA_VISIBLE_DEVICES decides which cards the runtime can see at all, --device picks among the ones it can see, and --tensor-split decides how much of the model each one carries. The split mode itself is set with --split-mode or its short form -sm.

The flag people get wrong is --tensor-split, because its numbers are proportions rather than gigabytes and rather than device numbers. A split of 2,1 puts two thirds of the layers on the first card and one third on the second — which is what you want when the first card has twice the memory. It is not a request for 2 GB and 1 GB, and it is not a list of GPU indices.

# Restrict the runtime to two specific cards
CUDA_VISIBLE_DEVICES=0,1 llama-cli 
  -m model.gguf 
  --split-mode layer 
  --tensor-split 2,1 
  -ngl 999

Here -ngl controls how many layers to offload; set it high enough to keep everything on the GPUs. If you leave layers on the CPU, you are back to the situation multi-GPU was meant to solve, and the pipeline will be limited by the slowest stage.

To try tensor parallelism instead, change one argument:

CUDA_VISIBLE_DEVICES=0,1 llama-cli 
  -m model.gguf 
  --split-mode tensor 
  --tensor-split 1,1 
  -ngl 999

Note that the split proportions that suit layer splitting on mismatched cards do not necessarily suit tensor mode, precisely because tensor mode prefers matched hardware.

Running the same setup through Ollama

Ollama is llama.cpp underneath, so the same behaviour appears there through environment variables instead of flags. The equivalent of spreading the model across visible cards is OLLAMA_SCHED_SPREAD=1 alongside CUDA_VISIBLE_DEVICES, set in the service environment and applied after a restart. If you change the variable and see no difference, confirm the service actually restarted — an environment variable set in your interactive shell will not reach a daemon that was already running.

Planning memory beyond the weights

Weight size is not the same as total memory use. The KV cache scales with context length and model architecture, and the runtime itself carries overhead. This is why a model whose file size sits comfortably under your combined VRAM can still fail to load at a long context, and why raising the context length can push a previously working split over the edge.

The practical implication for split planning is to leave headroom rather than filling both cards to the brim. If you are sizing a purchase, estimate the weights, then add an allowance for the KV cache at the context length you actually intend to use, plus runtime overhead. Treat any single number you see online as a starting estimate, not a specification.

Troubleshooting common symptoms

Most multi-GPU problems present as one of a few recognisable patterns. Work through them in this order, from least invasive to most.

  1. An out-of-memory error from the GPU backend despite enough combined VRAM. Check that -ngl is high enough to offload the layers you expect, and reduce context length to see whether the KV cache is the constraint rather than the weights.
  2. One GPU is idle or barely used. Verify the runtime can see both cards, then check whether --tensor-split is distributing as you intended. Remember the values are proportions, so 1,1 and 2,1 mean very different things.
  3. Generation is slower than a single GPU. This is expected behaviour under layer splitting when the model already fit on one card. Reverting with --split-mode none and --main-gpu is the quickest way to confirm the diagnosis.
  4. Tensor mode is unstable or disappointing. Confirm the cards are matched, then fall back to layer. The mode is documented as experimental, so a regression to the default is a legitimate outcome, not a configuration failure.

If none of these resolve the problem, the missing information is usually the actual per-GPU memory allocation at load time and the interconnect between the cards. Those two facts determine which mode can work at all.

A reproducible way to compare the two modes

No benchmark numbers are reproduced here, because none were supplied for a specific machine. If you want a defensible comparison on your own hardware, record the following before drawing conclusions: GPU models and VRAM per card, OS and driver version, llama.cpp build or commit, model name and quantization, context length, the exact --split-mode and --tensor-split values, and the -ngl setting.

Then run the same prompt set at least three times per mode, and report prefill and token generation separately rather than as one figure. Layer splitting and tensor parallelism affect those two phases differently, and a single blended number will hide the effect you are trying to measure. State the variation between repetitions; on a home server, background load and thermal behaviour can move results enough to matter.

Deciding for your own machine

If your goal is to run a model that does not fit on one card, start with the default layer mode, set -ngl high, and use --tensor-split to weight the split toward the larger card. This is the configuration the documentation recommends for memory-constrained setups and the one that tolerates mismatched hardware.

If your goal is throughput and you have matched cards, test tensor mode against layer on your own workload, with a measurement protocol rather than an impression. Keep the layer configuration as your known-good fallback, since tensor mode remains experimental.

If you are still deciding whether to buy a second GPU at all, the question to answer first is whether you are short on capacity or short on speed. Layer splitting solves the first problem and generally does not solve the second.

Readers working through a multi-GPU build are welcome to share their goal, card combination, chosen split mode, and measured result in the community, so that others can compare configurations rather than guess.

FAQ

Does adding a second GPU make token generation faster?

Not automatically. Layer splitting, the default, is a capacity strategy: the cards take turns rather than working simultaneously, so it pools VRAM but does not inherently increase speed. The llama.cpp documentation notes that throughput gains from distributing computation depend on the split mode and on interconnect speed relative to a single GPU's speed.

Do my two GPUs need to be the same model?

For layer splitting, no — mismatched cards are fine, which is why it is the recommended option for home servers where the cheapest route to more memory is often a different card. For tensor parallelism, identical cards are the sensible target configuration.

Is tensor parallelism ready for production use?

The llama.cpp documentation marks the tensor split mode as experimental. It is worth testing on matched hardware, but keep a working layer-mode configuration as your fallback.

What does --tensor-split 2,1 actually do?

It weights the split so that the first card carries two thirds and the second carries one third. The values are proportions, not gigabytes, and they are not device numbers.

References

  1. https://github.com/ggerganov/llama.cpp/blob/master/docs/multi-gpu.md
  2. https://localai.computer/homeserver/guides/running-across-cards

Related terms

Pipeline parallelism
Splitting a model by layer so each GPU holds a contiguous slice and tokens pass through the cards in order. This is what llama.cpp's default layer mode does.
Tensor parallelism
Splitting the weights within layers, and the KV cache, across GPUs so every GPU works on the same layer simultaneously. Available in llama.cpp as the experimental tensor mode.
KV cache
Memory used to store attention keys and values during inference. It grows with context length and, under layer splitting, lives on the GPU that owns the corresponding layer.
--tensor-split
A llama.cpp flag that sets how much of the model each GPU carries, expressed as proportions rather than gigabytes or device indices.
-ngl / --n-gpu-layers
The number of model layers to offload to the GPU. Set it high enough that the whole model stays on accelerators when using multiple GPUs.

Leave a Reply

Your email address will not be published. Required fields are marked *