How to Run llama-server for Multiple Users: Parallel Slots, Continuous Batching, and Metrics

Read time: ~8 minutes. What you’ll build: one OpenAI-compatible llama-server process that can accept several concurrent chat requests, keep useful prompt state between turns, expose queue and slot metrics, and survive a small load test without guessing at tuning values.

Version note: the flags and endpoints below were checked against the official llama.cpp master server documentation on August 26, 2026. llama.cpp changes quickly; run llama-server --help on your installed binary before copying the production command.

TL;DR — a practical four-slot server

llama-server \
  --model /models/model.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --parallel 4 \
  --cont-batching \
  --ctx-size 32768 \
  --batch-size 2048 \
  --ubatch-size 512 \
  --cache-prompt \
  --cache-reuse 256 \
  --metrics \
  --api-key "replace-this-key"

This creates four server slots, keeps continuous batching explicitly enabled, exposes /metrics, and retains prompt reuse. It is a starting configuration, not a universal optimum: the right slot count and context size depend on your model, backend, memory, prompt lengths, and latency target.

The important rule is simple: increase concurrency one step at a time and measure the queue, time to first token, and generation throughput after every change. More slots can improve aggregate throughput while making each individual response slower or exhausting KV-cache memory.

1. Understand the three controls

llama-server has three related controls that are easy to confuse.

--parallel N creates server slots

The official server README defines -np, --parallel N as the number of server slots. A slot holds the active state for one sequence. With four slots, the server can have up to four requests actively assigned at once; additional requests can be deferred until a slot becomes available.

Current builds use -1 for automatic slot selection. For a repeatable deployment, set an explicit value while tuning:

llama-server -m model.gguf --parallel 2

Start with 2 if this is a personal server shared by a few tools. Try 4 only after confirming memory headroom and acceptable per-request latency. Avoid jumping straight to 8 or 16: every active sequence needs KV-cache capacity, and the model still shares the same compute device.

--cont-batching mixes work from active requests

Continuous batching—also called dynamic batching—lets the server batch tokens from requests that arrive at different times instead of waiting for one fixed batch to finish. The official documentation lists it as enabled by default in current builds.

Keep the flag explicit in deployment scripts so an older package or changed default does not silently alter behavior:

--cont-batching

Slots provide the concurrent sequences; continuous batching lets the backend process those sequences efficiently together. You normally want both for a multi-user endpoint.

--batch-size and --ubatch-size control prompt processing

These two flags do not set the number of users:

  • --batch-size is the logical maximum batch size.
  • --ubatch-size is the physical maximum batch size processed at once.

Larger values can improve prompt prefill throughput, but consume more memory. The official server benchmark varies both values separately, which is the right model to copy: keep them fixed while testing slot count, then tune batches in a second pass.

--batch-size 2048 --ubatch-size 512

If the server fails during prompt ingestion or runs out of memory, reduce --ubatch-size first. If it is stable but long prompts prefill slowly, compare a few values with the same prompt set rather than assuming larger is always faster.

2. Set context size deliberately

Context is not free. Longer contexts increase KV-cache memory, and concurrent requests multiply the amount of active sequence state the server may need.

For a coding assistant, start with the smallest context that handles the real workload:

--ctx-size 16384

Move to 32768 only if repository context or long conversations regularly exceed 16K tokens. Do not select the model’s advertised maximum just because it exists. A shorter working context often leaves enough memory for more slots and avoids swapping or failed allocations.

After startup, inspect what each slot reports:

curl -s http://127.0.0.1:8080/slots \
  -H "Authorization: Bearer replace-this-key"

GET /slots is enabled by default in the current server. Its response includes each slot’s id, processing state, context capacity, token counts, and timing information. This endpoint is more useful than inferring capacity from the launch command alone.

3. Keep prompt caching on when requests share a prefix

Concurrent coding agents repeatedly send the same system prompt, tool schema, repository instructions, and earlier conversation turns. Reprocessing all of that on every request wastes prefill work.

Current llama-server builds enable prompt caching by default. Keep it explicit and add chunk reuse when prompts contain large repeated blocks that move within the request:

--cache-prompt --cache-reuse 256

The request body also supports cache_prompt, which defaults to true. A basic OpenAI-compatible request needs no special cache field:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer replace-this-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-model",
    "messages": [
      {"role": "system", "content": "You are a careful coding assistant."},
      {"role": "user", "content": "Explain this repository structure."}
    ],
    "max_tokens": 256
  }'

If follow-up requests still reprocess most of the prompt, use the dedicated llama-server KV-cache reuse guide. It covers --cache-ram, idle-slot caching, disk save/restore, SWA models, and the cases where matching prefixes are not enough.

4. Watch the queue instead of guessing

Enable the Prometheus-compatible endpoint:

--metrics

Then inspect the metrics that answer the concurrency question:

curl -s http://127.0.0.1:8080/metrics \
  -H "Authorization: Bearer replace-this-key" \
  | grep -E 'requests_processing|requests_deferred|busy_slots|tokens_seconds'

The official metric list includes:

  • llamacpp:requests_processing: requests currently processing.
  • llamacpp:requests_deferred: requests waiting rather than processing.
  • llamacpp:n_busy_slots_per_decode: average busy slots per decode call.
  • llamacpp:prompt_tokens_seconds: average prompt-processing throughput.
  • llamacpp:predicted_tokens_seconds: average generation throughput.

Interpret them together:

  • Deferred requests stay above zero: demand exceeds available slots or compute. Test one more slot only if memory has room.
  • Busy slots rise but predicted tokens/s per user collapses: the device is compute-bound; more slots are hurting latency.
  • Prompt throughput is the bottleneck: investigate cache hits and batch settings before adding slots.
  • The process swaps or allocates out: reduce context, slot count, or physical batch size.

Do not optimize only for aggregate tokens per second. An interactive coding assistant also needs acceptable queue time and time to first token.

5. Run a controlled concurrency test

First verify health:

curl -s http://127.0.0.1:8080/health

The healthy response is {"status":"ok"}. Now create one request payload:

REQUEST='{
  "model": "local-model",
  "messages": [{"role":"user","content":"Write a short Python binary search function."}],
  "max_tokens": 128,
  "stream": false
}'

Send four requests concurrently:

for i in 1 2 3 4; do
  curl -s http://127.0.0.1:8080/v1/chat/completions \
    -H "Authorization: Bearer replace-this-key" \
    -H "Content-Type: application/json" \
    -d "$REQUEST" \
    -o "/tmp/llama-response-$i.json" &
done
wait

While the requests run, inspect /slots and /metrics from another terminal. Repeat the exact workload with --parallel 1, 2, and 4. Record:

  1. total wall-clock time for all requests;
  2. slowest individual response;
  3. deferred-request count;
  4. prompt and predicted token throughput;
  5. peak GPU or system memory.

Use the smallest slot count that meets your queue target. That preserves memory for context and often gives more predictable interactive latency.

For a more rigorous test, llama.cpp ships an official k6-based server benchmark. Its documented example starts the server with continuous batching, metrics, eight slots, and explicit context and batch sizes, then runs 500 chat-completion iterations with eight virtual users. Treat those values as a benchmark example—not a recommended production preset—and substitute your own model and workload.

6. Production command for a small team

This is a conservative template for a local-network service or one machine running several coding clients:

llama-server \
  --model /models/model.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --api-key "$(openssl rand -hex 32)" \
  --parallel 4 \
  --cont-batching \
  --ctx-size 32768 \
  --batch-size 2048 \
  --ubatch-size 512 \
  --cache-prompt \
  --cache-reuse 256 \
  --metrics

Binding to 127.0.0.1 means remote machines cannot connect directly. For a team deployment, put the server behind a trusted reverse proxy or private network instead of changing the host to 0.0.0.0 and exposing port 8080 to the internet. Keep the API key at the proxy boundary and add TLS there.

If you enable file or shell tools, concurrency tuning is no longer the main risk. Follow the safe llama-server built-in tools guide and run tools inside a restricted container. Four parallel slots executing commands directly on your host create four opportunities for prompt injection or destructive actions.

7. Common mistakes

Setting --parallel equal to expected users

Ten occasional users do not necessarily need ten slots. Size for simultaneous active requests, not account count. Observe requests_deferred during real peaks.

Increasing slots and context together

That changes two major memory variables at once. Tune slot count with a fixed context first; only then test a larger context.

Disabling continuous batching to fix latency

This can reduce aggregate efficiency without fixing the real bottleneck. Check whether the queue comes from insufficient compute, slow prefill, or too many long outputs before disabling batching.

Benchmarking with different prompts every run

Prompt length, output length, and cache overlap materially change results. Reuse the same request set for every configuration comparison.

Exposing /slots and /metrics publicly

Monitoring endpoints reveal workload and capacity details. Keep them on localhost or protect them at the reverse proxy.

Use this order:

  1. Start with two explicit slots and the context your prompts actually need.
  2. Confirm continuous batching and prompt caching are enabled.
  3. Load-test the same request set at one, two, and four slots.
  4. Choose based on queue time and slowest response, not only total throughput.
  5. Add KV-cache persistence or tool isolation only after the base server is stable.

For a ready local coding model, pair this setup with Qwen3.6-35B-A3B in llama.cpp. For repeated long prompts, continue with the KV-cache reuse guide. For agentic file and command access, use the containerized built-in tools setup.

Sources