# vLLM on a DGX Spark — Qwen3.6 Setup Guide

Stand up a high-concurrency **Qwen3.6-35B-A3B (FP8)** inference server on a fresh
**NVIDIA DGX Spark (GB10, 128 GB unified, ARM64 / `sm_121`)** using Docker.
Every step and flag below was validated on a real DGX Spark — including the
non-obvious gotchas that cost time the first run.

## What you get (measured on a DGX Spark)

| Concurrency | Aggregate output | Per-stream |
|---|---|---|
| 1 request   | ~46 tok/s   | 46 tok/s |
| 4 requests  | ~123 tok/s  | ~31 tok/s |
| **8 requests** | **~173 tok/s** | ~22 tok/s |
| 16 requests | ~168 tok/s (plateau) | ~10 tok/s |

- Single OpenAI-compatible endpoint on `:8000` (`/v1/chat/completions`, etc.)
- **Continuous batching + prefix caching** (on by default) → multiple agents run
  at once instead of queuing. `--max-num-seqs 8` is the throughput sweet spot on
  this box; beyond ~8 concurrent it plateaus and just adds latency.
- **Tool/function calling**, **vision (images)**, and **thinking-off** all working
  on the same model and endpoint.

> Why FP8 MoE? The GB10 is memory-bandwidth-bound. A sparse MoE (only ~3B params
> active per token) + continuous batching amortizes that bandwidth across
> concurrent requests, which is exactly where this hardware shines.

---

## Prerequisites

- DGX Spark running DGX OS (Docker + NVIDIA Container Toolkit preinstalled).
- ~70 GB free disk (23 GB image + 35 GB weights + headroom).
- Internet access. The Qwen FP8 repo is **public — no Hugging Face token needed**
  (you can set one for faster downloads / higher rate limits, but it's optional).

Sanity check Docker can see the GPU:

```bash
docker run --rm --gpus all nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi
```

---

## Step 1 — Pull the vLLM image

```bash
docker pull vllm/vllm-openai:cu130-nightly
```

> `cu130-nightly` is the ARM64 + CUDA 13 build that supports the GB10's `sm_121`.
> For a reproducible deploy, pin a specific digest once you've validated one:
> `docker pull vllm/vllm-openai@sha256:<digest>`.

## Step 2 — Pre-stage the model weights (~35 GB)

Downloading first means the container starts in ~minutes instead of waiting on a
35 GB pull at launch.

```bash
# DGX OS has `hf` preinstalled; if not: pipx install huggingface_hub
hf download Qwen/Qwen3.6-35B-A3B-FP8
```

Weights land in `~/.cache/huggingface`, which the container mounts in Step 3.

## Step 3 — Create the launch script

Save as `~/vllm/run-vllm.sh`, then `chmod +x ~/vllm/run-vllm.sh`:

```bash
#!/usr/bin/env bash
# Launch Qwen3.6-35B-A3B-FP8 under vLLM on :8000.
set -euo pipefail

IMAGE="vllm/vllm-openai:cu130-nightly"
MODEL="Qwen/Qwen3.6-35B-A3B-FP8"

docker rm -f vllm 2>/dev/null || true
docker run -d --name vllm --ipc=host --restart unless-stopped \
  --gpus all -p 8000:8000 \
  ${HF_TOKEN:+-e HF_TOKEN="${HF_TOKEN}"} \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  "$IMAGE" \
  "$MODEL" \
    --served-model-name agent \
    --host 0.0.0.0 --port 8000 \
    --gpu-memory-utilization 0.85 \
    --max-num-seqs 8 \
    --max-model-len 131072 \
    --default-chat-template-kwargs '{"enable_thinking": false}' \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --trust-remote-code

echo "Launched. Watch startup:  docker logs -f vllm"
```

Run it:

```bash
~/vllm/run-vllm.sh
docker logs -f vllm          # ready when you see "Application startup complete"
```

First boot takes ~6–7 min (weight load + CUDA graph capture). Ctrl-C stops the
log tail; the container keeps running.

> ### The flags that matter (and the traps)
> - **`"$MODEL"` with NO `vllm serve` in front.** This image's ENTRYPOINT is
>   already `["vllm","serve"]`. If you write `... "$IMAGE" vllm serve "$MODEL"`
>   it doubles to `vllm serve vllm serve ...` and dies with
>   `unrecognized arguments: serve Qwen/...`.
> - **`--served-model-name agent`** — vLLM *validates* the `model` field in
>   requests (unlike llama.cpp, which ignores it). Clients must send the served
>   name (`"model":"agent"`), or omit this flag and send the full repo id.
> - **`--default-chat-template-kwargs '{"enable_thinking": false}'`** — without
>   it, Qwen3.6 leaks its `<think>` reasoning into responses. This sets it off by
>   default for every request (a request can still override per-call).
> - **`--enable-auto-tool-choice --tool-call-parser qwen3_xml`** — required for
>   function calling. Without them, any request with `tool_choice:"auto"` returns
>   `400 "auto" tool choice requires ...`. Qwen3.6 emits the XML
>   `<tool_call><function=…>` format, so the parser is **`qwen3_xml`** — NOT the
>   legacy `hermes` (JSON) parser.

---

## Step 4 — Verify

```bash
# 1) Model is served under the expected name
curl -s http://localhost:8000/v1/models | jq -r '.data[].id'      # -> agent

# 2) Plain chat — thinking should be OFF (clean answer, no <think>)
curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"agent","messages":[{"role":"user","content":"In one sentence, what is a load balancer?"}],"max_tokens":60}' \
  | jq -r '.choices[0].message.content'

# 3) Tool calling — expect finish_reason "tool_calls" with get_weather(Paris)
curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"agent",
       "messages":[{"role":"user","content":"What is the weather in Paris?"}],
       "tools":[{"type":"function","function":{"name":"get_weather","description":"Get weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
       "tool_choice":"auto","max_tokens":200}' \
  | jq '{finish:.choices[0].finish_reason, tool_calls:.choices[0].message.tool_calls}'
```

**Vision (images) — same model, same endpoint.** Qwen3.6-35B-A3B is multimodal
(`Qwen3_5MoeForConditionalGeneration`), so no second model is needed. Quick test
with a public image URL:

```bash
curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"agent","max_tokens":50,
       "messages":[{"role":"user","content":[
         {"type":"text","text":"Describe this image in one sentence."},
         {"type":"image_url","image_url":{"url":"https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gull_portrait_ca_usa.jpg/320px-Gull_portrait_ca_usa.jpg"}}
       ]}]}' \
  | jq -r '.choices[0].message.content'
```

(You can also pass a `data:image/png;base64,...` URI instead of a URL.)

---

## Step 5 — (Optional) Benchmark concurrency

Save as `~/vllm/bench.sh`, `chmod +x`, run it. Each request is forced to exactly
256 output tokens so the numbers are clean:

```bash
#!/usr/bin/env bash
URL=http://localhost:8000/v1/chat/completions
OUT=256
P="Write a thorough, detailed explanation of how TCP congestion control works."
req(){ curl -s "$URL" -H 'Content-Type: application/json' \
  -d "{\"model\":\"agent\",\"messages\":[{\"role\":\"user\",\"content\":\"$P\"}],\"max_tokens\":$OUT,\"min_tokens\":$OUT,\"ignore_eos\":true}" \
  | jq -r '.usage.completion_tokens // 0'; }
printf "%-6s %-9s %-9s %-12s %s\n" conc wall_s out_tok AGG_t/s per_stream
for C in 1 4 8 16; do
  tmp=$(mktemp -d); s=$(date +%s.%N)
  for n in $(seq 1 "$C"); do ( req >"$tmp/$n" ) & done; wait
  e=$(date +%s.%N); t=0; for n in $(seq 1 "$C"); do t=$((t+$(cat "$tmp/$n"))); done
  awk -v c=$C -v s=$s -v e=$e -v t=$t 'BEGIN{d=e-s;a=t/d;printf "%-6d %-9.1f %-9d %-12.1f %.1f\n",c,d,t,a,a/c}'
  rm -rf "$tmp"
done
```

---

## Step 6 — (Optional) Auto-start on boot

The `--restart unless-stopped` in the script already revives the container after
a crash or reboot. For tighter control, wrap it in a systemd unit instead:

```ini
# /etc/systemd/system/vllm.service
[Unit]
Description=vLLM Qwen3.6 server
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=%h/vllm/run-vllm.sh
ExecStop=/usr/bin/docker stop vllm

[Install]
WantedBy=multi-user.target
```

```bash
sudo systemctl daemon-reload && sudo systemctl enable --now vllm
```

---

## Tuning levers (edit `run-vllm.sh`, re-run)

| Flag | Effect |
|---|---|
| `--max-num-seqs 8` | Max concurrent streams. 8 is the GB10 sweet spot; raise only if you have many agents *and* accept lower per-stream speed; lower toward 4 if per-token speed sags. |
| `--max-model-len 131072` | Context ceiling. If startup OOMs, lower it (e.g. `65536`) — KV cache for 8 long sequences is the tightest part of the memory budget. |
| `--gpu-memory-utilization 0.85` | Fraction of the 128 GB vLLM may use. Leave headroom for the OS; lower if the box gets unstable. |

---

## Troubleshooting quick reference

| Symptom | Cause / fix |
|---|---|
| `unrecognized arguments: serve Qwen/...` | You wrote `vllm serve` before the model. Image entrypoint already includes it — pass just the model id. |
| `400 ... "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser` | Tool calling not enabled. Add `--enable-auto-tool-choice --tool-call-parser qwen3_xml`. |
| Responses contain `<think>...` reasoning | Add `--default-chat-template-kwargs '{"enable_thinking": false}'`. |
| `404` / "model not found" on requests | Client sent a `model` name that isn't the served name. Send `"model":"agent"` (or your chosen `--served-model-name`). |
| Container exits during load, logs show `out of memory` | Lower `--max-model-len` and/or `--max-num-seqs`. |
| Stop the server | `docker stop vllm` (or `docker rm -f vllm`). |

---

## Notes

- **Engine choice:** vLLM is the right pick when you run *multiple agents at once*
  (continuous batching → high aggregate throughput). If instead you want the
  fastest *single* stream, llama.cpp with Qwen3.6's built-in MTP speculative
  decoding wins for one request — but it doesn't batch, so concurrent requests
  serialize. Pick based on your workload.
- **Don't use Gemma for vision** on this hardware — its projector is currently
  broken in the GGUF/llama.cpp path (garbage image output). Qwen3.6's built-in
  vision (above) is the reliable route.