# vLLM on DGX Spark (GB10): `400 Unexpected message role.`

A fix writeup for anyone running **vLLM serving a Qwen3.x model** (FP8/Spark or
otherwise) and seeing agents intermittently fail with HTTP 400.

## Symptom

Agent / tool-calling requests fail with:

```
Error: 400 Unexpected message role.
```

Server side, vLLM logs a Jinja template error:

```
jinja2.exceptions.TemplateError: Unexpected message role.
  File ".../vllm/renderers/hf.py", line 502, in safe_apply_chat_template
  File "<template>", line 144, in top-level template code
  ...
POST /v1/chat/completions HTTP/1.1" 400 Bad Request
```

It's often **intermittent** — especially if vLLM is only one backend behind a
load balancer. In our setup it was the *backup* upstream, so only the requests
that overflowed onto the vLLM box failed; everything served by the primary
(llama.cpp) box worked. If you have a single vLLM backend you'll see it on
every affected request.

## Root cause

vLLM applies the **model repo's bundled `chat_template.jinja`** (via Hugging
Face `tokenizer.apply_chat_template`). The Qwen3.x template hard-whitelists the
roles it knows and **raises** on anything else:

```jinja
{%- if message.role == "system" %} ...
{%- elif message.role == "user" %} ...
{%- elif message.role == "assistant" %} ...
{%- elif message.role == "tool" %} ...
{%- else %}
    {{- raise_exception('Unexpected message role.') }}   <-- this line
{%- endif %}
```

So any message whose `role` is **not** `system` / `user` / `assistant` / `tool`
blows up. The usual culprit is a `developer` role (OpenAI o1+ style instruction
messages that many agent frameworks/SDKs now emit), but it can be any custom
role your orchestration layer sends.

Note: `llama.cpp --jinja` running the **GGUF's embedded** template tolerates the
same role, which is why a mixed pool (llama.cpp primary + vLLM backup) only fails
on the vLLM side — it makes the bug look flaky.

### How to confirm the exact role

- Reproduce directly against vLLM with a minimal body containing the suspect
  role (e.g. `{"role":"developer", ...}`) — a plain `user` message will succeed,
  isolating the role as the trigger.
- Or grep your load balancer's access log for the 400s and confirm they all
  point at the vLLM upstream.

## Fix: a tolerant chat template

Don't change your agents — make the template stop raising. Render any
unrecognized role as a normal ChatML turn (preserving its label) instead of
throwing. This matches the permissive behavior of llama.cpp.

**1. Pull the model's template out of the running container:**

```bash
SNAP=/root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B-FP8/snapshots/<hash>
docker exec vllm cat $SNAP/chat_template.jinja > /opt/vllm/chat_template.patched.jinja
```

**2. Patch the `else` branch** — replace the raise with a passthrough:

```diff
 {%- else %}
-    {{- raise_exception('Unexpected message role.') }}
+    {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
 {%- endif %}
```

(If you'd rather coerce the role, map it to a known one instead — e.g.
`developer` → `system`. But beware: this template also raises
`"System message must be at the beginning."`, so a mid-conversation
`developer`→`system` remap can trip a *different* error. The generic passthrough
above is the safest, lowest-surprise fix.)

**3. Validate it renders BEFORE restarting** (catch typos offline):

```bash
docker cp /opt/vllm/chat_template.patched.jinja vllm:/tmp/ct.jinja
docker exec vllm python3 - <<'PY'
from jinja2.sandbox import ImmutableSandboxedEnvironment
env = ImmutableSandboxedEnvironment(trim_blocks=True, lstrip_blocks=True)
env.globals['raise_exception'] = lambda m: (_ for _ in ()).throw(RuntimeError(m))
t = env.from_string(open('/tmp/ct.jinja').read())
msgs = [{'role':'system','content':'be brief'},
        {'role':'developer','content':'follow the style guide'},
        {'role':'user','content':'hello'}]
print(t.render(messages=msgs, add_generation_prompt=True, enable_thinking=False))
PY
```

**4. Point vLLM at the patched template.** Mount it in and add the flag:

```bash
docker run -d --name vllm --ipc=host --gpus all -p 8000:8000 \
  -v /root/.cache/huggingface:/root/.cache/huggingface \
  -v /opt/vllm:/opt/vllm \
  vllm/vllm-openai:cu130-nightly \
  Qwen/Qwen3.6-35B-A3B-FP8 \
    --served-model-name agent \
    --chat-template /opt/vllm/chat_template.patched.jinja \
    --enable-auto-tool-choice --tool-call-parser qwen3_xml \
    --max-model-len 131072 --gpu-memory-utilization 0.85 \
    --trust-remote-code
```

`--chat-template <file>` overrides the repo's bundled template globally for the
server. Bake it into whatever launch script/systemd unit you use so a restart
keeps the fix.

**5. Verify the previously-failing role now returns 200:**

```bash
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"agent","messages":[
        {"role":"system","content":"be brief"},
        {"role":"developer","content":"answer in one word"},
        {"role":"user","content":"hi"}],"max_tokens":8}'
```

## DGX Spark / GB10 gotchas worth knowing

- **Slow first boot is normal.** FP8 weight load (~tens of GB) + `torch.compile`
  can take **4–6 minutes** before `:8000` answers. Don't mistake a long load for
  a crash — `docker logs -f vllm` shows the safetensors load %, then a
  `torch.compile took Ns` line, then Uvicorn startup. A readiness poll with a
  short timeout will falsely report failure.
- **`--served-model-name`** matters: vLLM *validates* the request `model` field
  (llama.cpp ignores it). If clients send `model: "agent"`, you must serve under
  that name or they 404.
- **Tool calling** on Qwen3.x needs `--enable-auto-tool-choice
  --tool-call-parser qwen3_xml` — it emits XML `<tool_call><function=...>`, not
  Hermes-style JSON.
- **Thinking off:** `--default-chat-template-kwargs '{"enable_thinking": false}'`
  (or set it per request); otherwise reasoning leaks into `content`.
- **Unified memory** is shared with the OS — watch `--gpu-memory-utilization`
  and `--max-num-seqs` so KV cache + model + OS all fit; pushing too high sends
  the box into swap.

## TL;DR

The model repo's chat template throws on roles outside
`system/user/assistant/tool` (your agents send `developer`). Override vLLM with a
patched template that renders unknown roles instead of raising
(`--chat-template /path/to/patched.jinja`), validate it offline, and bake the
flag into your launch script.