# Upgrading the DGX Spark Agent from FP8 to NVFP4

This upgrades the agent server from the first guide (Qwen3.6-35B-A3B in FP8) to the NVFP4 build of the same model. Same box, same model, roughly double the throughput. It is a drop-in swap on port 8000, so nothing downstream has to change. If the FP8 server is not already running, set that up first using the original guide.

## What you get

NVFP4 is a four-bit format built for the Blackwell GPU in the GB10. The weights drop from about 35 GB to about 20 GB, which speeds up every token and frees a large amount of memory for the KV cache. Measured on a GX10, same machine, same test:

| Requests at once | FP8 | NVFP4 |
|---|---|---|
| 1 | 46 tok/s | 88 tok/s |
| 4 | 123 tok/s | 164 tok/s |
| 8 | 173 tok/s | 317 tok/s |
| 16 | 168 tok/s (flat) | 341 tok/s (still scaling) |

## The one gotcha: vLLM version

NVIDIA's NVFP4 checkpoint will not load on older vLLM builds, including some nightlies. The failure happens at startup and looks like this:

    KeyError: 'layers.0.mlp.experts.w2_input_scale'

The fix is a newer release, not a patched checkpoint. Use vllm/vllm-openai:v0.25.1 or later and it loads without complaint. Do not spend time editing the weights.

## Steps

1. Pull the image:

    docker pull vllm/vllm-openai:v0.25.1

2. Launch NVFP4 on port 8000. This is the FP8 launch from the first guide with three changes: the model id, the newer image, and a speculative-decoding setting the NVFP4 model supports.

    docker rm -f vllm 2>/dev/null || true
    docker run -d --name vllm --ipc=host --restart unless-stopped \
      --gpus all -p 8000:8000 \
      -v /root/.cache/huggingface:/root/.cache/huggingface \
      vllm/vllm-openai:v0.25.1 \
      nvidia/Qwen3.6-35B-A3B-NVFP4 \
        --served-model-name agent \
        --host 0.0.0.0 --port 8000 \
        --gpu-memory-utilization 0.85 \
        --max-num-seqs 8 \
        --max-model-len 196608 \
        --default-chat-template-kwargs '{"enable_thinking": false}' \
        --enable-auto-tool-choice \
        --tool-call-parser qwen3_xml \
        --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
        --limit-mm-per-prompt '{"image":4}' \
        --trust-remote-code

   First boot loads the weights and compiles graphs. Give it about five minutes and watch it with: docker logs -f vllm . It is ready when the log says "Application startup complete".

3. Confirm it is serving the right model and context:

    curl -s http://127.0.0.1:8000/v1/models

   You want id "agent" and max_model_len 196608.

4. Smoke test a plain reply, a tool call, and an image, so you know nothing regressed:

    curl -s http://127.0.0.1:8000/v1/chat/completions \
      -H 'Content-Type: application/json' \
      -d '{"model":"agent","messages":[{"role":"user","content":"say ok"}],"max_tokens":5}'

## Rollback

Keep the FP8 launch command from the first guide. If anything about NVFP4 misbehaves, remove this container and run the FP8 one. Both use port 8000 and the name "agent", so downstream clients never notice the switch.

    docker rm -f vllm
    # then run your saved FP8 launch command

## Notes

- If several machines sit behind one proxy, keep max-model-len the same on all of them so the advertised context matches everywhere.
- max-num-seqs 8 is a safe default. NVFP4 leaves enough spare memory to raise it if you want more parallel streams. The KV cache on a GX10 held over three million tokens in testing.
- The model is multimodal. The same server answers text and images, so a vision endpoint does not need a second process.