The Same Box, Twice as Fast
A follow-up to the DGX Spark writeup. Same GB10 box, same model, one change to the storage format, and the numbers roughly double.
A few weeks ago I wrote about setting up a small AI server on a GX10, the little NVIDIA GB10 box on my desk. It ran a model called Qwen3.6-35B-A3B in a format called FP8, and it was quick enough that I stopped thinking about it. Then someone posted their numbers for the same box running a different format, and the numbers were better than mine. So I spent an evening finding out whether the same hardware could go faster without buying anything.
It can. Close to twice as fast, on the exact same box, running the exact same model.
What actually changed
The only thing I changed was the format the model is stored in. FP8 packs each number into eight bits. The new one, NVFP4, packs most of them into four. Fewer bits means less memory to move for every word the model writes, and moving memory is the slow part on this chip. NVFP4 was built for the Blackwell chip inside the GB10, so this is not a hack. It is the format the hardware was designed to run.
Same model, same answers, half the storage. The weights that used to take about 35 GB now take about 20. That spare room matters more than I expected, and I will come back to it.
The numbers
Here is what the old FP8 setup did and what the new NVFP4 setup does, measured the same way on the same machine.
| Requests at once | FP8 | NVFP4 |
|---|---|---|
| 1 | 46 tok/s | 88 tok/s |
| 4 | 123 tok/s | 164 tok/s |
| 8 | 173 tok/s | 317 tok/s |
| 16 | 168 tok/s | 341 tok/s |
A single answer now comes back at about 88 tokens per second, up from 46. That is the number you feel when you are chatting with it.
The bottom row is the more interesting one. With FP8, once about eight requests were running at once, the box stopped getting faster. It settled around 173 and adding more work did nothing. NVFP4 keeps climbing. At sixteen parallel requests it was still going up, past 340, and it had memory to spare the whole time. That spare memory is the reward for the smaller weights. The space the model gave back went to the part that tracks every conversation at once, and there was about four times more of it than before. So the box does not only answer faster. It holds a bigger crowd before it strains.
The bug, again
Last time there was a bug, so it feels right that there was one this time too. The first launch failed on the spot. The version of the serving software I had been running could not read NVIDIA's NVFP4 files. It went looking for one specific piece of each expert layer, could not find it under the name it expected, and quit with a short error and a long stack trace.
The fix was boring in the best way. A newer release of the same software read the files without complaint. No patching, no forum archaeology, just a version bump. If you ever see a KeyError with w2_input_scale in it, that is this, and the answer is a newer vLLM.
The handoff
The first post ended with me sending my brother a markdown file and letting his agent do the setup while we both slept. This one ends the same way, because the upgrade is smaller than the original build. I wrote down the model name, the one command that launches it, the version you need so you do not trip on the bug, and how to check it worked. Then I sent him the file.
He already has the old setup running from last time. If the new one misbehaves, the old one is one command away, so there is no risk in trying it. By the time he reads this, the file will already be on his box, and his agent will have done the swap or told him why it could not.
Same box. Twice as fast. One file.
If you want to run the same upgrade yourself, here is the file I sent him.