A bigger model on the same box
The third post about the GB10 box. A model five times the size, a third of the speed, and a 3090 sitting idle because of it.
Third post about the little GB10 box on my desk. The first one set it up with Qwen3.6-35B in FP8. The second swapped the storage format to NVFP4 and roughly doubled the speed. This one swaps the model entirely, and the numbers go the other way.
That is the point. I gave up speed on purpose.
What is running now
DeepSeek V4 Flash, packaged for the Spark by a recipe that pins everything by digest. The weights are 185 GB on disk and about 95 GB once loaded, against the 20 GB the NVFP4 Qwen took. It is a much larger model living in the same 128 GB of unified memory, and there is almost nothing left over.
It uses speculative decoding the same way the Qwen setup did, but with the model's own draft stages rather than a bolted on one. Three native DSpark stages, a fixed draft depth of five, and a separate 64 expert draft derived from the full 216 expert target so the draft does not eat the memory the KV cache needs.
The numbers
Aggregate decode throughput, measured the same way as the last two posts, on the same machine.
| Requests at once | Qwen FP8 | Qwen NVFP4 | DeepSeek V4 Flash |
|---|---|---|---|
| 1 | 46 tok/s | 88 tok/s | 37 tok/s |
| 4 | 123 tok/s | 164 tok/s | 67 tok/s |
| 8 | 173 tok/s | 317 tok/s | 104 tok/s |
| 16 | 168 tok/s | 341 tok/s | 106 tok/s |
So it is roughly a third of the NVFP4 setup at the top end, and slower than the original FP8 build too. The flat line after eight is my own scheduler cap, not the hardware. Requests past eight queue instead of failing.
I knew it would be slower. A 95 GB model on a memory bandwidth bound chip is going to be slower than a 20 GB one. What I wanted to know was whether the box could hold something this size at a usable speed at all, and 37 tokens a second on a single stream is usable for agent work where most turns are short.
The bug
There is always one. Chat worked immediately. Structured output worked. Every request that carried tools returned a 500.
The image ships one version of a grammar library, and the serving code inside it imports a function that only exists in a later version. The import sits at the top of the module, so it fires whether or not you use the feature, and no configuration flag routes around it. Ordinary chat never touches that path, which is why the box looked healthy.
The fix is a one line layer on top of the published image that installs the newer library and nothing else. I pinned it as a local image and told compose never to pull, so an upstream refresh cannot quietly undo it.
That last part matters more than the fix. If a future update puts me back on the stock image, the first symptom is that tool calling breaks while chat keeps working, which is exactly the failure that looks like something else.
The 3090 is cold
Before this box existed, a 3090 in another machine did all the inference. Then the Spark got fast enough to take the load, and I pointed everything at it.
Right now the 3090 has a model loaded, 22 GB of its 24 GB of memory occupied, and it is drawing 22 watts doing nothing at all. Every one of the last hundred requests went to the Spark. The card is warm in the sense that the weights are resident and cold in the sense that it has not done a useful thing in days.
That is a waste of a perfectly good card, and I have not decided what to do about it. It cannot hold anything near the size of what the Spark is running, so putting it back in the same pool does not make sense. The interesting options are the ones where being a separate machine is the point rather than a limitation. Embeddings and reranking for search. Speech to text. Image generation, which it is already set up for and which the Spark has no room to run alongside a 95 GB model. A small fast model for the jobs where latency matters more than quality.
If you have a spare 24 GB card sitting next to a bigger box and you have found something genuinely worth pointing it at, I would like to hear it.
The file
Same as the last two posts. Everything I did is written down in one markdown file, pinned by digest and revision, including the three checks that tell you whether it actually works and the one that catches the tools bug.
dgx-spark-deepseek-v4-setup.md
It replaces the Qwen instructions rather than extending them. The two models cannot share the box, so you pick one.