# DeepSeek V4 Flash on a DGX Spark

This replaces the earlier Qwen instructions. Same box, different model. If you
already ran the Qwen setup, nothing here conflicts with it, but the two cannot
run at the same time because either one wants most of the unified memory.

Tested on a GB10 with 128 GB unified memory. Everything below is pinned by
digest or revision, so you should get the same bits I did.

## What you need

- A DGX Spark or GX10 class box, GB10, 128 GB unified memory
- Docker with the NVIDIA runtime
- About 190 GB of free disk. The weights alone are 185 GB on disk and the
  loaded footprint is roughly 95 GB.
- No Hugging Face token is required for the public image, but the compose file
  passes one through if you have it set.

## The files

Two files in a directory of your choosing. The first is the upstream recipe,
untouched. The second holds every local change, so an upstream update never
silently reverts your fixes.

`compose.yaml`:

```yaml
services:
  deepseek-v4-flash:
    image: ghcr.io/0xsero/deepseek-v4-flash-0731-spark-sparkinfer@sha256:2e077489a83a0360952828051fe7f7a32c1801e5ce8436d85f7267583d614ff4
    pull_policy: always
    restart: unless-stopped
    network_mode: host
    ipc: host
    shm_size: 16gb
    environment:
      HF_TOKEN: ${HF_TOKEN:-}
      MODEL_REPO: 0xSero/deepseek-v4-flash-0731-spark
      MODEL_REVISION: 22f28d32b9b29b4352eaa380ff8c2c170b2847ab
      MAX_MODEL_LEN: "262144"
      MAX_NUM_SEQS: "4"
      MAX_NUM_BATCHED_TOKENS: "8224"
      MODE: dspark
      DSPARK_TOKENS: "5"
      DSPARK_CAPACITY: "0"
      DSPARK_DYNAMIC_DRAFT_DEPTH: "0"
      DSPARK_DYNAMIC_DRAFT_DEPTH_WINDOW: "8"
      DSPARK_DRAFT_EXPERTS: "64"
      DSPARK_STRUCTURED_EXPERTS_PER_CATEGORY: "32"
      VLLM_USE_B12X_WO_PROJECTION: "1"
      GPU_MEMORY_UTILIZATION: "0.9465"
      VERIFY_MODEL_CHECKSUMS: "1"
    volumes:
      - ./data:/models
      - ./cache:/cache
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
```

`compose.override.yaml`:

```yaml
services:
  deepseek-v4-flash:
    environment:
      SERVED_MODEL_NAME: agent
      MAX_MODEL_LEN: "196608"
      MAX_NUM_SEQS: "8"
    image: ds4-xgrammar-fix:local
    pull_policy: never
```

`Dockerfile.xgrammar-fix`:

```dockerfile
FROM ghcr.io/0xsero/deepseek-v4-flash-0731-spark-sparkinfer@sha256:2e077489a83a0360952828051fe7f7a32c1801e5ce8436d85f7267583d614ff4
RUN pip install --no-cache-dir --no-deps xgrammar==0.2.2 \
 && python -c "from xgrammar import StructuralTag, normalize_tool_choice; print('xgrammar ok')"
```

## Why each override exists

Do not skip these. Each one is a real failure I hit.

**xgrammar 0.2.2.** The published image ships xgrammar 0.1.27, and its vLLM
does an unconditional top level import of `normalize_tool_choice`, which only
exists from 0.2.2. Plain chat works fine, so the box looks healthy. Every
request carrying `tools` returns a 500. Because the import is at module level,
no vLLM flag or structured output backend avoids it. The dependency has to be
corrected, which is what the one line Dockerfile does. `--no-deps` is
deliberate: you are correcting one under pinned transitive dependency, not
inviting pip to re-resolve a carefully built CUDA environment.

**`pull_policy: never`.** The base recipe sets `always`. Without this, compose
tries to pull your local only tag and fails.

**`SERVED_MODEL_NAME: agent`.** Only if you front several backends behind one
name. vLLM validates the `model` field on every request, so if your router
sends `agent` and the container serves `deepseek-v4-flash-0731-spark`, every
request 404s. Set it to whatever your clients already ask for, or drop this
line if you talk to the box directly.

**`MAX_MODEL_LEN: 196608`.** The upstream 262144 is a coin flip on this
hardware. Its startup check wants 7.93 GiB and the box yields somewhere between
7.66 and 7.98 GiB depending on the boot, so the same command succeeds or fails
across a reboot with nothing changed. 196608 clears every time.

**`MAX_NUM_SEQS: 8`.** The scheduler cap, not the KV cache, is what bounds
concurrency here. Four leaves throughput on the table.

## Build and start

```bash
docker build -f Dockerfile.xgrammar-fix -t ds4-xgrammar-fix:local .
docker compose up -d
```

First start downloads and verifies 185 GB, so give it time. Watch it with
`docker compose logs -f`.

## Verify, in this order

Chat working does not mean the box is working. Check all three.

```bash
# 1. It is serving, and under the name you expect
curl -s http://localhost:8000/v1/models | jq '.data[0].id'

# 2. Structured output
curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
 "model":"agent","messages":[{"role":"user","content":"Give me 17 times 19 as JSON."}],
 "response_format":{"type":"json_schema","json_schema":{"name":"r","schema":{"type":"object",
   "properties":{"answer":{"type":"integer"}},"required":["answer"]}}},"max_tokens":64}' \
 | jq -r '.choices[0].message.content'
# expect {"answer": 323}

# 3. Tool calling. This is the one the xgrammar bug breaks.
curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
 "model":"agent","messages":[{"role":"user","content":"What is the weather in Atlanta? Use the tool."}],
 "tools":[{"type":"function","function":{"name":"get_weather","description":"Get weather",
   "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
 "tool_choice":"auto","max_tokens":128}' | jq '.choices[0].message.tool_calls'
# expect a get_weather call, not a 500
```

## What to expect

Aggregate decode throughput measured on my box, 256 token completions across
different coding prompts:

| Requests at once | tok/s aggregate |
|---:|---:|
| 1 | 36.8 |
| 4 | 66.6 |
| 8 | 104.2 |
| 16 | 105.8 |

It flattens after 8 because `MAX_NUM_SEQS` is 8. Requests past that queue
rather than fail.

Single stream is slower than the Qwen setups on the same box. That is the
trade. You are running a much larger model in the same 128 GB.

## The trap to remember

Every local change lives in `compose.override.yaml` and the one line
Dockerfile, and nothing else. That is on purpose. **A recipe update will
quietly put you back on the upstream image and the upstream model name, and the
first symptom is that tool calling breaks while ordinary chat keeps working.**
If tools start returning 500s after an update, check which image the container
is actually running before you debug anything else:

```bash
docker inspect deepseek-v4-flash --format '{{.Config.Image}}'
# must be ds4-xgrammar-fix:local
```