Running Beam‑1.0 (501B) on workstation GPUs

The question

Reflection's vLLM recipe lists B200/B300 or 8× H100 for the NVFP4 checkpoint. Many teams that want to self-host for privacy or cost have RTX PRO 6000 cards (Blackwell workstation chip, SM120, 96 GB, PCIe, no NVLink) instead. Nobody had published whether Beam runs there, which flags it needs, or how fast and how good it is. So I measured it:

• Setup: does the official image work on SM120, and what breaks?
• Quality: does the official NVFP4 checkpoint behave as expected on this hardware?
• Speed and cost: single user, many users, with and without DFlash2.
• Smaller: can it go down to 2 GPUs, and what does that cost in quality?

Setup

What it took on SM120

docker run -d --name beam --gpus '"device=0,1,2,3"' --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  --entrypoint vllm vllm/vllm-openai:reflection-beam \
  serve reflection/Beam-1.0-501B-A23B-NVFP4 \
  --tensor-parallel-size 4 --max-model-len 65536 \
  --kv-cache-dtype fp8 --moe-backend marlin \
  --reasoning-parser beam --enable-auto-tool-choice --tool-call-parser beam \
  --speculative-config '{"method":"dflash","model":"reflection/Beam-1.0-501B-A23B-DFlash2",
    "num_speculative_tokens":7,"attention_backend":"FLASHINFER",
    "enable_adaptive_verification":false}'

Quality

Three models, same harness: the official NVFP4 on 4 GPUs, mrfakename's REAP version (half the experts removed) in BF16 on 8 GPUs, and my 4-bit build of that REAP model on 2 GPUs. MMLU-Pro uses 30 questions per subject (420 total); IFEval and HumanEval are complete.

The official NVFP4 checkpoint works on RTX PRO 6000: 0.843 on MMLU-Pro, 0.939 on IFEval, 158/164 on HumanEval. No sign of broken output from the Marlin weight-only path on SM120: 1 empty answer in 420 MMLU-Pro questions, 2 in 541 IFEval prompts.

The 4-bit step is within noise; the pruning step is not. Comparing answers question by question:

Exact two-sided McNemar test on discordant pairs. One sampled run each, so part of each difference is sampling noise.

Where the pruning hurts: REAP picked its experts on code and agent text. Math, physics and computer science stay within a few questions of the official model. Health, law, economics and philosophy each lose 7 to 9 questions out of 30. Coding (HumanEval) is 156 vs 158.

Speed

DFlash2 makes a single user 1.67× faster (100.5 → 167.7 tok/s), most on code (194.7 tok/s) where the draft model guesses well. The drafter's mean acceptance length was about 3.2–3.8 tokens, and 70–79% of first guesses were accepted. I didn't measure DFlash2 under many concurrent users, where speculative decoding usually helps less.

Context: with FP8 KV cache the 4-GPU server holds 957,317 tokens of KV cache (with DFlash2 loaded), 14.6 full 64k conversations at once. The 2-GPU build holds 553,601 (8.4×). Beam's sliding-window attention (44 of 52 layers use a 512-token window) keeps the cache small.

Cost

At full load the official model costs about $1.70 per million output tokens on rented RTX PRO 6000s. A single user alone on the box costs about 12× more per token, so self-hosting only pays when the server is shared. Per GPU, the 2-GPU build serves 1.7× more tokens, but only for work where its knowledge gap doesn't matter.

The 2-GPU coding build

mrfakename's REAP model removes 128 of each layer's 256 experts, keeping the 16 most-used per routing group on ~925k tokens of code/agent text. It was released with a perplexity check only. In BF16 it still needs 8 GPUs.

I quantized its experts to NVFP4 weights (BF16 activations) with llm-compressor's data-free model_free_ptq, same split as Reflection's official NVFP4: experts in 4-bit, attention, router, dense layers, embeddings and LM head in BF16. Result: 145 GiB (155 GB), fits on 2× RTX PRO 6000, 80 tok/s for one user, 989 tok/s at 64 users.

Use it when you only have 2 GPUs and the work is mostly code. For anything knowledge-heavy, the official model on 4 GPUs is clearly better (0.84 vs 0.70 MMLU-Pro). All pruning credit goes to mrfakename; Beam is Reflection's.

docker run -d --name beam-reap --gpus '"device=0,1"' --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  --entrypoint vllm vllm/vllm-openai:reflection-beam \
  serve peterhdd/Beam-1.0-REAP-255B-A23B-NVFP4A16 \
  --tensor-parallel-size 2 --max-model-len 65536 \
  --kv-cache-dtype fp8 --moe-backend marlin \
  --reasoning-parser beam --enable-auto-tool-choice --tool-call-parser beam

Limits

• One sampled run per benchmark; MMLU-Pro is a 420-question subset. Fine for comparing these three models, not for leaderboard claims.
• Not tested: long-context quality (runs used 64k max length), tool calling at scale, non-English, DFlash2 under concurrency, DFlash2 with the 2-GPU build.
• Speed uses random prompts at fixed lengths; real traffic with long prompts or long thinking will differ.
• Cost assumes a fully busy rented server and ignores setup time, storage and egress.

Reproduce

Every script, raw lm-eval sample file, HumanEval answer and benchmark log is in the dataset peterhdd/beam-rtx-pro-6000-runs: serve.sh, quick_eval.sh (MMLU-Pro + IFEval), humaneval_chat.py, speed.sh, real_speed.py, quantize_reap.py.