Red Hat pairs FP8 and DSpark to double Nemotron 3.5 Super VL throughput

Red Hat AI has released two optimized checkpoints for NVIDIA’s 120-billion-parameter Nemotron 3.5 Super VL multimodal mixture-of-experts model: an FP8-quantized target model and a DSpark speculative-decoding model. Red Hat says the combination can double inference throughput on NVIDIA Hopper GPUs while preserving output accuracy.

What changed

The FP8 checkpoint reduces weights and activations from BF16 to 8-bit floating point. Red Hat says that halves model-weight memory and storage requirements, while Hopper’s native FP8 Tensor Cores offer roughly twice the theoretical matrix-multiplication throughput of BF16.

The implementation is selective rather than blanket quantization. Red Hat quantized attention, expert and Mamba projection layers in the main model, but retained original precision for shared experts, routers, normalization layers and the output head. It used the vLLM project’s LLM Compressor workflow and published a reproducible creation script with the model card.

Red Hat evaluated BF16 and FP8 variants under the same conditions—four NVIDIA H200 GPUs, vLLM 0.31.0 and the Inspect AI and Harbor evaluation frameworks—and reports preserved accuracy across seven reasoning and agentic benchmarks.

Why DSpark matters

DSpark attacks a different bottleneck. Instead of proposing draft tokens one at a time, it drafts eight to 16 tokens in one forward pass and uses a lightweight Markov head to restore dependencies between those parallel proposals. The target model then verifies the candidates together.

Red Hat’s four-H100 tests separate the effects: DSpark primarily improves single-user streaming speed, while FP8 primarily raises aggregate throughput under concurrency. Combining both configurations outperformed either alone at every tested concurrency level across HumanEval, mathematical reasoning and retrieval-augmented generation prompts.

Who should care

Inference teams running large multimodal or long-running agent workloads on Hopper-class GPUs now have a concrete tuning path rather than a generic claim about quantization. The useful operational distinction is workload-shaped: DSpark targets interactivity, FP8 targets loaded-system throughput, and the two can be composed.

The checkpoints are available through Red Hat Inference Hub on Red Hat AI Factory with NVIDIA. Teams should reproduce the published benchmark setup against their own prompt lengths, concurrency and quality thresholds before treating the reported gains as a capacity-planning constant.