Benchmarking LLM Inference on Desktop-Scale NVIDIA Grace Blackwell Clusters

Kevin Bhimani, Christian Escobar, Daniel Mihovch, Rudolf Eigenmann, Sunita ChandrasekaranarXiv 2026 · arXiv

TL;DR

This paper provides a comprehensive performance guide for running Large Language Models (LLMs) on "desktop-scale" clusters—specifically two NVIDIA DGX Spark (GB10) systems connected via 200 GbE. The authors demonstrate that, contrary to traditional datacenter wisdom, Tensor Parallelism (TP) consistently outperforms Pipeline Parallelism (PP) for inter-node scaling on this hardware, provided that low-latency RDMA (RoCEv2) is used instead of standard TCP.

What Problem Does This Paper Address?

As organizations seek to run LLMs locally for privacy and cost reasons, "desktop-scale" GPU systems like the NVIDIA DGX Spark are becoming popular. These systems use the Grace Blackwell (GB10) architecture, which features a unified memory pool (LPDDR5x) shared between the CPU and GPU, rather than the dedicated High-Bandwidth Memory (HBM) found in flagship datacenter GPUs.

When a model is too large for one system, or when you want to increase throughput, you must distribute it across multiple nodes. The two standard strategies are:

  1. Tensor Parallelism (TP): Shards individual layers across GPUs; requires frequent communication during every decoding step.
  2. Pipeline Parallelism (PP): Assigns different layers to different GPUs; requires communication only at the boundaries between stages.

Conventional wisdom suggests using PP across nodes to minimize communication overhead. However, the DGX Spark lacks NVLink (the ultra-fast intra-server interconnect) and relies on 200 GbE. The authors set out to determine the optimal deployment strategy for this specific hardware profile.

Methods

The researchers conducted a massive benchmarking campaign using vLLM, an industry-standard LLM serving framework. They evaluated:

  • Parallelism Strategies: TP=2, PP=2, and a "two-replica" baseline (running two independent instances of the model).
  • Hardware Setup: Two ASUS Ascent GX10 systems (DGX Spark design) connected directly via ConnectX-7 ports using RDMA over Converged Ethernet (RoCEv2).
  • Workload: A mix of input/output lengths ranging from 128 to 8,192 tokens, tested at concurrency levels from 1 to 64.
  • Metrics: Beyond raw throughput (tokens/sec), they measured Inter-Token Latency (ITL), Time to First Token (TTFT), and Goodput (the rate of requests that meet specific latency targets).

Datasets (Models)

The study evaluated 17 open-weight checkpoints spanning a wide range of architectures and sizes (8B to 235B parameters):

  • Dense Models: Llama-3.1 (8B), Qwen-3.6 (27B), Gemma-4 (31B).
  • Mixture-of-Experts (MoE): Qwen-3.6-35B-A3B, Nemotron-3.5-Lightning, gpt-oss-120b, Mistral-Medium, and Qwen-3-235B.
  • Precisions: The benchmarks compared BF16, FP8, and the new 4-bit NVFP4 and MXFP4 formats.

Key Insights and Ideas

  • TP Wins Inter-Node: In 93 out of 96 comparisons, TP=2 delivered higher throughput than PP=2. Because the GB10 is memory-bandwidth bound, TP allows both GPUs to read their respective weight partitions concurrently, effectively doubling the available bandwidth. With RDMA, the communication overhead of TP is low enough that this "parallel read" benefit wins out.
  • The "Quantization First" Rule: Reducing model precision (e.g., from BF16 to NVFP4) on a single node improved latency more significantly than adding a second node at higher precision.
  • Replication vs. Splitting: If a model fits on one Spark, the best strategy depends on load. Split the model (TP=2) if you care about low latency or have few users. Replicate the model (two independent 1x instances) if you have high concurrency and want maximum total throughput.
  • Energy Penalty: Scaling to two nodes is never "free." A single Spark always uses less GPU energy per token than a two-node setup at the same concurrency. TP=2 is slightly more energy-efficient than PP=2 because it finishes the work faster.

Key Results

  • Throughput: TP=2 achieved a median 1.59x peak-throughput gain over a single node, while PP=2 only managed 1.39x.
  • Latency: TP=2 reduced unloaded inter-token latency to 0.52–0.73x of the single-node baseline. In contrast, PP=2 provided almost no latency reduction for single requests.
  • RDMA is Mandatory: Switching from TCP to RDMA was the "reversal factor." Under TCP, the communication overhead was so high that PP was preferred; under RDMA, TP became the clear winner.
  • Model Capacity: Using two nodes increased KV-cache capacity by 2.09–4.38x, allowing for much longer context windows or higher concurrency than a single node could handle.

Limitations and Future Work

  • Scale: The study was limited to a two-node cluster. The trade-offs might shift again as the number of nodes increases beyond two.
  • Energy Scope: The energy measurements only included GPU power, excluding the CPU, networking gear, and power supplies.
  • Software Versioning: The results are specific to the vLLM version and kernels used (e.g., the Marlin kernel for NVFP4). As software optimizations evolve, the performance gaps may change.
  • Future Work: The authors suggest exploring "chunked prefill" optimizations and more complex hybrid parallelism strategies for even larger clusters.

Summary updated

Abstract

Two NVIDIA DGX Spark GB10 systems connect directly over 200 GbE to serve larger language models and increase inference capacity, but the benefits depend on how they are deployed. We benchmark 17 open-weight checkpoints spanning dense and mixture-of-experts architectures, 8B to 235B parameters and four weight precisions on one and two RDMA-connected systems. Using vLLM, we compare tensor parallelism (TP=2), pipeline parallelism (PP=2) and a two-replica baseline at concurrency 1 to 64, with three independent launches per configuration. TP=2 delivers a median 1.59x peak-throughput gain over one Spark, versus 1.39x for PP=2 and has higher throughput in 93 of 96 matched comparisons with separated 95% Welch intervals. It reduces unloaded inter-token latency to 0.52x to 0.73x of one Spark, whereas PP=2 provides little reduction. TP=2 also achieves the highest goodput under a strict 100 ms inter-token latency objective wherever both strategies are supported and the objective is met. For models fitting on one Spark, TP=2 generally favors low-concurrency serving, while replication offers higher throughput at high concurrency. One Spark uses less GPU energy per output token at matched concurrency. Across three precision ladders, NVFP4 reduces unloaded latency more than adding a second Spark at BF16. NCCL microbenchmarks and comparisons with an earlier TCP campaign suggest that collective communication overhead contributes to the advantage of TP=2. These results provide deployment guidance for desktop-scale inference.
BibTeX
@misc{bhimani2026benchmarking,
  author = {Kevin Bhimani and Christian Escobar and Daniel Mihovch and Rudolf Eigenmann and Sunita Chandrasekaran},
  title = {{Benchmarking LLM Inference on Desktop-Scale NVIDIA Grace Blackwell Clusters}},
  year = {2026},
  eprint = {2611.20289},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20289},
}
Figure