TL;DR
This paper provides the first comprehensive cross-platform scaling analysis of Vision-Language-Action (VLA) models for robotics, evaluating performance across both edge and datacenter GPUs. The authors demonstrate that architectural optimizations like chunked decoding allow modern edge devices (like the NVIDIA Jetson AGX Orin) to achieve high-frequency inference that can actually outperform older datacenter hardware (like the NVIDIA V100) without sacrificing task accuracy.
What Problem Does This Paper Address?
Vision-Language-Action (VLA) models have emerged as powerful generalist policies for robotic control, enabling robots to translate visual inputs and natural language instructions directly into motor actions. However, deploying these large models in real-world robotics introduces a massive challenge: the tension between computational demands and hardware constraints.
While algorithmic variants are proliferating, we lack a systematic understanding of how these models scale across different hardware tiers (from low-power edge boards on mobile robots to massive cloud clusters) and power budgets. Most existing literature evaluates models on a single, fixed hardware setup. This paper fills that gap by profiling accuracy, latency, throughput, and memory usage across the entire edge-to-cloud spectrum, helping engineers avoid over-provisioning hardware or making suboptimal deployment trade-offs.
Methods
The authors conduct a rigorous empirical evaluation of five representative VLA architectures on two distinct hardware classes:
- Edge Platform: NVIDIA Jetson AGX Orin, tested across multiple power modes (15W, 30W, 50W, and MAX).
- Datacenter GPUs: NVIDIA H100, A100, A6000, and V100.
The evaluated models include three established baselines—OpenVLA, SpatialVLA, and OpenVLA-OFT—alongside two architectures developed/adapted by the authors:
- VOTE: An architecture optimized to generate fewer action tokens using a special token (ST)-based action head, evaluated in three configurations (VOTE-1T, VOTE-2T, and VOTE-MLP4) to test different output granularities and chunk sizes.
- QwenVLA: A newly proposed model designed to explore smaller language backbones, substituting the typical 7B parameter LLM with a compact Qwen 2.5-1.5B backbone while maintaining the same vision stack.
Datasets
The models were evaluated using the LIBERO benchmark, a standard suite for robot manipulation consisting of four distinct task categories:
- LIBERO-Spatial: Evaluates understanding of spatial relationships.
- LIBERO-Object: Tests performance across various object types.
- LIBERO-Goal: Measures task-oriented behaviors.
- LIBERO-Long: Challenges the model's ability to generalize to long-horizon tasks with diverse layouts.
For training and fine-tuning context, the baselines utilize the Open X-Embodiment (OXE) dataset, while QwenVLA was pre-trained on the LLaVA v1.5 data mix before being fine-tuned on LIBERO using Optimized Fine-Tuning (OFT) and Low-Rank Adaptation (LoRA).
Key Insights and Ideas
- Action Heads Don't Bloat Memory; Backbones Do: A key architectural finding is that altering the action head (e.g., changing token counts or MLP depth) has a negligible impact on peak VRAM. Memory footprint is almost entirely driven by the size of the LLM backbone and the choice of vision encoder.
- The Edge Can Beat the Cloud (Sometimes): In a surprising twist, an optimized model (VOTE-MLP4) running on an edge device in MAX power mode achieved higher throughput than the same model running on an older datacenter GPU (V100). This challenges the blanket assumption that cloud deployment is always superior for heavy robotic inference.
- Non-Linear Power Scaling at the Edge: Reducing power budgets on the Jetson Orin results in non-linear performance drops. Compute-heavy models suffer severe throughput losses when dropping from 50W to 30W, whereas smaller models like QwenVLA retain a much higher fraction of their performance, making them highly resilient choices for strictly power-constrained environments.
Key Results
- Accuracy: The VOTE-1T variant achieved the highest overall success rate on the LIBERO benchmark at 96.9%, outperforming the strongest baseline, OpenVLA-OFT (95.3%). Meanwhile, the lightweight QwenVLA (78.8% success rate) managed to outperform the standard, much larger OpenVLA baseline (76.5%).
- Memory Footprint: QwenVLA proved to be the most lightweight, requiring only 7.39 GB of peak VRAM, compared to OpenVLA-OFT's demanding 19.20 GB.
- Latency & Throughput: On the premier H100 datacenter GPU, VOTE-MLP4 reached an astonishing throughput of 474.78 Hz (over 64× faster than standard OpenVLA). On the edge-based Jetson Orin (MAX mode), VOTE-MLP4 clocked in at 55.57 Hz, successfully outperforming the datacenter V100's 32.28 Hz.
Limitations and Future Work
While this work provides an excellent blueprint for VLA hardware selection, the authors note a few areas for expansion. The current evaluation relies on simulated environments via the LIBERO benchmark; future work will need to validate these scaling trends in real-world robotic deployments where physical latencies interact with computational ones. Additionally, the authors plan to investigate the impact of model quantization strategies (e.g., INT8/FP4 precision) to see how aggressive compression alters the accuracy-efficiency trade-off on edge hardware.
Summary updated