TL;DR
PanoVLA introduces a panorama-aware Vision-Language-Action (VLA) policy and a whole-body teleoperation system designed specifically for mobile bimanual robots. By integrating 360-degree global spatial context with a specialized panoramic encoder, the system overcomes the "tunnel vision" of traditional local-view cameras, boosting end-to-end task success rates from 30.0% to 73.4% in complex, multi-stage real-world environments.
What Problem Does This Paper Address?
Most state-of-the-art Vision-Language-Action (VLA) models excel at tabletop manipulation but struggle when transitioned to mobile robots. The authors identify two core bottlenecks hindering this transition:
- The Data Challenge (Coordinated Teleoperation): Collecting high-quality demonstrations for a mobile bimanual robot is notoriously difficult. Operators typically have to juggle separate control interfaces—like a joystick for the wheels and controllers for the arms—leading to disjointed, unnatural movements.
- The Perception Challenge (Limited Field of View): Standard VLA models rely heavily on local observations from front-facing or wrist-mounted cameras. During long-horizon mobile tasks, turning the robot base or moving its arms frequently causes target objects or landmarks to disappear from view or become occluded, leading to task failure.
While adding more cameras seems like an obvious fix, simply stacking multiple image streams doesn't give a robot a coherent, scene-wide spatial understanding. Furthermore, standard vision encoders fail to interpret the non-perspective distortions inherent in 360-degree panoramic images.
Methods
To solve these challenges, the authors developed a dual-pronged approach consisting of a novel teleoperation framework and a specialized model architecture.
1. Whole-Body Teleoperation via GMR
The data collection system uses a single wearable VR setup (headset, two hand controllers, and three sparse body trackers). Instead of controlling the base and arms separately, the system uses a General Motion Retargeting (GMR) formulation. It models the mobile chassis as a "planar virtual floating base" and jointly optimizes both base and arm commands from the operator's natural body movements. A cost-driven mode-switching strategy then translates these virtual commands into physical wheel velocities.
2. PanoVLA Architecture
PanoVLA is built on a Mixture-of-Transformers (MoT) architecture featuring three distinct expert blocks:
- VLM Expert: A pretrained PaliGemma model that processes local views, text instructions, and robot proprioceptive states.
- Panorama Expert: The core novelty. It takes 360-degree dual-fisheye images, projects them into an Equirectangular Projection (ERP) image, and processes them using MTPano—a panoramic foundation model that captures both semantic and geometric (depth) features despite panoramic distortions. These features are tokenized and fused with the VLM tokens via joint self-attention.
- Action Expert: Predicts continuous action chunks using conditional flow matching.
Datasets
The authors introduced a new real-world mobile bimanual manipulation dataset. It consists of 800 successful teleoperated trajectories totaling roughly 5.5 hours of multimodal demonstrations (recorded at 25 Hz, yielding ~500k synchronized frames).
The dataset spans four distinct real-world tasks designed to test spatial layout variations and long-horizon tracking:
- Move Pen: Single-target localization and short-horizon pick-and-place.
- Move Block: Transferring blocks across different sides of the workspace.
- Open Curtain: Sequential interaction requiring the robot to turn and track states on both its left and right sides.
- Wipe Table: Long-horizon manipulation spanning a large table surface where the starting point goes completely out of local view.
Key Insights and Ideas
- Geometry-Aware Pretraining Matters: Naively feeding a raw panoramic image into a standard vision encoder (like SigLIP) yields poor results because these encoders are trained on standard perspective images. Utilizing a dedicated panoramic foundation model (MTPano) that inherently understands panoramic geometry is crucial for robust feature extraction.
- Tracking State Over Long Horizons: The biggest advantage of panoramic awareness isn't just finding an object initially; it's remembering where things are after the robot has turned away. In tasks like Wipe Table, local-view models fail during the final phase because they lose track of the original starting position. The panoramic context preserves global spatial consistency.
- Unified Whole-Body Control: Jointly optimizing the base and arms as a single floating-base system makes teleoperation fluid enough to collect high-quality data efficiently, proving that hardware-agnostic VR tracking is viable for complex mobile platforms.
Key Results
PanoVLA was evaluated in closed-loop, real-world trials against several baselines, including a standard local-view model ($\pi_{0.5}$) and variants that naively appended panoramic views.
- Overall Performance: PanoVLA achieved an average Stage Completion Rate (SCR) of 91.3% and an end-to-end Success Rate (SR) of 73.4%, vastly outperforming the standard local-view baseline (58.6% SCR / 30.0% SR).
- Long-Horizon Resilience: On the grueling Wipe Table task, the raw panorama baseline's success rate plummeted to 20.0% at the final stage due to a loss of spatial consistency. PanoVLA maintained a 46.7% success rate, demonstrating the effectiveness of its dedicated Panorama Expert module.
Limitations and Future Work
While PanoVLA marks a significant step forward, the authors note a few limitations:
- Data Efficiency: The policy still requires task-specific fine-tuning on the collected dataset and does not yet achieve zero-shot generalization to entirely unseen tasks.
- Hardware Constraints: The system relies on a top-mounted panoramic camera, which can still experience specific occlusions or height-based blind spots depending on the environment layout.
- Fixed Action Horizons: The action expert relies on a fixed 32-step action chunking horizon, which might not be optimal for tasks requiring highly dynamic or variable-speed adjustments.
Future directions include scaling up the dataset diversity to explore open-world zero-shot transfer capabilities.
Summary updated