Learning Panorama-Aware VLA for Mobile Manipulation with Whole-Body Teleoperation

Donglin Yang, Haoran Chen, Xingyu Chen, Lixing Liu, Manyi Li, Changhe Tu, Ke Xu, Xiaojian Ma, Si LiuarXiv 2026 · arXiv

TL;DR

PanoVLA introduces a panorama-aware Vision-Language-Action (VLA) policy and a whole-body teleoperation system designed specifically for mobile bimanual robots. By integrating 360-degree global spatial context with a specialized panoramic encoder, the system overcomes the "tunnel vision" of traditional local-view cameras, boosting end-to-end task success rates from 30.0% to 73.4% in complex, multi-stage real-world environments.

What Problem Does This Paper Address?

Most state-of-the-art Vision-Language-Action (VLA) models excel at tabletop manipulation but struggle when transitioned to mobile robots. The authors identify two core bottlenecks hindering this transition:

  1. The Data Challenge (Coordinated Teleoperation): Collecting high-quality demonstrations for a mobile bimanual robot is notoriously difficult. Operators typically have to juggle separate control interfaces—like a joystick for the wheels and controllers for the arms—leading to disjointed, unnatural movements.
  2. The Perception Challenge (Limited Field of View): Standard VLA models rely heavily on local observations from front-facing or wrist-mounted cameras. During long-horizon mobile tasks, turning the robot base or moving its arms frequently causes target objects or landmarks to disappear from view or become occluded, leading to task failure.

While adding more cameras seems like an obvious fix, simply stacking multiple image streams doesn't give a robot a coherent, scene-wide spatial understanding. Furthermore, standard vision encoders fail to interpret the non-perspective distortions inherent in 360-degree panoramic images.

Methods

To solve these challenges, the authors developed a dual-pronged approach consisting of a novel teleoperation framework and a specialized model architecture.

1. Whole-Body Teleoperation via GMR

The data collection system uses a single wearable VR setup (headset, two hand controllers, and three sparse body trackers). Instead of controlling the base and arms separately, the system uses a General Motion Retargeting (GMR) formulation. It models the mobile chassis as a "planar virtual floating base" and jointly optimizes both base and arm commands from the operator's natural body movements. A cost-driven mode-switching strategy then translates these virtual commands into physical wheel velocities.

Overview of the whole-body teleoperation pipeline

2. PanoVLA Architecture

PanoVLA is built on a Mixture-of-Transformers (MoT) architecture featuring three distinct expert blocks:

  • VLM Expert: A pretrained PaliGemma model that processes local views, text instructions, and robot proprioceptive states.
  • Panorama Expert: The core novelty. It takes 360-degree dual-fisheye images, projects them into an Equirectangular Projection (ERP) image, and processes them using MTPano—a panoramic foundation model that captures both semantic and geometric (depth) features despite panoramic distortions. These features are tokenized and fused with the VLM tokens via joint self-attention.
  • Action Expert: Predicts continuous action chunks using conditional flow matching.
Fig. 2. Overview of the proposed PanoVLA framework, which incorporates panoramic context into a vision-language-action model for mobile manipulation.

Datasets

The authors introduced a new real-world mobile bimanual manipulation dataset. It consists of 800 successful teleoperated trajectories totaling roughly 5.5 hours of multimodal demonstrations (recorded at 25 Hz, yielding ~500k synchronized frames).

The dataset spans four distinct real-world tasks designed to test spatial layout variations and long-horizon tracking:

  • Move Pen: Single-target localization and short-horizon pick-and-place.
  • Move Block: Transferring blocks across different sides of the workspace.
  • Open Curtain: Sequential interaction requiring the robot to turn and track states on both its left and right sides.
  • Wipe Table: Long-horizon manipulation spanning a large table surface where the starting point goes completely out of local view.

Key Insights and Ideas

  • Geometry-Aware Pretraining Matters: Naively feeding a raw panoramic image into a standard vision encoder (like SigLIP) yields poor results because these encoders are trained on standard perspective images. Utilizing a dedicated panoramic foundation model (MTPano) that inherently understands panoramic geometry is crucial for robust feature extraction.
  • Tracking State Over Long Horizons: The biggest advantage of panoramic awareness isn't just finding an object initially; it's remembering where things are after the robot has turned away. In tasks like Wipe Table, local-view models fail during the final phase because they lose track of the original starting position. The panoramic context preserves global spatial consistency.
  • Unified Whole-Body Control: Jointly optimizing the base and arms as a single floating-base system makes teleoperation fluid enough to collect high-quality data efficiently, proving that hardware-agnostic VR tracking is viable for complex mobile platforms.

Key Results

PanoVLA was evaluated in closed-loop, real-world trials against several baselines, including a standard local-view model ($\pi_{0.5}$) and variants that naively appended panoramic views.

  • Overall Performance: PanoVLA achieved an average Stage Completion Rate (SCR) of 91.3% and an end-to-end Success Rate (SR) of 73.4%, vastly outperforming the standard local-view baseline (58.6% SCR / 30.0% SR).
  • Long-Horizon Resilience: On the grueling Wipe Table task, the raw panorama baseline's success rate plummeted to 20.0% at the final stage due to a loss of spatial consistency. PanoVLA maintained a 46.7% success rate, demonstrating the effectiveness of its dedicated Panorama Expert module.
Stage-wise performance on Wipe Table task

Limitations and Future Work

While PanoVLA marks a significant step forward, the authors note a few limitations:

  • Data Efficiency: The policy still requires task-specific fine-tuning on the collected dataset and does not yet achieve zero-shot generalization to entirely unseen tasks.
  • Hardware Constraints: The system relies on a top-mounted panoramic camera, which can still experience specific occlusions or height-based blind spots depending on the environment layout.
  • Fixed Action Horizons: The action expert relies on a fixed 32-step action chunking horizon, which might not be optimal for tasks requiring highly dynamic or variable-speed adjustments.

Future directions include scaling up the dataset diversity to explore open-world zero-shot transfer capabilities.

Summary updated

Abstract

Mobile manipulation is a key capability for embodied intelligence, enabling robots to accomplish complex multi-stage tasks in open-world environments. However, mobile manipulation poses two key challenges for vision-language-action (VLA) policies: At the data level, the efficient collection of high-quality whole-body demonstrations demands the coordinated control of both the mobile base and the robotic arms; at the model level, existing VLA models predominantly rely on local camera observations, whose limited field of view hinders global spatial understanding. To address these challenges, we develop a whole-body teleoperation system and a panoramic-aware VLA policy. The system enables coordinated control of a wheeled bimanual robot through a single VR interface and supports the acquisition of a real-world mobile manipulation dataset comprising 5.5 hours of multimodal demonstrations. Building upon this dataset, we propose PanoVLA, a panorama-aware vision-language-action policy for mobile bimanual manipulation. Built upon a Mixture-of-Transformers architecture, PanoVLA introduces global spatial context through dedicated panorama encoding and fusion modules, enabling effective integration of panoramic observations with language instructions and robot states for action generation. Evaluation on four real-world mobile manipulation tasks demonstrates that PanoVLA achieves an average stage completion rate of 91.3\% and an end-to-end success rate of 73.4\%, substantially outperforming local-view baselines. These results demonstrate that incorporating panoramic spatial context improves spatial understanding and closed-loop manipulation performance in mobile robots.
BibTeX
@misc{yang2026learning,
  author = {Donglin Yang and Haoran Chen and Xingyu Chen and Lixing Liu and Manyi Li and Changhe Tu and Ke Xu and Xiaojian Ma and Si Liu},
  title = {{Learning Panorama-Aware VLA for Mobile Manipulation with Whole-Body Teleoperation}},
  year = {2026},
  eprint = {2608.02257},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.02257},
}
Figure