Towards Pose-Robust Few-Shot Vision-Language-Action Models

Wenhao Li, Zhongyi Han, Yuhao Wei, Liang Han, Yilong Yin, Liqiang NiearXiv 2026 · arXiv

TL;DR

WACO (World-Anchored Canonicalization) is a framework that makes few-shot Vision-Language-Action (VLA) models robust to "pose shifts"—like a bumped camera tripod or a slightly misaligned mobile robot base—without requiring new training data or ground-truth extrinsics. By using the robot’s own arm as a calibration target and the static scene as a "world anchor," WACO re-centers observations back to the training viewpoint, recovering nearly 90% of the performance lost to these common real-world disturbances.

What Problem Does This Paper Address?

The current "gold standard" for deploying a robot manipulation policy involves fine-tuning a pre-trained VLA model on a few dozen teleoperated demonstrations. While effective, this process creates a fragile dependency: the model "memorizes" the exact spatial relationship between the camera and the robot base.

In the real world, tripods get bumped, cameras are re-mounted after maintenance, and mobile bases rarely re-dock in the exact same millimeter-perfect spot. These small "pose shifts" (even just a few centimeters or degrees) cause standard VLA models to reach for the wrong place. Existing fixes usually involve either massive data augmentation (rendering thousands of synthetic viewpoints) or requiring perfect "ground-truth" camera calibration at test time—neither of which is practical for quick, few-shot deployment.

**Figure 1: Overview.** Standard VLA models fail when the camera or base moves because they've memorized the training setup. WACO estimates these shifts and "restores" the training view so the original policy can succeed.

Methods

The authors introduce WACO, which treats the robot itself as the solution. The framework consists of three main pillars:

  1. Two-Anchor Shift Estimation: To figure out how things moved, WACO uses two "anchors" from a single RGB-D frame.
  2. The Arm Anchor: Since the robot knows its own joint angles, it can project a 3D model of its arm into the camera's view. By matching this model to the actual pixels, it estimates the camera-to-base transform.
  3. The World Anchor: The arm alone isn't enough to tell if the camera moved or the robot base moved (a classic ambiguity). WACO compares the current static background to a reference photo of the empty workspace to lock down the camera's position in the world.
  4. Channel-Separated Canonicalization: Once the shifts are estimated, WACO "undoes" them. It warps the camera image back to the training viewpoint and mathematically rotates/translates the robot's internal state and actions to match the original base frame.
  5. Error-Aware Fine-Tuning: No estimator is perfect. WACO fine-tunes the VLA model using "round-trip" augmentations—intentionally adding the types of visual artifacts (like holes or blurry edges) that occur during image warping—so the model learns to be robust to the canonicalizer's own minor errors.
**Figure 2: Model Pipeline.** The system uses the arm and world anchors to estimate shifts, then applies canonicalization to the image and robot state before passing them to the unmodified VLA policy.

Datasets

The researchers evaluated WACO using:

  • LIBERO-PoseShift: A new benchmark introduced in this paper that adds graded camera, base, and compound shifts to the popular LIBERO manipulation suite.
  • LIBERO-Plus: A benchmark for extreme viewpoint changes (up to 75-degree orbits).
  • Real-World UR5e Tasks: Four physical tasks of increasing difficulty, such as "Toast into Toaster," which requires high precision.

Key Insights and Ideas

  • The Robot is its own Calibration Target: You don't need a checkerboard or a specialized calibration phase. The robot's arm, driven by forward kinematics, provides a metric "metric anchor" already present in every frame.
  • The $AX=YB$ Ambiguity: A key insight is that a camera shift and a base shift can look identical if you only look at the arm. You must use the static background (the world anchor) to distinguish between the two, as they require different action corrections.
  • Analytical vs. Learned Robustness: Instead of trying to teach a neural network to be "smart" enough to handle every possible viewpoint through brute-force data, WACO uses geometry to transform the problem back into a domain the network already understands.
**Figure 3: Two-Anchor Estimation.** (a) The arm anchor matches the robot model to depth data. (b) The world anchor matches the scene to a reference. (c) A consistency gate rejects physically impossible estimates.

Key Results

  • Simulation: On the LIBERO-PoseShift benchmark, WACO outperformed the standard few-shot recipe by 13.2 points, recovering 90% of the performance gap compared to an "oracle" that has perfect knowledge of the shifts.
  • Real Robot: In physical tests on a UR5e, WACO raised the success rate under pose shifts by a staggering 36.7 points (from 43.3% to 80.0%).
  • Data Efficiency: The advantage of WACO actually increases as you have fewer demonstrations. With only 10 demos, WACO achieved 82.4% success, significantly higher than the standard recipe's 63.2%.
  • Speed: The entire estimation and canonicalization process adds only about 770ms of overhead at the start of an episode and 11ms per control step, making it viable for real-time use.

Limitations and Future Work

  • Depth Dependency: The "Arm Anchor" currently requires an RGB-D camera to metrically match the robot model to the scene. Future work could explore purely RGB-based anchors.
  • Static Scene Assumption: The "World Anchor" relies on the background remaining relatively static. While the authors implemented an "arm-only fallback" for dynamic scenes, it is less robust to compound shifts.
  • Extreme Occlusions: If a large object blocks a significant portion of the static background, the world anchor's accuracy degrades, though the system's "consistency gate" is designed to detect and handle these failures gracefully.

Summary updated

Abstract

Fine-tuning a vision-language-action (VLA) model on a few dozen demonstrations is the standard way to put a manipulation policy on a robot, yet the resulting policy silently assumes that the camera and the robot base stay exactly where they were during data collection: a bumped tripod or a base re-docked a few centimetres away makes it reach for the wrong place. We show that both failures are one phenomenon, a change of the camera-to-base extrinsic that the policy has memorised, and that the robot carries everything needed to undo it. We introduce WACO, a world-anchored canonicalization framework for VLA models: the arm, placed by forward kinematics at the measured joint angles, serves as a metric calibration target, and the static scene serves as the world anchor that tells a moved camera from a moved robot. From a single RGB-D frame, WACO estimates both shifts without ground-truth extrinsics or extra data, canonicalises the image to the calibration viewpoint and the state, action and start pose to the calibration base frame, and fine-tunes the unmodified policy under the artefacts of its own estimator. We also introduce LIBERO-PoseShift, the first benchmark for RGB VLAs with graded camera, base and compound shifts. Across LIBERO-PoseShift, the LIBERO-Plus camera track and four real-world tasks on a UR5e robot, WACO consistently outperforms the standard few-shot recipe and prior viewpoint-robust methods, raising shifted success by 13.2 points in simulation and by 36.7 points on the real robot, recovering 90% of the simulation gap to a ground-truth oracle and 87% of the real-robot gap, and establishing a new state of the art among methods that use no perturbed training data, with no test-time calibration. Project page: https://peacelwh.github.io/WACO/.
BibTeX
@misc{li2026towards,
  author = {Wenhao Li and Zhongyi Han and Yuhao Wei and Liang Han and Yilong Yin and Liqiang Nie},
  title = {{Towards Pose-Robust Few-Shot Vision-Language-Action Models}},
  year = {2026},
  eprint = {2611.24360},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24360},
}
Figure