Self-Calibrating Vision-Agent-Action: Deploying Pick-and-Place to New Robot Cells without Demonstrations or Fine-Tuning

Jae Hun Hwang, Yechan Seo, Hong-In WonarXiv 2026 · arXiv

TL;DR

Vision-Agent-Action (VAA) is a modular framework that allows robots to perform pick-and-place tasks in entirely new environments without needing new demonstrations or fine-tuning. By having the robot "self-calibrate"—observing its own motion to map commands and camera geometry—it achieves a 74% success rate in changed cells where state-of-the-art end-to-end models (VLAs) struggle to reach 21%.

What Problem Does This Paper Address?

Deploying a trained robot to a new "cell" (a different factory station or home setup) is notoriously difficult. Even if the task remains the same, physical changes—like a different gripper, a rotated base frame, or a new camera angle—often break end-to-end Vision-Language-Action (VLA) models.

Currently, the standard solution is to collect new demonstrations and fine-tune the model for every specific setup. This is a massive bottleneck for scaling robotics. The authors identify that gripper and base-frame changes are the primary "VLA killers," as they fundamentally alter the relationship between a digital command and the resulting physical motion.

Methods

VAA moves away from the "black box" end-to-end approach in favor of a modular, self-calibrating pipeline. It separates semantic interpretation (what to do) from geometric execution (how to move).

  1. Self-Calibration (Once per cell): Before starting a task, the robot performs a series of "probes." It moves its arm to measure the command map (how control signals translate to Tool Center Point displacement). It observes its own fingertips to estimate camera intrinsics, pose, and lens distortion.
  2. Semantic Grounding: It uses pre-trained, off-the-shelf models—specifically Qwen-27B (VLM) for object labeling and Qwen-30B (LLM) for instruction interpretation—to identify targets and destinations.
  3. Geometric Execution: Instead of predicting raw actions from pixels, VAA uses measured 3D depth to plan grasps. It maps the semantic choices from the VLM onto the measured 3D scene geometry to execute precise movements.

Datasets

The study is conducted in simulation using the LIBERO benchmark suite:

  • LIBERO-spatial: Used for the primary controlled study (10 tasks, 20 episodes per cell).
  • LIBERO-Goal: Used to test two-step manipulation chains (e.g., "put the bowl on the stove, then move it to the cabinet").
  • Controlled Design Cells: The authors created 30 unique cell variations, systematically changing arms (Panda, UR5e, RB5), grippers (Hand-E, PGC, 2FG7), camera poses, lighting, and coordinate frames.

Key Insights and Ideas

  • The Geometry Gap: The "aha moment" of the paper is the diagnosis that VLAs fail not because they don't recognize objects, but because they can't handle "out-of-distribution" physical geometry. A 45-degree rotation of the robot base is enough to make most end-to-end models fail completely.
  • Self-Modeling as a Solution: By using its own body as a calibration target, the robot can build an internal model of its new environment autonomously. This removes the need for human-labeled data or specialized calibration rigs.
  • Depth Matters: While many VLAs rely solely on RGB images, VAA demonstrates that using measured depth for grasp planning is a critical factor in successfully transferring to new hardware.
  • Modularity Wins in Transfer: While end-to-end models are powerful in static environments, modular systems that combine high-level "reasoners" (LLMs/VLMs) with low-level "geometers" are far more robust to physical changes.

Key Results

  • Superior Transfer: In cells with changed grippers or rotated frames, the "frozen" VAA achieved a 64% success rate, and a refined version reached 74%. In contrast, the strongest public VLA (X-VLA) managed only 21% on the same subset.
  • Chain Success: On two-step tasks (LIBERO-Goal), VAA completed 54 out of 60 chains. All four evaluated VLA baselines failed to complete a single chain in the changed cells.
  • Ablation Impact: Reusing the calibration from the training setup (simulating a lack of self-calibration) caused VAA's success rate to plummet from 83% to 37%, highlighting that the measurement phase is the "secret sauce."

Limitations and Future Work

  • Simulation vs. Reality: The study was performed entirely in simulation. While the physics are complex, real-world sensor noise and lighting might pose additional challenges for the self-calibration modules.
  • Parallel-Jaw Assumption: VAA currently assumes a parallel-jaw gripper model. It failed on grippers with non-linear closing paths (like the Robotiq 2F-85), suggesting a need for more complex gripper-specific motion models.
  • Task Complexity: The evaluation focused on pick-and-place. Future work could extend this to more nuanced manipulation tasks like pouring, wiping, or using tools, which require more than just 3D point-to-point planning.

Summary updated

Abstract

In simulation, Vision-Agent-Action (VAA) performs pick-and-place in robot cells with gripper or base-frame changes without robot demonstrations or fine-tuning. It uses pre-trained vision-language and language models for interpretation, measures the command map and camera geometry from its own motion and fingertips, estimates lens distortion, and plans grasps from measured object depth. A controlled study of eight public LIBERO-fine-tuned vision-language-action models (VLAs) varies robot hardware, camera geometry, appearance, and base frame. Gripper and frame changes dominate their losses in the tested fractional design. Across 18 cells with these changes, frozen VAA achieves 230/360 successes (64%), the refined VAA 268/360 (74%), and the strongest released VLA on this subset 76/360 (21%). Paired ablations support the self-calibration bundle, lens estimation under compounded changes, and the measured-depth pipeline. On a two-step LIBERO-Goal chain with object state carried between instructions, VAA completes 54/60 chains in changed cells, while four evaluated VLAs complete none.
BibTeX
@misc{hwang2026self,
  author = {Jae Hun Hwang and Yechan Seo and Hong-In Won},
  title = {{Self-Calibrating Vision-Agent-Action: Deploying Pick-and-Place to New Robot Cells without Demonstrations or Fine-Tuning}},
  year = {2026},
  eprint = {2611.19198},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19198},
}
Figure