AuRA: From Atoms to Bits through Autonomous Robot Interaction

Xinchen Yu, Yoonkyo Jung, Haoran Chen, Zikui Cai, Aayush Talreja, Seungjae Lee, Hyunjin Kim, Furong HuangarXiv 2026 · arXiv

TL;DR

AuRA (Autonomous Real-to-sim Asset Reconstruction) is a new system that automates the creation of physics-informed digital twins for deformable objects. By replacing manual human manipulation with autonomous robot interaction and leveraging robot state data to bypass visual tracking errors, AuRA reduces human labor by 88.5% and achieves a 2.8× higher reconstruction yield compared to human-operated baselines.

What Problem Does This Paper Address?

Creating high-fidelity "digital twins"—virtual models that replicate a real object's geometry, appearance, and physical behavior—is essential for scaling robot learning in simulation. However, the current "real-to-sim" pipeline is bottlenecked by the "atoms loop": humans must manually manipulate objects, record the interactions, and often perform tedious manual annotations to track object motion.

Beyond the labor cost, human-driven data collection suffers from two technical hurdles:

  1. Visual Occlusion: The very act of manipulating an object (e.g., a hand or gripper grabbing a plushie) obscures parts of the object, leading to incomplete geometric and appearance models.
  2. Tracking Noise: Estimating the precise motion of a human hand or a visually tracked gripper is prone to errors in segmentation and depth lifting, which in turn biases the physical parameters (like stiffness) inferred during reconstruction.

Methods

AuRA introduces a three-stage autonomous pipeline to move from "atoms to bits" without human intervention beyond initial object placement.

The AuRA pipeline automates data acquisition, post-processing, and reconstruction to create digital twins.
  1. Autonomous Data Acquisition: The system uses multi-view RGB-D cameras to scan a stationary object. It generates grasp candidates using AnyGrasp and filters them for kinematic feasibility and workspace constraints. The robot then executes a grasp-and-lift interaction while recording synchronized video and robot joint states.
  2. Data Post-processing:
    • Object Tracking: AuRA uses Grounded-SAM2 for segmentation and MV-TAP for multi-view point tracking.
  3. Controller Trajectories: Instead of trying to visually track the gripper (which is often occluded), AuRA derives the gripper's path deterministically from the robot's internal joint states and calibrated geometry.
  4. Physics-Informed Reconstruction: The system fits a spring-mass model to the observed dynamics using the PhysTwin backend. To solve the occlusion problem, AuRA registers a "pre-interaction prior"—a high-quality scan of the object taken before the robot touched it—to the first frame of the interaction, filling in the gaps caused by the gripper.

Datasets

The authors evaluated AuRA using 13 diverse deformable objects (primarily plush toys and sponges). They compared AuRA's performance against human-collected data from three groups: amateurs, trained users, and experts. The primary baseline for reconstruction quality was the human-collected PhysTwin pipeline.

Key Insights and Ideas

  • Robot States > Visual Tracking: One of the most effective "aha" moments is the realization that visual tracking of the manipulator is a liability. By using the robot's own proprioception (joint states), AuRA provides perfectly clean "ground truth" for the forces/motions driving the simulation, leading to more stable physics parameter estimation.
  • Priors as Gap-Fillers: By capturing a 360-degree scan before interaction, the system can "hallucinate" the parts of the object hidden behind the gripper during the actual movement. This significantly improves the visual quality of the resulting digital twin.
  • Yield vs. Effort: The paper highlights that human experts might be fast, but they are inconsistent. AuRA’s automated filtering of "bad" grasps before they happen leads to a much higher success rate (yield) in the final reconstruction phase compared to humans who might record unusable, highly-occluded, or kinematically awkward interactions.

Key Results

AuRA demonstrates that automation doesn't just save time; it improves reliability.

AuRA significantly outperforms human operators in terms of reconstruction yield and labor efficiency.
  • Efficiency: AuRA reduced active human labor per successful asset from 284.4 seconds to just 32.7 seconds (an 88.5% reduction).
  • Reliability: The end-to-end reconstruction yield (the percentage of attempts that resulted in a successful digital twin) was 74.3% for AuRA, compared to only 26.9% for human-operated collection overall.
  • Quality: In future-prediction tasks (predicting how an object will move under new forces), AuRA improved 7 out of 9 metrics compared to the human baseline, including a 28.1% reduction in Chamfer distance (a measure of geometric error).

Limitations and Future Work

While AuRA is a major step toward automated digital twin factories, some limitations remain:

  • Interaction Diversity: The current system is limited to simple "grasp-and-lift" interactions. More complex behaviors like poking, sliding, or folding would be needed to capture the full range of an object's material properties.
  • Initial Setup: A human is still required to place the object in the workspace and initialize the system.
  • Physics Representation: The system relies on a spring-mass model. While effective for many deformable objects, it may struggle with highly complex materials (e.g., fluids or granular materials) that require more sophisticated solvers like MPM (Material Point Method).

Future work suggests expanding the repertoire of autonomous interactions and integrating active learning to select interactions that specifically target the most uncertain physical parameters of an object.

Summary updated

Abstract

Simulation enables robot-learning experiments at scale, but acquiring physics-informed digital twins often requires human manipulation and manual annotation of recorded interactions. To reduce this reliance on human effort, we present AuRA, a system that automates robot interaction and data preparation for digital-twin reconstruction. AuRA combines execution-aware grasp selection, gripper-motion measurements derived from robot states, and registered pre-interaction geometry and appearance priors. These choices enable autonomous collection, avoid reliance on visual gripper tracking, and recover object information obscured by the robot. Across 13 deformable objects, AuRA reduces active human labor and wall-clock collection time per successfully reconstructed asset by 88.5% and 55.4%, respectively. Relative to the human-collected PhysTwin baseline, AuRA improves seven of nine metrics for future prediction under known gripper motion, including a 28.1% reduction in one-way $L_1$ Chamfer distance. It also achieves similar mean silhouette IoU and Chamfer distance under the evaluated unseen interactions. These results highlight the potential of autonomous robot interaction to support more efficient digital-twin acquisition with comparable reconstruction quality.
BibTeX
@misc{yu2026aura,
  author = {Xinchen Yu and Yoonkyo Jung and Haoran Chen and Zikui Cai and Aayush Talreja and Seungjae Lee and Hyunjin Kim and Furong Huang},
  title = {{AuRA: From Atoms to Bits through Autonomous Robot Interaction}},
  year = {2026},
  eprint = {2610.34452},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34452},
}
Figure