TL;DR
AuRA (Autonomous Real-to-sim Asset Reconstruction) is a new system that automates the creation of physics-informed digital twins for deformable objects. By replacing manual human manipulation with autonomous robot interaction and leveraging robot state data to bypass visual tracking errors, AuRA reduces human labor by 88.5% and achieves a 2.8× higher reconstruction yield compared to human-operated baselines.
What Problem Does This Paper Address?
Creating high-fidelity "digital twins"—virtual models that replicate a real object's geometry, appearance, and physical behavior—is essential for scaling robot learning in simulation. However, the current "real-to-sim" pipeline is bottlenecked by the "atoms loop": humans must manually manipulate objects, record the interactions, and often perform tedious manual annotations to track object motion.
Beyond the labor cost, human-driven data collection suffers from two technical hurdles:
- Visual Occlusion: The very act of manipulating an object (e.g., a hand or gripper grabbing a plushie) obscures parts of the object, leading to incomplete geometric and appearance models.
- Tracking Noise: Estimating the precise motion of a human hand or a visually tracked gripper is prone to errors in segmentation and depth lifting, which in turn biases the physical parameters (like stiffness) inferred during reconstruction.
Methods
AuRA introduces a three-stage autonomous pipeline to move from "atoms to bits" without human intervention beyond initial object placement.
- Autonomous Data Acquisition: The system uses multi-view RGB-D cameras to scan a stationary object. It generates grasp candidates using AnyGrasp and filters them for kinematic feasibility and workspace constraints. The robot then executes a grasp-and-lift interaction while recording synchronized video and robot joint states.
- Data Post-processing:
- Object Tracking: AuRA uses Grounded-SAM2 for segmentation and MV-TAP for multi-view point tracking.
- Controller Trajectories: Instead of trying to visually track the gripper (which is often occluded), AuRA derives the gripper's path deterministically from the robot's internal joint states and calibrated geometry.
- Physics-Informed Reconstruction: The system fits a spring-mass model to the observed dynamics using the PhysTwin backend. To solve the occlusion problem, AuRA registers a "pre-interaction prior"—a high-quality scan of the object taken before the robot touched it—to the first frame of the interaction, filling in the gaps caused by the gripper.
Datasets
The authors evaluated AuRA using 13 diverse deformable objects (primarily plush toys and sponges). They compared AuRA's performance against human-collected data from three groups: amateurs, trained users, and experts. The primary baseline for reconstruction quality was the human-collected PhysTwin pipeline.
Key Insights and Ideas
- Robot States > Visual Tracking: One of the most effective "aha" moments is the realization that visual tracking of the manipulator is a liability. By using the robot's own proprioception (joint states), AuRA provides perfectly clean "ground truth" for the forces/motions driving the simulation, leading to more stable physics parameter estimation.
- Priors as Gap-Fillers: By capturing a 360-degree scan before interaction, the system can "hallucinate" the parts of the object hidden behind the gripper during the actual movement. This significantly improves the visual quality of the resulting digital twin.
- Yield vs. Effort: The paper highlights that human experts might be fast, but they are inconsistent. AuRA’s automated filtering of "bad" grasps before they happen leads to a much higher success rate (yield) in the final reconstruction phase compared to humans who might record unusable, highly-occluded, or kinematically awkward interactions.
Key Results
AuRA demonstrates that automation doesn't just save time; it improves reliability.
- Efficiency: AuRA reduced active human labor per successful asset from 284.4 seconds to just 32.7 seconds (an 88.5% reduction).
- Reliability: The end-to-end reconstruction yield (the percentage of attempts that resulted in a successful digital twin) was 74.3% for AuRA, compared to only 26.9% for human-operated collection overall.
- Quality: In future-prediction tasks (predicting how an object will move under new forces), AuRA improved 7 out of 9 metrics compared to the human baseline, including a 28.1% reduction in Chamfer distance (a measure of geometric error).
Limitations and Future Work
While AuRA is a major step toward automated digital twin factories, some limitations remain:
- Interaction Diversity: The current system is limited to simple "grasp-and-lift" interactions. More complex behaviors like poking, sliding, or folding would be needed to capture the full range of an object's material properties.
- Initial Setup: A human is still required to place the object in the workspace and initialize the system.
- Physics Representation: The system relies on a spring-mass model. While effective for many deformable objects, it may struggle with highly complex materials (e.g., fluids or granular materials) that require more sophisticated solvers like MPM (Material Point Method).
Future work suggests expanding the repertoire of autonomous interactions and integrating active learning to select interactions that specifically target the most uncertain physical parameters of an object.
Summary updated