Abstract
Building high-quality digital assets of robotic arms is crucial yet challenging for the Real2Sim2Real pipeline. Current approaches naively bind static 3D Gaussians according to URDF links, forcing them to follow an URDF-rigged motion passively. However, real-world arm motion is noisy, and the idealized URDF-rigged motion cannot accurately model it, leading to severe rendering artifacts in 3D Gaussians. To address these challenges, we propose RoboArmGS, a novel hybrid representation that refines the URDF-rigged motion with learnable Bézier curves, enabling more accurate real-world motion modeling. To be more specific, we present a learnable Bézier Curve motion refiner that corrects per-joint residuals to address mismatches between real-world motion and URDF-rigged motion. RoboArmGS enables the learning of more accurate real-world motion while achieving a coherent binding of 3D Gaussians across arm parts. To support future research, we contribute a carefully collected dataset named RoboArm4D, which comprises several widely used robotic arms for evaluating the quality of building high-quality digital assets. We evaluate our approach on RoboArm4D, and RoboArmGS achieves state-of-the-art performance in real-world motion modeling and rendering quality. The code and dataset will be released.
Introduction
Creating a "digital twin" of a robotic arm—a photorealistic, controllable 3D model—is a cornerstone for robot learning and simulation. While 3D Gaussian Splatting (3DGS) has revolutionized scene reconstruction, applying it to articulated robots presents unique challenges. The standard approach relies on the Universal Robot Description Format (URDF), a mathematical blueprint of links and joints. However, a fundamental "sim-to-real" gap exists: idealized URDF kinematics do not account for joint friction, control latency, or mechanical "slop" present in physical systems. When 3D Gaussians are forced to follow these idealized models, the result is often blurry textures and "floating" artifacts.
Methodology
RoboArmGS introduces a two-pronged hybrid representation to bridge the gap between idealized kinematics and real-world observations.
Structured Gaussian Binding (SGB)
Instead of treating 3D Gaussians as an unconstrained cloud of points, RoboArmGS anchors each Gaussian to a specific face on the robot's 3D mesh. This ensures the digital asset remains structurally solid and maintains its topology during complex movements. A "binding-aware" densification strategy is employed to add detail where necessary without compromising the connection to the underlying skeleton.
Bézier-based Motion Refiner (BMR)
The core innovation of RoboArmGS is the Bézier-based Motion Refiner. Rather than discarding the URDF model, the system uses it as a strong prior and learns a smooth residual correction. These corrections are parameterized by Bézier curves, which are inherently continuous and well-suited for modeling the subtle, time-varying deviations between commanded poses and physical positions. By jointly optimizing Gaussian appearance and these Bézier corrections, the model learns the precise physical motion of the robot.
Experimental Evaluation
RoboArm4D Dataset
To evaluate the proposed method, the researchers introduced RoboArm4D, a benchmark comprising high-quality monocular video and synchronized joint data for three widely used industrial robots.
The dataset includes:
- Franka FR3 (7-DoF)
- Universal Robots UR-5e (6-DoF)
- ABB IRB 120 (6-DoF)
Results
RoboArmGS demonstrates significant improvements in rendering quality, particularly in Novel-Pose Synthesis. On the UR5e platform, the method achieved a PSNR of 39.559, representing an 8.6 dB improvement over existing deformable 3DGS methods. Qualitatively, the model eliminates structural distortions and "cloudy" floaters, producing sharp renders that align perfectly with ground-truth video, even at articulated grippers.
Conclusion
RoboArmGS provides a robust framework for transforming casual video into high-precision digital assets. By refining idealized kinematics with learnable Bézier curves and employing structured Gaussian binding, it achieves state-of-the-art performance in real-world motion modeling and photorealistic rendering.
Summary updated