TL;DR
The authors introduce HumanoidCSL-20K, a massive dataset of over 20,000 Chinese Sign Language sequences designed to bridge the gap between human video and humanoid robot execution. By providing four aligned versions of every motion—from raw video fits to collision-free robot trajectories—the benchmark makes the complex pipeline of robot signing "traceable," allowing researchers to isolate and fix errors at every stage.
What Problem Does This Paper Address?
Teaching a humanoid robot to sign is exceptionally difficult. Unlike walking or simple manipulation, sign language requires precise, high-frequency coordination of handshapes, palm orientations, and bimanual movements.
Currently, researchers often rely on monocular video fits (like SMPL-X) to get human motion data. However, as shown in Figure 2, these fits are often "not robot-ready":
- Local Discontinuity: Monocular fits suffer from fast articulation and occlusions, leading to "jumps" or "frozen" limbs.
- Morphology Mismatch: Robots have different proportions and bulkier links than humans. A human pose that looks fine might cause the robot to crush its own fingers or hands (self-collision).
- Control Complexity: Tracking twelve finger joints while maintaining balance is a high-dimensional nightmare for standard controllers.
Existing systems are usually evaluated end-to-end, making it impossible to tell if a failure was due to a bad video fit, poor retargeting, or a weak controller.
Methods
The core of this work is a "traceable" pipeline that transforms raw human motion into executable robot trajectories through several distinct, recorded steps:
- Local Human-Motion Repair: Instead of global smoothing (which blurs linguistic cues), the authors use a local repair method. It flags frames with high-velocity "jumps" or low-confidence keypoints and rebuilds them using spherical interpolation from reliable neighboring frames.
- Robot Geometry Repair: To fix the morphology mismatch, the authors implement a two-stage sequence-level optimization. It first adjusts the arms to avoid hand-body collisions and then fine-tunes finger joints to resolve intra-hand penetrations, all while staying within a 20° budget of the original pose.
- Two-Stage RL Curriculum: They train a PPO-based controller using a curriculum. The robot first learns to balance and track the body and wrists (Stage 1) before adding fine finger tracking (Stage 2).
- Traceability: Crucially, every frame in the dataset carries "provenance" records—metadata explaining exactly how that frame was modified (e.g., "retained source fit" vs. "two-anchor interpolation").
Datasets
The authors introduce HumanoidCSL-20K, which extends the existing CSL-Daily video corpus. It contains:
- 20,648 sentence-level sequences of Chinese Sign Language.
- Four aligned versions for each sequence: Source fit, Repaired human motion, Direct robot reference, and Geometry-repaired robot reference.
- 2.46 million robot frames in total.
Key Insights and Ideas
- The "Learnability" of References: One of the most striking findings is that repairing the reference motion makes the control task significantly easier. A controller trained on "Raw" references might only complete 57% of replays, whereas the same controller architecture achieves 100% completion on "Repaired" references.
- Morphology Mismatch is a Major Hurdle: Without geometry repair, over 50% of frames in a direct retargeting attempt contain self-collisions. This highlights that simply "copying" human motion is insufficient for humanoid embodiments.
- Local vs. Global Repair: Global smoothing (like Gaussian filters) often degrades the linguistic content of signs (measured by BLEU scores). The authors' local, observation-supported repair preserves content while successfully removing 97% of abnormal motion spikes.
Key Results
The benchmarks show massive improvements across the board, as summarized in Figure 1:
- Motion Quality: Abnormal arm steps were reduced from 19.34% to 0.62%.
- Geometric Feasibility: The share of penetration-free clips jumped from a dismal 1.21% to 88.39%.
- Training Utility: Using the repaired data and the two-stage curriculum, the ActionScore ( a sign-specific fidelity metric) improved from 34.99 to 54.10, and the robot completed 100% of the evaluation replays.
Limitations and Future Work
While a major step forward, the authors note several limitations:
- Scope: The dataset currently covers only one sign language (CSL) and one humanoid model (Unitree G1).
- Manual Cues Only: The current benchmark focuses on hands and arms. Future work needs to incorporate "non-manual" cues like facial expressions, head tilt, and gaze, which are vital for full sign language comprehension.
- Real-World Validation: While simulation results are robust and qualitative hardware replays were successful, large-scale quantitative testing on physical robots remains a future goal.
Summary updated