Traceable Human-to-Humanoid Sign Language Benchmarking

Ao Liu, Shengeng Tang, Lechao Cheng, Yanbin Hao, Bingkun Bao, Richang HongarXiv 2026 · arXiv

TL;DR

The authors introduce HumanoidCSL-20K, a massive dataset of over 20,000 Chinese Sign Language sequences designed to bridge the gap between human video and humanoid robot execution. By providing four aligned versions of every motion—from raw video fits to collision-free robot trajectories—the benchmark makes the complex pipeline of robot signing "traceable," allowing researchers to isolate and fix errors at every stage.

What Problem Does This Paper Address?

Teaching a humanoid robot to sign is exceptionally difficult. Unlike walking or simple manipulation, sign language requires precise, high-frequency coordination of handshapes, palm orientations, and bimanual movements.

Currently, researchers often rely on monocular video fits (like SMPL-X) to get human motion data. However, as shown in Figure 2, these fits are often "not robot-ready":

  1. Local Discontinuity: Monocular fits suffer from fast articulation and occlusions, leading to "jumps" or "frozen" limbs.
  2. Morphology Mismatch: Robots have different proportions and bulkier links than humans. A human pose that looks fine might cause the robot to crush its own fingers or hands (self-collision).
  3. Control Complexity: Tracking twelve finger joints while maintaining balance is a high-dimensional nightmare for standard controllers.

Existing systems are usually evaluated end-to-end, making it impossible to tell if a failure was due to a bad video fit, poor retargeting, or a weak controller.

Why fitted human motion is not directly robot-ready: (a) local jumps, (b) self-penetration after retargeting, and (c) execution failure.

Methods

The core of this work is a "traceable" pipeline that transforms raw human motion into executable robot trajectories through several distinct, recorded steps:

  • Local Human-Motion Repair: Instead of global smoothing (which blurs linguistic cues), the authors use a local repair method. It flags frames with high-velocity "jumps" or low-confidence keypoints and rebuilds them using spherical interpolation from reliable neighboring frames.
  • Robot Geometry Repair: To fix the morphology mismatch, the authors implement a two-stage sequence-level optimization. It first adjusts the arms to avoid hand-body collisions and then fine-tunes finger joints to resolve intra-hand penetrations, all while staying within a 20° budget of the original pose.
  • Two-Stage RL Curriculum: They train a PPO-based controller using a curriculum. The robot first learns to balance and track the body and wrists (Stage 1) before adding fine finger tracking (Stage 2).
  • Traceability: Crucially, every frame in the dataset carries "provenance" records—metadata explaining exactly how that frame was modified (e.g., "retained source fit" vs. "two-anchor interpolation").
The HumanoidCSL-20K pipeline: (a) data construction from video to repaired robot reference, and (b) the two-stage RL curriculum.

Datasets

The authors introduce HumanoidCSL-20K, which extends the existing CSL-Daily video corpus. It contains:

  • 20,648 sentence-level sequences of Chinese Sign Language.
  • Four aligned versions for each sequence: Source fit, Repaired human motion, Direct robot reference, and Geometry-repaired robot reference.
  • 2.46 million robot frames in total.

Key Insights and Ideas

  • The "Learnability" of References: One of the most striking findings is that repairing the reference motion makes the control task significantly easier. A controller trained on "Raw" references might only complete 57% of replays, whereas the same controller architecture achieves 100% completion on "Repaired" references.
  • Morphology Mismatch is a Major Hurdle: Without geometry repair, over 50% of frames in a direct retargeting attempt contain self-collisions. This highlights that simply "copying" human motion is insufficient for humanoid embodiments.
  • Local vs. Global Repair: Global smoothing (like Gaussian filters) often degrades the linguistic content of signs (measured by BLEU scores). The authors' local, observation-supported repair preserves content while successfully removing 97% of abnormal motion spikes.

Key Results

The benchmarks show massive improvements across the board, as summarized in Figure 1:

  • Motion Quality: Abnormal arm steps were reduced from 19.34% to 0.62%.
  • Geometric Feasibility: The share of penetration-free clips jumped from a dismal 1.21% to 88.39%.
  • Training Utility: Using the repaired data and the two-stage curriculum, the ActionScore ( a sign-specific fidelity metric) improved from 34.99 to 54.10, and the robot completed 100% of the evaluation replays.
Overview of HumanoidCSL-20K results showing improvements in motion quality, geometric feasibility, and training utility.

Limitations and Future Work

While a major step forward, the authors note several limitations:

  • Scope: The dataset currently covers only one sign language (CSL) and one humanoid model (Unitree G1).
  • Manual Cues Only: The current benchmark focuses on hands and arms. Future work needs to incorporate "non-manual" cues like facial expressions, head tilt, and gaze, which are vital for full sign language comprehension.
  • Real-World Validation: While simulation results are robust and qualitative hardware replays were successful, large-scale quantitative testing on physical robots remains a future goal.

Summary updated

Abstract

Sign data collection is costly, and teleoperation scales poorly, motivating reuse of large video corpora. Humanoid signing requires converting video-derived human motion into robot trajectories while preserving linguistic motion cues. Errors from fitting, human-motion repair, retargeting, robot geometry repair, and control are hard to separate from the final trajectory alone. We introduce HumanoidCSL-20K, a dataset and benchmark of 20,648 sentence-level Chinese Sign Language sequences, each with four aligned versions: the source, the repaired human motion, the direct robot reference, and the geometry-repaired robot reference. Observation-supported local human-motion repair, full-robot geometry repair, and cross-representation provenance make each transformation traceable. Paired evaluations measure human-motion continuity and content preservation, robot-reference feasibility, and physical execution. A sign-specific kinematic-reference protocol scores handshape, location, palm orientation, and inter-hand relation over the full planned motion. Full-corpus results show fewer abnormal arm / hand steps and less inter-hand and hand-body penetration after repair. Control experiments separate reference learnability from curriculum effects, while component scores expose remaining execution errors.
BibTeX
@misc{liu2026traceable,
  author = {Ao Liu and Shengeng Tang and Lechao Cheng and Yanbin Hao and Bingkun Bao and Richang Hong},
  title = {{Traceable Human-to-Humanoid Sign Language Benchmarking}},
  year = {2026},
  eprint = {2609.33354},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.33354},
}
Figure