ROSE: A Shared Cross-Embodiment Facial-Motion Latent Space for Humanoid Robot Expression

Yongji Fu, Man Cao, Yi Zhou, Fei Lin, Mark F. Hansen, Guanqun CaoarXiv 2026 · arXiv

TL;DR

ROSE (RObot facial expression through Shared motion Embeddings) is a full-stack framework that enables humanoid robots to perform natural, audio-synced facial expressions. By learning a shared "motion language" across humans and robots and utilizing a high-density 34-actuator biomimetic head, it bypasses the lag and artifacts of traditional video-based methods to drive physical hardware directly from text and audio.

What Problem Does This Paper Address?

Creating socially approachable humanoid robots requires facial expressions that are both physically expressive and perfectly synchronized with speech. Current systems face two major hurdles:

  1. Physical Constraints: Humanoid heads are small. Packing enough motors to control subtle movements (like cheek flaring or eyelid squinting) is a mechanical nightmare.
  2. Algorithmic Latency: Most "talking head" AI models generate a video or 3D mesh first, which then must be "inverted" into motor commands. This "render-and-invert" pipeline introduces significant latency and visual artifacts, often resulting in a "dead-eyed" look or out-of-sync lips.

The authors propose a unified approach that treats hardware design and motion generation as a single, integrated problem.

Methods

The ROSE framework consists of four interconnected components:

  • Biomimetic Hardware: A 19.5 cm robot head featuring 34 actuators. It uses a flexible tendon-sheath transmission, allowing motors to be placed in the back of the head while tendons pull on specific facial action points.
  • ROSE-VAE: A Variational Autoencoder that learns a shared, identity-independent latent space. It factorizes "who" the face is (identity) from "how" it moves (motion), allowing it to translate human or digital-human movements into a format the robot understands.
  • ROSE-Flow: A dual-stream Multimodal Diffusion Transformer (MMDiT). It takes text (for global intent/emotion) and audio (for fine-grained lip-sync) to generate motion trajectories within the shared latent space. Crucially, it uses "frame-local audio attention" to ensure the mouth moves precisely with the sound.
  • ROSE-ACT: A mapping module that translates these latent trajectories directly into actuator commands. It is trained via "motor babbling"—observing the robot move and learning the relationship between motor states and visual outcomes.
Fig. 1: ROSE overview. Human, digital-human, and face-robot observations are mapped by ROSE-VAE into a shared facial-motion latent space. ROSE- Flow generates motion-latent trajectories from text and/or audio, and ROSE-ACT maps them to the actuator actions of a real robot. The right side illustrates real-time interaction, human-to-robot retargeting, and text/audio to expression applications.

Datasets

The authors utilized a diverse mix of data to ensure the model generalizes across different "embodiments":

  • Human Data: 32,021 clips from public datasets like HDTF, RAVDESS, and VFHQ.
  • Digital Human Data: 1,760 clips of high-fidelity digital avatars with expression captions to provide supervision for text-to-motion.
  • Robot Data: 129,431 synchronized frames of the physical ROSE robot collected through motor babbling to train the actuator mapping.

Key Insights and Ideas

  • Cross-Embodiment Learning: The core "aha moment" is treating human, digital, and robotic faces as different versions of the same underlying motion. By learning a shared motion basis, the robot can "borrow" the naturalness of human expressions.
  • Skipping the Image: By mapping latent vectors directly to servos (ROSE-ACT), the system avoids the computational cost of rendering pixels. This allows for much higher control frequencies and lower latency.
  • Decoupled Actuation: Using tendons instead of rigid linkages (see Figure 3) solves the spatial density problem, enabling 34 degrees of freedom in a volume where traditional designs might only fit a handful.
The 34-actuator layout uses tendons to enable complex expressions in a compact head.

Key Results

  • Superior Fidelity: ROSE outperformed state-of-the-art baselines (like SadTalker and FLOAT) in image quality (FID) and temporal consistency (FVD).
  • Better Lip-Sync: It achieved the lowest lip-sync error (LSE-D) on both HDTF and RAVDESS datasets.
  • Real-Time Performance: The direct latent-to-actuator interface increased the execution frame rate from 15 FPS to 60 FPS, making interactions feel significantly more responsive.
  • User Preference: In a user study, ROSE was rated significantly higher for realism, emotional resonance, and lip-sync accuracy compared to existing robot-retargeting methods.
Qualitative examples of the robot performing complex, multi-stage expressions.

Limitations and Future Work

  • Hardware Complexity: While the 34-actuator tendon system is highly expressive, it is mechanically complex to maintain and calibrate compared to simpler motor-driven faces.
  • Training Requirements: ROSE-ACT requires a "motor babbling" phase for each specific robot instance. Future work could explore zero-shot transfer between different robot designs.
  • Social Context: While the model handles text and audio, it does not yet incorporate visual feedback from the environment (e.g., reacting to a user's smile in real-time).

Summary updated

Abstract

Natural, coherent full-face dynamic expressions are central to making humanoid robots socially approachable, yet existing systems commonly realize only limited lip motion or scripted expressions. The limitation is both physical and algorithmic. A compact head leaves little room for densely packed, independently controllable actuation; meanwhile, learned systems often generate an avatar video, landmarks, or facial parameters and then infer actuator commands through visual retargeting. This render-and-invert path adds latency and artifacts, and it does not reliably transfer generated behavior to a physical robot. We present ROSE (RObot facial expression through Shared motion Embeddings), a framework that integrates biomimetic hardware, cross-embodiment motion representation learning, and multimodal motion generation. On the hardware side, we build a flexible tendon-sheath transmission structure that integrates 34 actuators into a 19.5 cm biomimetic head. By decoupling actuator placement from facial action points, it enables coordinated full-face expression within a compact volume. On the algorithmic side, ROSE-VAE learns an identity-independent shared facial-motion latent space from human, digital-human, and robot observations. ROSE-Flow directly maps text, audio, or their combination to trajectories in this motion space without using video as a generator input. ROSE-ACT then converts these trajectories into constrained actuator commands. Our system demonstrates how high-density biomimetic hardware and a shared executable facial-motion representation can jointly unify cross-embodiment learning, multimodal generation, and physical robot expression.
BibTeX
@misc{fu2026rose,
  author = {Yongji Fu and Man Cao and Yi Zhou and Fei Lin and Mark F. Hansen and Guanqun Cao},
  title = {{ROSE: A Shared Cross-Embodiment Facial-Motion Latent Space for Humanoid Robot Expression}},
  year = {2026},
  eprint = {2611.20950},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20950},
}
Figure