SIDQL: An Efficient Keyframe Extraction and Motion Reconstruction Framework in Motion Capture

Xuling Zhang, Ziru Zhang, Yuyang Wang, Lik-Hang Lee, Pan HuiarXiv 2024 · DOI · arXiv

Abstract

High-fidelity motion capture (mocap) is essential for immersive Metaverse experiences, yet the resulting high-frequency data streams often exceed network capacities, causing perceptible latency. This paper introduces SIDQL (Spherical Interpolation based Deep Q-Learning), a framework that optimizes data transmission by intelligently selecting keyframes and reconstructing intermediate motion. By employing a spherical coordinate system to preserve physical constraints and a Deep Q-Learning agent to automate keyframe selection, SIDQL achieves a decision latency of 0.004s and maintains low reconstruction errors.

Introduction

In the Metaverse, digital avatars must mirror real-world movements with minimal lag. Current mocap systems generate massive amounts of data, necessitating efficient compression techniques. Traditional methods often rely on keyframe extraction followed by interpolation in Cartesian space. However, these approaches face two primary challenges: the "stretchy bone" problem, where linear interpolation in 3D space alters bone lengths, and subjective keyframe selection based on human-labeled datasets rather than mathematical optimality.

Methodology

The SIDQL framework addresses these issues through a dual-pronged approach involving physics-informed geometry and reinforcement learning.

Overview of the SIDQL framework, illustrating the reconstruction process and keyframe extraction.

1. Spherical Reconstruction Algorithm

To eliminate the "stretchy bone" artifact, SIDQL converts motion data from Cartesian $(x, y, z)$ coordinates to a spherical coordinate system $(r, \theta, \phi)$. By treating the bone length ($r$) as a constant, the framework ensures that limbs do not unnaturally stretch or shrink during reconstruction. The system uses third-order polynomial functions to interpolate the angular components ($\theta, \phi$) and the root point's trajectory, ensuring fluid transitions between keyframes.

2. Deep Q-Learning (DQL) for Keyframe Extraction

Rather than relying on human-labeled data, SIDQL utilizes a DQL agent to identify optimal keyframes. The agent's reward function is tied directly to the minimization of the Mean Angle Error (MAE) across the reconstructed sequence. This allows the model to learn mathematically optimal frame selections that maximize reconstruction accuracy.

Notation definitions for the SIDQL framework.

Evaluation and Results

The framework was evaluated using the CMU Graphics Lab Motion Capture Database, focusing on 23 major joints downsampled to 30 FPS.

  • Accuracy: For a 60-frame sequence, extracting just 5 keyframes resulted in a reconstruction error of less than 0.09.
  • Efficiency: SIDQL demonstrated a decision time of 0.004 seconds per frame, making it 225 times faster than traditional "Greedy" optimization algorithms (0.9s).
  • Performance: The model significantly outperformed baseline methods like random or uniform selection, particularly in non-linear movements such as jumping and running.

Conclusion

SIDQL provides a robust solution for real-time motion capture in bandwidth-constrained environments. By integrating physical constraints into the coordinate system and automating keyframe selection via RL, the framework balances high fidelity with ultra-low latency. Future work will focus on addressing artifacts in high-velocity movements and deploying the system on physical hardware.

Summary updated

Abstract

Metaverse, which integrates the virtual and physical worlds, has emerged as an innovative paradigm for changing people's lifestyles. Motion capture has become a reliable approach to achieve seamless synchronization of the movements between avatars and human beings, which plays an important role in diverse Metaverse applications. However, due to the continuous growth of data, current communication systems face a significant challenge of meeting the demand of ultra-low latency during application. In addition, current methods also have shortcomings when selecting keyframes, e.g., relying on recognizing motion types and artificially selected keyframes. Therefore, the utilization of keyframe extraction and motion reconstruction techniques could be considered a feasible and promising solution. In this work, a new motion reconstruction algorithm is designed in a spherical coordinate system involving location and velocity information. Then, we formalize the keyframe extraction problem into an optimization problem to reduce the reconstruction error. Using Deep Q-Learning (DQL), the Spherical Interpolation based Deep Q-Learning (SIDQL) framework is proposed to generate proper keyframes for reconstructing the motion sequences. We use the CMU database to train and evaluate the framework. Our scheme can significantly reduce the data volume and transmission latency compared to various baselines while maintaining a reconstruction error of less than 0.09 when extracting five keyframes.
BibTeX
@misc{zhang2024sidql,
  author = {Xuling Zhang and Ziru Zhang and Yuyang Wang and Lik-Hang Lee and Pan Hui},
  title = {{SIDQL: An Efficient Keyframe Extraction and Motion Reconstruction Framework in Motion Capture}},
  year = {2024},
  eprint = {2407.00925},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.00925},
}
Figure