A multimedia art interaction model for motion recognition based on data-driven model

Zhen WangPeerJ Comput. Sci 2023 · DOI

TL;DR

This paper introduces a novel multimodal fusion model for high-precision human motion recognition in multimedia art. By combining skeletal data from OpenPose with inertial data from wearable devices, the model achieves a remarkable 97.85% recognition accuracy, significantly outperforming existing methods and paving the way for more intuitive and immersive human-computer interactions in digital art.

What Problem Does This Paper Address?

The field of multimedia art design is increasingly focused on creating engaging and interactive experiences for users. A key challenge in this domain is accurately and efficiently recognizing human motion to translate gestures and actions into seamless interactions within digital environments. Traditional methods often struggle with issues like occlusion, varying lighting conditions, and the need for complex sensor setups. Furthermore, while existing human motion recognition techniques are advancing, there's a need for a robust system that can handle the nuances of artistic expression and interaction, moving beyond simple action classification to a more comprehensive understanding of movement. This paper aims to bridge this gap by developing a data-driven, multimodal approach that leverages both visual and inertial data for superior motion recognition in multimedia art contexts.

Methods

The core of this research lies in a multimodal fusion approach that combines two primary data streams:

  • Video-based Skeleton Extraction: The OpenPose framework is utilized to extract human skeleton information from video input. This involves identifying key joint points and their relationships, effectively creating a digital representation of the human body's pose. This method is chosen for its ability to capture detailed spatial information about body movements.
  • Inertial Data from Wearable Devices: Inertial sensors embedded in wearable devices (specifically, a bracelet in this study) are used to capture motion intensity. This includes data like acceleration and gyroscope readings, which provide complementary information about the dynamics and intensity of movements, especially when visual data might be occluded or less informative.

These two distinct data streams are then fused and fed into a Recurrent Neural Network (RNN). The RNN is chosen for its ability to process sequential data and capture temporal dependencies, which are crucial for understanding the continuity and context of human actions. The model is trained in a data-driven manner, learning to map the combined features to specific human motions. The paper highlights the novelty of this multimodal fusion, emphasizing that it's not just about extracting skeleton data but about synergistically combining it with inertial data for a more robust and precise recognition system.

Datasets

The paper describes experiments conducted with human volunteers. While specific large-scale public datasets are not explicitly detailed for training the final model, the authors mention using two open datasets, UTD-MHAD and C-MHAD, for comparative analysis and to imitate the data collected in their experiment (video and wrist-collected inertial data). The experimental setup involved three volunteers performing specific actions, with each experiment repeated five times. Two subjects were used for training the model, and one for testing.

Key Insights and Ideas

  • Multimodal Fusion is Key: The paper strongly advocates for combining visual (skeleton) and inertial (motion intensity) data. This synergy overcomes the limitations of each individual modality, leading to more accurate and robust motion recognition.
  • RNNs for Temporal Context: The use of RNNs is critical for capturing the sequential nature of human actions, allowing the model to understand actions in context rather than as isolated events.
  • OpenPose for Rich Skeleton Data: Leveraging OpenPose provides a detailed and accurate representation of human pose, which is fundamental for understanding the spatial aspects of movement.
  • Wearable Devices for Intensity: Inertial sensors offer a practical way to quantify movement intensity, complementing the spatial information from video and providing a more complete picture of the action.
  • Ergonomics in Multimedia Art: The research directly ties into improving user experience and ergonomics in multimedia art design, aiming for more intuitive and natural interactions.

Key Results

The proposed multimodal fusion model achieved a recognition accuracy of 97.85%. This significantly outperforms traditional methods like:

  • Backpropagation Neural Network (BPNN): 94.35% accuracy
  • K-Nearest Neighbor (KNN): 90.12% accuracy

The paper also presents detailed results for precision (P), recall (R), and F1 score across different actions (Sitting, Standing, Left wave, Right wave, Both wave), all showing high performance metrics, with the mean F1 score reaching 97.13%. The study also demonstrated the model's effectiveness in a practical application with a local video special effects production company, where it was praised for its identification efficiency.

Limitations and Future Work

  • Data Volume for Testing: The paper acknowledges that for actual testing in the art interaction application, the amount of data was small, which might affect the robustness of the results.
  • Subtle Motion Differences: The authors note that distinguishing between very similar motions remains a challenge.
  • Future Work: The researchers plan to:
  • Integrate a more comprehensive array of data sources.
  • Employ advanced methodologies.
  • Enhance model performance and tackle challenges with missing data instances.
  • Refine the ability to discern subtle differences between similar motions.
  • Explore more sophisticated feature extraction methods, including contextual information from multiple sensors and temporal modeling.

Summary updated

BibTeX
@article{wang2023multimedia,
  author = {Zhen Wang},
  title = {{A multimedia art interaction model for motion recognition based on data-driven model}},
  journal = {PeerJ Comput. Sci.},
  year = {2023},
  doi = {10.7717/peerj-cs.1662},
}
Figure