Human Pose Driven Object Effects Recommendation

Zhaoxin Fan, Fengxin Li, Hongyan Liu, Jun He, Xiaoyong DuarXiv 2022 · DOI · arXiv

Introduction

In the rapidly evolving landscape of micro-video platforms like TikTok and Kwai, "object effects"—virtual stickers, props, or advertisements—are vital for user engagement and monetization. However, recommending the right digital item for a specific video is a complex challenge. Traditional computer vision models often suffer from background bias, where the system over-indexes on background scenery (e.g., a kitchen) rather than the actual human action (e.g., dancing), leading to irrelevant recommendations like a frying pan instead of a disco ball. Furthermore, the relationship between videos and items is often sparse and ambiguous, making it difficult for machines to distinguish between similar-looking actions.

The PoseRec Framework

To address these challenges, researchers have developed PoseRec, a novel recommendation system that shifts the focus from raw pixels to 3D human skeletal structures. By analyzing body language rather than background noise, PoseRec achieves a significantly higher signal-to-noise ratio.

Overview of the PoseRec architecture, highlighting the video-side pose extraction and item-side prototype learning.

The framework consists of three primary components:

  1. Pose-Driven Feature Extraction: The system utilizes BlazePose to extract 33 3D human landmarks from video frames. These landmarks are processed as a spatio-temporal graph through a Spatio-Temporal Graph Convolutional Network (ST-GCN), ensuring the model learns the core action while ignoring irrelevant visual details.
  2. Item-Aware Implicit Prototype Learning: To handle item ambiguity, the model maps items into "prototypes" representing shared characteristics. This allows the system to link diverse items (e.g., a sports drink and a tennis racket) to the same underlying human behavior.
  3. Pose-Aware Transductive Hard-Negative Mining: Training effective recommendation models requires high-quality "negative samples." PoseRec uses pose similarity as a proxy: if two videos feature vastly different poses (e.g., cooking vs. sprinting), the system can safely pull negative samples from one to train the other, avoiding the pitfalls of random sampling.

Item-Aware Implicit Prototype Learning

A key innovation in PoseRec is the discovery that items should be grouped by the role they play in an interaction rather than just their category. For instance, the system may group certain basketballs with "decorations" and others with "sports" depending on how the human in the video interacts with them.

Visualization of how prototypes help disambiguate item recommendations based on human pose.

As shown in the visualization above, the prototype module allows the network to correctly identify relevant items even when the visual data is sparse. Without prototypes, the model may struggle to distinguish between semantically different items that appear in similar contexts.

Key Results and Insights

The researchers evaluated PoseRec using a custom benchmark, Pose-OBE, containing 212 micro-videos and 1,087 unique items. The results demonstrate that 33 coordinates can be more informative for recommendation than millions of raw pixels:

  • Superior Performance: PoseRec outperformed traditional recommendation baselines (like DeepFM) and state-of-the-art video architectures (like C3D) by approximately 13% on the NDCG@5 metric.
  • Category Accuracy: The system achieved an 83.28% accuracy (Recall@20) in suggesting the correct category of object.
  • Ablation Study: The importance of the "Hard-Negative Mining" module was particularly striking; its removal caused performance to drop by nearly 88% in some metrics, highlighting that the selection of training examples is as critical as the architecture itself.

Conclusion and Future Work

PoseRec demonstrates that a human-centric approach to video analysis can overcome the limitations of pixel-based models. While currently focused on videos featuring people, future work aims to scale this approach to larger datasets and develop end-to-end systems capable of real-time processing for live video streams. By prioritizing the "action" over the "noise," PoseRec sets a new standard for personalized digital content recommendation.

Summary updated

Abstract

In this paper, we research the new topic of object effects recommendation in micro-video platforms, which is a challenging but important task for many practical applications such as advertisement insertion. To avoid the problem of introducing background bias caused by directly learning video content from image frames, we propose to utilize the meaningful body language hidden in 3D human pose for recommendation. To this end, in this work, a novel human pose driven object effects recommendation network termed PoseRec is introduced. PoseRec leverages the advantages of 3D human pose detection and learns information from multi-frame 3D human pose for video-item registration, resulting in high quality object effects recommendation performance. Moreover, to solve the inherent ambiguity and sparsity issues that exist in object effects recommendation, we further propose a novel item-aware implicit prototype learning module and a novel pose-aware transductive hard-negative mining module to better learn pose-item relationships. What's more, to benchmark methods for the new research topic, we build a new dataset for object effects recommendation named Pose-OBE. Extensive experiments on Pose-OBE demonstrate that our method can achieve superior performance than strong baselines.
BibTeX
@misc{fan2022human,
  author = {Zhaoxin Fan and Fengxin Li and Hongyan Liu and Jun He and Xiaoyong Du},
  title = {{Human Pose Driven Object Effects Recommendation}},
  year = {2022},
  eprint = {2209.08353},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.08353},
}
Figure