Abstract
Identifying the primary subjects of interest in consumer-shot videos is a challenging task due to the subjective nature of "importance." This paper introduces a hybrid framework that combines Deep Neural Networks (DNNs) for feature extraction with Conditional Random Fields (CRFs) to model interpersonal relationships. By analyzing spatial trajectories and facial features, the proposed model effectively distinguishes intended subjects from bystanders, achieving an accuracy of 83.6% and an F1-measure of 87.8%.
Introduction
While modern computer vision systems excel at detecting people, they often struggle to prioritize them. In handheld videos, such as a parent filming a child in a crowded park, most individuals in the frame are bystanders. Standard attention models typically treat all detected persons equally, which can lead to failures in automated video editing or retargeting. This research addresses the challenge of decoding a videographer's intent by identifying "important" people—those the camera operator intended to capture.
Methodology
The authors propose a multi-stage architecture that integrates individual feature learning with relational modeling.
Feature Extraction
For each detected person, the model extracts:
- Spatial Features: The 2D trajectory of the bounding box over a temporal window, capturing movement relative to the camera.
- Facial Features: Color histograms or high-level embeddings (e.g., FaceNet) to represent appearance.
DNN-CRF Architecture
The core of the system is a hybrid DNN-CRF model. Individual features are processed by a DNN to generate "unary" potentials, representing the likelihood of importance based on isolated data. A CRF layer then models the dependencies between all individuals in a frame.
The CRF utilizes the insight that if two people move in sync (highly correlated trajectories), they are likely part of the same group and share the same importance label. Because the number of people per frame is typically small, the model can be trained end-to-end using exact inference and backpropagation.
Experimental Results
The model was evaluated on two datasets: a YouTube dataset (99 videos with crowd-sourced labels) and a Videographer dataset (20 videos labeled by the original camera operators).
The full DNN-CRF model using both trajectories and color histograms achieved a top accuracy of 83.6%, significantly outperforming baseline SVM (65.3%) and vanilla DNN (75.5%) approaches. Ablation studies confirmed that the CRF's pairwise terms are essential for correcting labels in frames with multiple people, preventing the model from over-labeling solitary bystanders as important.
Conclusion and Future Work
This research demonstrates that modeling the correlation between people's movements is a powerful proxy for videographer intent. While highly effective, the model faces challenges when faces are small or group dynamics are overly complex. Future work will explore incorporating gaze direction and facial expressions to further refine the distinction between subjects and bystanders.
Summary updated