Abstract
We propose a co-detection and labeling (CoDeL) framework to identify persons that contain self-consistent appearance in multiple images. Our CoDeL model builds upon the deformable part-based model to detect human hypotheses and exploits cross-image correspondence via a matching classifier. Relying on a Gaussian process, this matching classifier models the similarity of two hypotheses and efficiently captures the relative importance contributed by various visual features, reducing the adverse effect of scattered occlusion. Further, the detector and matching classifier together make our model fit into a semi-supervised co-training framework, which can get enhanced results with a small amount of labeled training data. Our CoDeL model achieves decent performance on existing and new benchmark datasets.
Introduction
Traditional human detection and identification systems often operate on a single-image basis or rely heavily on face recognition. However, in real-world photo collections, such as family albums or TV show episodes, faces are frequently occluded, turned away, or blurred. This paper addresses the human co-detection problem: given a set of images containing a group of people, the goal is to detect every person and label them by identity.
The core challenge lies in the fact that while poses and backgrounds vary significantly, a person's clothing and overall appearance usually remain consistent within a specific event. CoDeL exploits this "self-consistent appearance" as a powerful prior to improve detection accuracy in scenarios where traditional methods fail.
The CoDeL Framework
The CoDeL framework is built on three main pillars designed to bridge the gap between detection and identity matching.
1. Human Representation
The model utilizes a modified Deformable Part-based Model (DPM), representing a person as a "root" filter for the whole body and several flexible "part" filters (head, torso, etc.). A dedicated face filter is added to provide a strong anchor when facial features are visible.
2. Gaussian Process (GP) Matching Classifier
To determine if two detections represent the same person, a GP classifier models similarity based on color histograms, texture (LBP), and SIFT features across different body parts. The GP is particularly effective because it learns the relative importance of different features, making it robust to partial occlusions. For instance, if a person is sitting behind a table, the GP learns to down-weight the importance of the legs for identification.
3. Semi-Supervised Co-Training
This iterative "engine" allows the model to bootstrap itself using a small amount of labeled data.
- Detector → Matcher: High-confidence detections are used as "weakly labeled" samples to retrain the matching classifier.
- Matcher → Detector: High-confidence matches across images provide evidence to "rescue" weak detections, which are then used to retrain the detector.
Experimental Evaluation
The authors evaluated CoDeL on a standard Pedestrian Dataset and a new Human Co-Detection (HCD) Dataset. The HCD dataset features diverse poses (sitting, eating) and complex backgrounds, providing a realistic test for general co-detection.
Key Results
- Detection Performance: On the HCD dataset, CoDeL achieved an Average Precision (AP) of 74.94%, significantly outperforming the standard DPM (69.94%). On the Pedestrian dataset, the improvement was even more pronounced, jumping from 59.7% to 74.4%.
- Matching Accuracy: The GP framework reached 91.43% matching accuracy on the pedestrian set by combining multiple visual features.
- Iterative Improvement: Detection performance steadily increased over 15 rounds of co-training, validating the effectiveness of the semi-supervised approach.
Conclusion and Future Work
CoDeL demonstrates that cross-image appearance consistency can significantly boost human detection and labeling. However, the system can still be challenged by individuals wearing similar uniforms or extreme changes in clothing. Future work includes integrating deeper spatial relationships into the matching process and developing an online version of the algorithm for real-time processing.
Summary updated