Abstract
Multimedia event detection has been receiving increasing attention in recent years. Besides recognizing an event, the discovery of evidences (which is refered to as "recounting") is also crucial for user to better understand the searching result. Due to the difficulty of evidence annotation, only limited supervision of event labels are available for training a recounting model. To deal with the problem, we propose a weakly supervised evidence discovery method based on self-paced learning framework, which follows a learning process from easy "evidences" to gradually more complex ones, and simultaneously exploit more and more positive evidence samples from numerous weakly annotated video segments. Moreover, to evaluate our method quantitatively, we also propose two metrics, PctOverlap and F1-score, for measuring the performance of evidence localization specifically. The experiments are conducted on a subset of TRECVID MED dataset and demonstrate the promising results obtained by our method.
Introduction: The Challenge of Event Recounting
In Multimedia Event Detection (MED), identifying an event like a "marriage proposal" is only half the battle. Users also need "event recounting"—an explanation of why a video was flagged, pinpointing specific evidence segments such as a person kneeling or a ring box being opened.
The primary hurdle is the scarcity of fine-grained annotations. Manually marking start and end times for evidence in thousands of videos is prohibitively expensive. Consequently, researchers must rely on "weakly labeled" data, where we know an event occurs but not exactly when. Standard models often fail here, mistaking background noise for relevant evidence.
Proposed Methodology: Self-paced Learning (SPL)
To address weak supervision, this paper introduces a framework inspired by human learning: starting with simple concepts before tackling complex ones.
- Semantic Concept Features: The model utilizes a library of pre-defined concepts (e.g., "hugging," "cheering") to represent video segments. This ensures that the generated recounting is human-readable.
- Easy-to-Hard Strategy: The SPL framework initially trains on "easy" samples—segments where the model has high confidence.
- Iterative Refinement: Using a Cyclic Coordinate Method (CCM), the system alternates between updating the classifier and lowering the "difficulty threshold" to incorporate more complex segments in subsequent rounds.
- Balanced Learning: Techniques from Exemplar-SVMs are adapted to handle the imbalance between sparse positive evidence and abundant background video.
Experimental Setup
The method was evaluated on subsets of the TRECVID MED dataset (MED13 and MED14). Two training scenarios were tested: 10Ex (10 positive examples) and 100Ex (100 positive examples). Concept libraries were built using three auxiliary datasets: TRECVID SIN (346 concepts), Google Sports (478 concepts), and YFCC (609 concepts). Performance was measured using two new metrics: PctOverlap and F1-score.
Results and Discussion
The SPL framework consistently outperformed baselines such as Random Forest, AdaBoost, and Basic Multiple Instance Learning (MIL).
Key findings include:
- Superior Localization: SPL achieved significantly higher PctOverlap and F1-scores compared to Basic MIL, especially in data-scarce scenarios (10Ex). For instance, on the SIN concept set (10Ex), SPL reached a PctOverlap of 0.4617 versus 0.3531 for MIL.
- Effective Convergence: As shown in the performance plots, the model typically converges to peak performance within 10 to 15 iterations.
- Robustness: The "pacing" strategy prevents the model from being misled by noisy background segments early in the training process.
Conclusion and Future Directions
The self-paced learning approach provides a robust solution for discovering evidence in weakly supervised settings. While effective, the model's reliance on a fixed concept library remains a limitation. Future research will explore non-linear pacing functions and the integration of temporal structures to model the chronological sequence of evidence within an event.
Summary updated