Introduction
Automated violence detection in video is a critical yet challenging task for content moderation and security. The primary difficulty lies in the subjective and visually diverse nature of "violence." Unlike concrete objects like a car or a tree, violence encompasses a wide spectrum of visual patterns—ranging from a silent sniper aiming a rifle to a chaotic street brawl or a scene involving blood.
Existing state-of-the-art systems typically attempt to bridge this gap by fusing multiple modalities, such as visual frames, audio signals, and motion trajectories. However, these models often struggle to generalize across different genres. A system trained on action movies might fail on horror films because the underlying distribution of violent acts changes. Most researchers rely on "average fusion," as more complex machine learning techniques often overfit the training data and fail when encountering new types of content.
The Subclass-Based Approach
Researchers from Renmin University of China have proposed a novel solution: breaking the broad concept of violence into ten concrete, visually distinct subclasses. By identifying specific attributes such as blood, gun, force, death, weapon, rope, fight, hit, bind, and aim, the model gains a more granular understanding of the scene.
Their technical pipeline utilizes a "learn-to-fuse" strategy:
- Multi-modal Extraction: Deep Convolutional Neural Networks (VGGNet and GoogleNet) extract visual features, while Mel-frequency Cepstral Coefficients (MFCC) handle audio, and Improved Dense Trajectories (IDT) capture motion.
- Subclass Classification: Linear SVMs are trained specifically for each of the ten subclasses.
- Coordinate Ascent Fusion: Instead of simple averaging, the system uses an optimization algorithm to determine the ideal weights for combining subclass detectors and raw features.
- Negative Bootstrap: To address the rarity of violent clips, the team used iterative sampling to identify the most informative non-violent examples for training.
Dataset and Manual Labeling
The study utilized the MediaEval 2015 Affective Impact of Movies task dataset, consisting of 10,900 short video clips. A key contribution of this work was the manual enrichment of the dataset; the authors labeled every violent clip in both the development and test sets with the ten subclasses.
This labeling process revealed a significant "dataset shift." For instance, the aim subclass appeared in 28% of training clips but only 7% of test clips. This shift explains why previous fusion attempts failed—they were over-optimizing for specific subclass distributions that did not persist in the test data. By focusing on individual subclass detectors, the new model remains stable even when the frequency of specific violent acts changes.
Key Insights: The Motion Surprise
One of the most surprising findings was the role of motion features. Historically, features like optical flow were considered essential for detecting physical violence like punches. However, the researchers found that when subclass-based image and audio features are used, motion features become dispensable. In fact, a system using only image and audio subclasses outperformed those that included complex motion trajectories, suggesting that the semantic information in subclasses captures the essence of the event more effectively than raw movement data.
Results and Performance
The subclass-based system achieved a new state-of-the-art on the MediaEval 2015 benchmark:
- Average Precision (AP): 0.303 (surpassing the previous best of 0.296).
- Precision at 100 (P100): 0.55, meaning 55% of the top 100 flagged videos were correctly identified as violent.
Critically, the system is "plug-and-play." While it required manual labels for training, it does not need subclass labels for new, unseen videos; it uses its internal detectors to make a final determination.
Conclusion and Future Work
By decomposing violence into subclasses, this research provides a more robust framework for multi-modal fusion. While the initial manual labeling is intensive and the definition of violence remains somewhat subjective, the approach significantly improves generalization. Future work may involve expanding the subclass vocabulary to include psychological violence and exploring the temporal relationships between subclasses, such as how the presence of a rope often precedes a bind event.
Summary updated