Abstract
Traditional video understanding models often rely on "black-box" 3D Convolutional Neural Networks (CNNs) that struggle with environmental bias and fine-grained entity recognition. This paper introduces the Perception-based Multi-modal Representation (PMR) framework, a novel approach that mimics human perception by explicitly modeling the interactions between actors, objects, and their environments. By decomposing video scenes into interpretable components, PMR achieves state-of-the-art performance in Temporal Action Proposal Generation (TAPG) and Video Paragraph Captioning (VPC) while providing a more transparent reasoning process.
Introduction
Most contemporary video understanding systems utilize pre-trained 3D CNN backbones, such as C3D or I3D, to extract features from video snippets. While effective for general classification, these models possess three critical flaws:
- Environment Bias: Models often over-rely on background context, misidentifying actions based on the setting (e.g., labeling any activity in a park as "jogging").
- Actor Overload: In crowded scenes, models fail to distinguish between primary actors and incidental background figures.
- Small Entity Neglect: Crucial details, particularly in egocentric videos where small objects are central to the action, are often lost during global feature compression.
To address these limitations, the PMR framework shifts the paradigm toward an entity-based approach, identifying the who (actors), the what (objects), and the where (environment) to construct a holistic understanding of the scene.
The PMR Framework: Four Beholder Modules
The core of the PMR architecture consists of four specialized "Beholder" modules designed to process specific facets of video data:
1. Environment Beholder
This module utilizes a standard 3D CNN to capture the global spatial-temporal context. It provides the necessary background information against which specific actions take place.
2. Actors Beholder
To isolate relevant human activity, this module employs a Faster R-CNN detector to localize individuals. It features an Adaptive Attention Mechanism (AAM), which uses a "hard" threshold to filter out background extras. Unlike traditional soft attention, this mechanism eliminates irrelevant entities entirely, significantly reducing noise in the final representation.
3. Objects Beholder
Recognizing that small objects are difficult to detect through visual features alone, the researchers utilized CLIP (Contrastive Language-Image Pre-training). By matching visual features against a broad vocabulary of object names, the system leverages linguistic priors to identify objects that are visually subtle but contextually significant.
4. Actors-Objects-Environment (AOE) Beholder
Acting as the framework's integration engine, the AOE Beholder uses self-attention mechanisms to model the complex relationships between the identified actors, objects, and the environment. This results in a unified, multi-modal representation that reflects the interactive nature of real-world events.
Experimental Evaluation
The PMR framework was evaluated on two primary downstream tasks using the ActivityNet datasets.
Temporal Action Proposal Generation (TAPG)
Using the BMN framework on ActivityNet-1.3, PMR achieved an AUC of 69.71% on the validation set and 70.10% on the testing set. These results surpass previous leading models such as AEI and ABN, demonstrating the framework's superior ability to localize action boundaries.
Video Paragraph Captioning (VPC)
For the ActivityNet Captions dataset, PMR was integrated into a "Transformer-in-Transformer" (TinT) architecture. The model produced more coherent and diverse multi-sentence descriptions, achieving a BLEU-4 score of 14.50 and a CIDEr score of 31.13. These metrics represent a substantial improvement over previous benchmarks like MART and Memory Transformer.
Discussion and Explainability
A key advantage of PMR is its inherent explainability. Because the model explicitly identifies actors and objects, its outputs can be audited. For instance, if the model identifies a "playing guitar" action, researchers can verify whether the system successfully detected both the instrument and the human interaction, rather than relying on a generic "music room" background feature.
Ablation studies confirmed the necessity of this multi-modal approach; removing either the Objects Beholder or the Actors Beholder resulted in a measurable decline in performance across all metrics.
Conclusion and Future Work
The PMR framework represents a significant step toward human-like video understanding. By prioritizing entity-based interactions over global black-box features, it provides both higher accuracy and greater transparency. Future research will focus on two areas: incorporating the visual dynamics and motion of objects (moving beyond linguistic priors) and expanding the Actors Beholder to recognize non-human entities, such as animals or robotic agents, to broaden the framework's applicability.
Summary updated