Abstract
Dense Video Captioning (DVC) involves localizing and describing all distinct events in an untrimmed video. Current models often suffer from "temporal bias," failing to detect short events or events occurring at unusual times. We introduce E2DVC (Event-Equalized Dense Video Captioning), a framework that utilizes visual feature clustering to identify "pseudo-events." This approach ensures the model allocates equal attention to every potential event regardless of its length or position. E2DVC achieves state-of-the-art performance on ActivityNet Captions and YouCook2, demonstrating a superior ability to capture short-duration events and maintain localization precision.
Introduction
Dense Video Captioning (DVC) provides a granular, play-by-play breakdown of video content. However, existing models typically struggle with Temporal Bias, which manifests in two primary ways:
- Data Distribution Bias: Training datasets often have events clustered at specific times. Models learn these patterns and become "blind" to events happening at unusual timestamps.
- Event Duration Bias: Long events contain more frames and features than short ones. In standard attention mechanisms, the volume of data from a long event can "drown out" the signal from a short event, leading the model to ignore the latter entirely.
Methodology: The E2DVC Framework
The core philosophy of E2DVC is to treat every potential event equally by using visual similarity as a guide rather than relying solely on learned temporal patterns.
Event Perception Module (EPM)
The EPM is a non-trainable module that performs agglomerative hierarchical clustering on visual features. It groups similar frames into "pseudo-events." Since it is a fixed algorithm, it avoids inheriting training set biases. A refinement mechanism ensures these clusters are temporally coherent by splitting discontinuous segments and discarding outliers.
Pseudo-Event Initialization (PEI)
In typical Transformer-based DVC models, "queries" (the slots that look for events) are initialized randomly. E2DVC instead "seeds" these queries using the center timestamps of the pseudo-events found by the EPM. This forces the model to start its search where visual changes actually occur.
Event-Enhanced Encoder (EEE)
The EEE incorporates pseudo-event labels directly into the encoding process. By explicitly labeling frame-to-cluster membership, the encoder can better distinguish between overlapping or adjacent events, preventing long-duration events from overwhelming the attention mechanism.
Experimental Results
E2DVC was evaluated on the ActivityNet Captions and YouCook2 benchmarks using various pre-trained visual features, including CLIP, C3D, and TSN.
State-of-the-Art Performance
E2DVC achieved a CIDEr score of 33.63 on ActivityNet Captions (using CLIP features), outperforming previous leading models like PDVC and CM². The model showed higher precision in finding exact start and end times, which translated to more accurate captions.
Mitigating Temporal Bias
A key strength of E2DVC is its ability to capture events across the entire temporal and duration spectrum. As shown in Figure 5, the predicted event distribution of E2DVC is significantly closer to the ground truth than baseline models, which tend to ignore events shorter than 10 seconds.
Conclusion
By leveraging visual clustering as an unbiased prior, E2DVC effectively addresses the long-standing issue of temporal bias in dense video captioning. The framework's ability to equalize attention across event scales leads to more comprehensive and accurate video descriptions. Future work will focus on refining boundary precision for exceptionally long events and exploring automated hyperparameter tuning for diverse video domains.
Summary updated