Abstract
The NII_UIT team's participation in TRECVID 2020 focused on practical, heuristic-based solutions for complex video analysis tasks, including Instance Search (INS) and Video Summarization (VSUM). By leveraging pretrained models and simple spatial-temporal rules, the team achieved competitive results in identifying specific people performing specific actions and generating character-centric summaries from large-scale datasets like the BBC EastEnders collection.
Introduction
The TRECVID 2020 competition presented significant challenges in video retrieval and summarization. The NII_UIT team approached these tasks by prioritizing robust, interpretable heuristics over complex end-to-end training. Their methodology centered on bridging the gap between individual object/face detection and the understanding of complex human-object relationships.
Instance Search (INS)
The primary challenge in Instance Search is the "co-occurrence trap," where a target person and a specific action appear in the same shot, but the action is performed by someone else. To mitigate this, the team categorized actions into two types:
- Human-Object Interaction (HOI): Actions like "holding a phone" or "sitting on a couch" were handled by detecting objects using EfficientDet. A spatial distance heuristic was then applied to ensure the target person's face was in close proximity to the detected object.
- Facial Expression (FE): Actions like "crying" or "laughing" were detected using an ensemble of facial expression recognition models (ESR) focused on the target person's face.
To streamline this process, the team mapped TRECVID action queries to existing classes in the MSCOCO and FER+ datasets.
Video Summarization (VSUM)
For character-centric summarization of the EastEnders series, the team developed a three-stage pipeline:
- Segmentation: Videos were divided into shots based on metadata.
- Scoring: Shots were scored based on the presence of the target character (face detection) and visual importance (calculated via VASNet, a self-attention network).
- Selection: The Knapsack algorithm was used to select the highest-scoring shots within the required summary duration.
Results and Analysis
In the INS task, the team's best-performing run achieved a Mean Average Precision (mAP) of 0.091. A key finding was that limiting the initial pool of shots for action reranking (e.g., top 10,000 vs. 15,000) could lead to better overall performance by reducing noise.
For VSUM, the pipeline successfully captured key narrative moments for major characters, with summary lengths appropriately scaled to the character's screen time and importance.
Conclusion and Future Work
The NII_UIT team demonstrated that simple spatial heuristics and high-quality pretrained models can be highly effective for large-scale video search and summarization. Future work will address current limitations, such as improving temporal consistency across dynamic shots and refining identity verification in crowded scenes to further reduce false positives.
Summary updated