NII_UIT AT TRECVID 2020

Duy-Dinh Le, Hung-Quoc Vo, Dung-Minh Nguyen, Tien-Van Do, Thinh-Le-Gia Pham, Tri-Le-Minh Vo, Thua-Ngoc Nguyen, Vinh-Tiep Nguyen, Thanh-Duc Ngo, Zheng Wang, Shin'ichi SatohTRECVID 2020 · OpenReview

Abstract

The NII_UIT team's participation in TRECVID 2020 focused on practical, heuristic-based solutions for complex video analysis tasks, including Instance Search (INS) and Video Summarization (VSUM). By leveraging pretrained models and simple spatial-temporal rules, the team achieved competitive results in identifying specific people performing specific actions and generating character-centric summaries from large-scale datasets like the BBC EastEnders collection.

Introduction

The TRECVID 2020 competition presented significant challenges in video retrieval and summarization. The NII_UIT team approached these tasks by prioritizing robust, interpretable heuristics over complex end-to-end training. Their methodology centered on bridging the gap between individual object/face detection and the understanding of complex human-object relationships.

Instance Search (INS)

The primary challenge in Instance Search is the "co-occurrence trap," where a target person and a specific action appear in the same shot, but the action is performed by someone else. To mitigate this, the team categorized actions into two types:

  1. Human-Object Interaction (HOI): Actions like "holding a phone" or "sitting on a couch" were handled by detecting objects using EfficientDet. A spatial distance heuristic was then applied to ensure the target person's face was in close proximity to the detected object.
  2. Facial Expression (FE): Actions like "crying" or "laughing" were detected using an ensemble of facial expression recognition models (ESR) focused on the target person's face.

To streamline this process, the team mapped TRECVID action queries to existing classes in the MSCOCO and FER+ datasets.

Mapping of TRECVID actions to COCO and FER+ classes

Video Summarization (VSUM)

For character-centric summarization of the EastEnders series, the team developed a three-stage pipeline:

  • Segmentation: Videos were divided into shots based on metadata.
  • Scoring: Shots were scored based on the presence of the target character (face detection) and visual importance (calculated via VASNet, a self-attention network).
  • Selection: The Knapsack algorithm was used to select the highest-scoring shots within the required summary duration.

Results and Analysis

In the INS task, the team's best-performing run achieved a Mean Average Precision (mAP) of 0.091. A key finding was that limiting the initial pool of shots for action reranking (e.g., top 10,000 vs. 15,000) could lead to better overall performance by reducing noise.

Quantitative performance of INS runs

For VSUM, the pipeline successfully captured key narrative moments for major characters, with summary lengths appropriately scaled to the character's screen time and importance.

Conclusion and Future Work

The NII_UIT team demonstrated that simple spatial heuristics and high-quality pretrained models can be highly effective for large-scale video search and summarization. Future work will address current limitations, such as improving temporal consistency across dynamic shots and refining identity verification in crowded scenes to further reduce false positives.

Summary updated

BibTeX
@inproceedings{le2020nii,
  author = {Duy-Dinh Le and Hung-Quoc Vo and Dung-Minh Nguyen and Tien-Van Do and Thinh-Le-Gia Pham and Tri-Le-Minh Vo and Thua-Ngoc Nguyen and Vinh-Tiep Nguyen and Thanh-Duc Ngo and Zheng Wang and Shin'ichi Satoh},
  title = {{NII\_UIT AT TRECVID 2020}},
  booktitle = {TRECVID},
  year = {2020},
  url = {https://openreview.net/forum?id=QRZ6OE8mxq},
}
Figure