PLUMCOT at TRECVid Instance Search 2018

Hervé Bredin, Benjamin Maurice, Ruiqing Yin, Camille Guinaudeau, Aman Behre, Claude Barras, Vivek Sharma, M. Saquib SarfrazTRECVID 2018

Abstract

In the TRECVid 2018 Instance Search (INS) task, the PLUMCOT team (LIMSI and KIT) explored the integration of neural embeddings for faces, places, and voices to identify specific individuals in specific locations. The study highlights that temporal scene segmentation is a critical factor for instance search, more than doubling retrieval performance by providing necessary context for place recognition. While the system integrated sophisticated audiovisual tools like SyncNet and speaker embeddings, the lack of annotated development data for hyperparameter tuning limited the effectiveness of multi-modal fusion.

Introduction

The TRECVid Instance Search (INS) task represents a complex multi-modal retrieval challenge: researchers must locate specific people within specific locations across a massive video corpus. Unlike simple face or landmark recognition, INS requires the successful intersection of both person and location constraints. The PLUMCOT team participated in the 2018 challenge to investigate how neural embeddings for faces, places, and voices could be fused, and specifically, whether temporal context and audio cues could improve retrieval accuracy.

System Architecture and Methodology

The PLUMCOT system was built upon a pipeline of specialized neural embeddings and the pyannote toolkit. The methodology focused on three primary components: feature extraction, temporal segmentation, and rank-based fusion.

Neural Embeddings

The team utilized several state-of-the-art pre-trained networks to generate embeddings:

  • Face Recognition: VGGFace2 was used for neural face embeddings.
  • Place Recognition: Places365 was employed for neural place embeddings.
  • Speaker Recognition: VoxCeleb1 and VoxCeleb2 were used for speaker embeddings and training the SyncNet talking-face detection module.

Temporal Scene Segmentation

Using the pyannote.video toolkit, the team segmented the video corpus into distinct "scenes." This segmentation was leveraged to improve place recognition through a temporal consistency assumption. In their primary run, the system updated the place score of every shot within a scene to match the highest score found anywhere within that scene. This approach assumes that if a location is identified in one part of a scene, it likely persists throughout the entire temporal segment, helping to filter out noise at the individual shot level.

Audiovisual Fusion

To incorporate audio cues, the team employed SyncNet to detect "talking faces." By identifying when a face on screen was likely the source of the audio, they could extract high-quality speaker samples to train query-specific speaker embeddings using pyannote.audio.

Because the team lacked development data to optimize raw score fusion, they utilized weighted rank fusion. Their primary formula heavily favored location over identity, using a weight of 0.9 for place rank and 0.1 for face rank.

Experimental Results

The evaluation on the TRECVid 2018 INS benchmark demonstrated the decisive impact of temporal context. The team submitted three primary runs:

  • Run 1 (With Scene Segmentation): Achieved a Mean Average Precision (mAP) of 0.230.
  • Run 2 (No Scene Segmentation): The score dropped significantly to 0.096.
  • Run 3 (With Audio Fusion): Achieved a score of 0.221.

The comparison between Run 1 and Run 2 reveals that temporal segmentation was responsible for a 140% improvement in performance. This suggests that place recognition models are often unreliable when applied to isolated shots, and that scene-level context acts as a powerful corrective filter.

Discussion and Limitations

Despite the sophisticated pipeline, the PLUMCOT team identified several hurdles that prevented the system from reaching its full potential.

The "Blind" Tuning Problem

A primary limitation was the lack of annotated development data for hyperparameter tuning. The fusion weights (such as the 0.9/0.1 split between place and face) were essentially educated guesses. This lack of optimization likely explains why Run 3, which included audio data, performed slightly worse than the visual-only Run 1. Without a development set, it was impossible to determine if the audio signal was inherently unhelpful or if the fusion weights were simply sub-optimal.

Evaluation Granularity

The team also noted that the challenge's evaluation structure made it difficult to isolate the performance of individual components. Because the organizers did not provide separate scores for the face and place sub-tasks, the researchers could not accurately estimate the relative contribution of each modality to the final result.

Conclusion

The PLUMCOT submission to TRECVid 2018 underscores that for instance search, "context is king." Temporal scene segmentation proved to be the most significant factor in system performance, far outweighing the incremental gains from additional modalities like audio in the absence of proper tuning. Future work will focus on utilizing now-available data to optimize fusion parameters and exploring more robust methods for integrating voice data when visual signals are degraded.

Summary updated

BibTeX
@inproceedings{bredin2018plumcot,
  author = {Hervé Bredin and Benjamin Maurice and Ruiqing Yin and Camille Guinaudeau and Aman Behre and Claude Barras and Vivek Sharma and M. Saquib Sarfraz},
  title = {{PLUMCOT at TRECVid Instance Search 2018}},
  booktitle = {TRECVID},
  year = {2018},
}
Figure