Introduction
Interactive video retrieval systems are essential for navigating thousands of hours of video content based on diverse user queries, including text, images, and audio. In competitive environments like the Video Browser Showdown (VBS), the challenge lies in balancing retrieval accuracy with extreme time constraints. Previous iterations, such as Fusionista and Fustar, utilized state-of-the-art AI models for tasks like keyframe extraction and question answering. However, these heavy models created significant computational bottlenecks when scaled to multi-terabyte datasets. Fusionista2.0 addresses these limitations by re-engineering the pipeline for maximum efficiency without sacrificing retrieval effectiveness.
System Architecture
Fusionista2.0 is divided into two primary phases: Indexing and Searching. The indexing phase focuses on rapid data preparation and feature extraction, while the searching phase utilizes a multi-modal back-end to process user queries in real-time.
Data Preparation and Extraction
To optimize the indexing stage, the authors replaced heavy AI-based scene detection with a deterministic ffmpeg workflow. This approach extracts intra-coded frames as structural keyframes, significantly reducing memory demands. For feature extraction, the system employs an ensemble of two CLIP variants (CLIP-Sig400M and CLIP-ViT-5B) with an empirically tuned weighting ($\alpha=0.7$) to better capture semantic nuances.
Lightweight Components
Efficiency is further gained by swapping standard models for streamlined alternatives:
- OCR & ASR: Vintern-1B-v3.5 is used for multilingual text recognition, while
faster_whisperaccelerates audio transcription by 4x, focusing on speed for ambient-heavy datasets. - Vision-Language Models (VLMs): Massive models were replaced with compact versions ($\le$ 1B parameters) to handle frequent, simple queries like object counting or color recognition instantly.
Interactive Reranking and UI/UX
A novel post-processing pipeline was introduced to refine search results. Using GPT-4o, the system generates clarifying yes/no questions based on the user's query. A lightweight VLM then evaluates retrieved candidates against these questions to automatically rerank them.
To improve the user experience, the frontend was migrated to Vite, incorporating virtual scrolling and sidebar navigation. These changes, combined with WCAG accessibility compliance, minimize redundant browsing and streamline the search process for the human operator.
Results and Discussion
The transition to lightweight components yielded a 75% reduction in overall retrieval time. The authors validated their choice of models using a custom benchmark of 200 video-question pairs, focusing on counting, image information, and video information.
As shown in the performance table, the chosen model (InternVL-1B-ffn6-Seq) achieved high accuracy (84% for counting) with an average inference time of just 4.80 seconds. While these lightweight models may lack the deep reasoning of 7B+ parameter models, their speed allows for a more fluid interactive search where the human remains in the loop for complex decision-making.
Conclusion
Fusionista2.0 demonstrates that for real-time, large-scale video retrieval, deterministic preprocessing and sub-1B parameter models can outperform heavier state-of-the-art alternatives in practical utility. By reducing computational overhead and optimizing the user interface, the system provides a robust framework for the upcoming VBS 2026 competition and broader real-world applications.
Summary updated