Introduction
The explosion of surveillance and mobile camera data has created a massive backlog of video that is difficult to search efficiently. Current video search systems generally fall into two categories, each with a significant drawback. Query-Agnostic (QA-index) methods are fast because they index videos in advance, but they are limited to predefined categories, failing on complex descriptions like "large black SUV in the intersection." Conversely, Query-Dependent (QD-search) methods offer flexibility but suffer from high latency because they must re-process raw video for every new query.
The LOVO Architecture
LOVO (compLex Object queries in large-scale VideO) bridges this gap by decoupling video processing from query execution. The system operates in three main phases:
- One-Time Visual Indexing: LOVO uses a Vision Transformer (ViT) to extract "visual embeddings" from key frames. These embeddings serve as compact mathematical summaries of the visual content. Crucially, it also predicts bounding boxes for these patches, allowing for open-vocabulary search without predefined labels.
- Vector Database Management: These embeddings are stored in a Milvus vector database. To maintain performance at scale, LOVO employs Product Quantization (PQ) for data compression and an Inverted Multi-Index for rapid Approximate Nearest Neighbor (ANN) searches.
- Two-Stage Search Strategy: When a user enters a query, it is converted into a text embedding. The system first performs a "Fast Search" to retrieve candidate frames. It then applies a "Cross-Modality Rerank" using a Transformer model that performs cross-attention between the text and visual features, ensuring high precision for detailed natural language queries.
Key Results and Performance
Evaluated across diverse datasets like Cityscapes and Bellevue Traffic, LOVO demonstrates significant improvements in both speed and accuracy.
- Efficiency: LOVO achieves search speeds up to 85x faster than QD-search baselines and total execution times 22x faster than existing methods like MIRIS. As shown in the runtime analysis, LOVO maintains performance where traditional QA-index methods fail to support complex queries entirely.
- Accuracy: By utilizing patch-level granularity and a deep reranking stage, LOVO consistently outperforms specialized vision-language models (like ZELDA and VISA) in Average Precision (AveP), particularly for queries involving spatial relationships (e.g., "side by side").
- Scalability: Unlike linear-scaling traditional methods, LOVO’s search time remains nearly constant as dataset size grows, making it suitable for truly large-scale video analytics.
Conclusion
LOVO effectively solves the "speed vs. flexibility" trade-off in video search. By indexing visual features once and leveraging a two-stage retrieval process, it enables high-accuracy, open-vocabulary queries at a fraction of the computational cost of previous state-of-the-art systems.
Summary updated