Towards Neuro-Symbolic Video Understanding

Minkyu Choi, Harsh Goel, Mohammad Omama, Yunhao Yang, Sahil Shah, Sandeep P. Chinchali

TL;DR

This paper introduces Neuro-Symbolic Video Search with Temporal Logic (NSVS-TL), a novel approach that decouples visual perception from temporal reasoning to enable robust long-term event understanding in videos. By combining state-of-the-art vision models with formal temporal logic and probabilistic automata, NSVS-TL significantly outperforms existing video-language models on complex, long-duration scene identification tasks.

What Problem Does This Paper Address?

The explosion of video data from sources like YouTube and autonomous vehicles presents a significant challenge for extracting meaningful information, especially for tasks requiring long-term temporal reasoning. Current state-of-the-art video-language foundation models, while adept at short-term semantic understanding, struggle with identifying complex events that unfold over extended periods or involve rare occurrences. This limitation stems from their tendency to fuse per-frame perception and temporal reasoning into a single, monolithic deep network, hindering their ability to capture intricate temporal dependencies. The authors aim to bridge this gap by proposing a system that explicitly separates these two crucial aspects of video understanding.

Methods

The core of NSVS-TL lies in its neuro-symbolic architecture, which disentangles visual perception from temporal reasoning.

  1. Neural Perception: The system first employs off-the-shelf vision or vision-language models (e.g., YOLOv8, CLIP) to extract semantic information from individual video frames. This step focuses on identifying "atomic propositions" relevant to a given query.
  2. Probabilistic Automaton Construction: Based on the detected atomic propositions, a probabilistic automaton (PA) is dynamically constructed. This PA models the temporal evolution of events within the video, with states representing combinations of propositions and transitions governed by probabilities.
  3. Temporal Logic Specification: User queries are translated into formal temporal logic (TL) specifications. These specifications, using operators like "Always" (□), "Eventually" (◊), and "Until" (U), precisely define the temporal relationships and conditions for identifying a scene of interest.
  4. Formal Verification: The constructed PA is then subjected to formal verification using model checking techniques (specifically, the STORM tool). This process rigorously checks if the PA satisfies the TL specification, thereby identifying and validating the desired scenes.

This decoupling allows for a more interpretable and robust approach to temporal reasoning, especially for long videos where traditional end-to-end models falter.

Datasets

The authors introduce the Temporal Logic Video (TLV) datasets, which include:

  • Synthetic TLV Datasets: Created by combining images from COCO and ImageNet datasets to generate a wide range of TL specifications.
  • Real TLV Datasets: Annotated autonomous vehicle (AV) datasets, specifically Waymo and NuScenes, using ground truth object labels to create TL specifications.

These datasets are designed to evaluate temporal reasoning capabilities across various complexities and video durations.

Key Insights and Ideas

  • Decoupling Perception and Reasoning is Crucial: The paper strongly argues that the failure of current models in long-term temporal reasoning is due to the conflation of visual perception and temporal reasoning. Separating these allows for more effective and interpretable analysis.
  • Temporal Logic as a Powerful Query Language: TL provides a formal and expressive way to define complex temporal relationships, enabling precise scene retrieval that goes beyond simple keyword matching.
  • Probabilistic Automata for Dynamic Modeling: The use of PAs allows for a flexible and probabilistic representation of video dynamics, which can be dynamically built and verified against TL specifications.
  • Formal Verification Guarantees: Employing model checking provides a rigorous guarantee that the identified scenes indeed satisfy the specified temporal logic conditions.

Key Results

  • Improved F1 Scores: NSVS-TL achieved a 9-15% improvement in F1 score for complex event identification compared to benchmarks using GPT-4 for reasoning on Waymo and NuScenes datasets.
  • Robustness to Video Length: The system demonstrates consistent performance across varying video lengths, outperforming LLM-based benchmarks that show significant performance degradation with longer videos (e.g., >500 seconds for GPT-3.5, >1000 seconds for GPT-4).
  • Superiority in Complex Scenarios: NSVS-TL outperforms LLM-based baselines on multi-event scenarios (e.g., "Event A until B," "Event A and B until C"), highlighting its advantage in handling intricate temporal dependencies.
  • Impact of Perception Models: The choice of neural perception model (e.g., YOLOv8x, Grounding Dino, CLIP) does influence performance, underscoring the importance of accurate per-frame semantic understanding.

Limitations and Future Work

  • Per-Frame Perception Limitation: The current NSVS-TL relies on per-frame neural perception, which may struggle with scenarios requiring the understanding of multi-frame semantics or complex sequential events (e.g., "man falling from a horse").
  • Future Work: The authors suggest extending the system to interpret multi-frame events, thereby enabling it to handle more complex video understanding tasks.

Summary updated

Abstract

The unprecedented surge in video data production in recent years necessitates efficient tools to extract meaningful frames from videos for downstream tasks. Long-term temporal reasoning is a key desideratum for frame retrieval systems. While state-of-the-art foundation models, like VideoLLaMA and ViCLIP, are proficient in short-term semantic understanding, they surprisingly fail at long-term reasoning across frames. A key reason for this failure is that they intertwine per-frame perception and temporal reasoning into a single deep network. Hence, decoupling but co-designing the semantic understanding and temporal reasoning is essential for efficient scene identification. We propose a system that leverages vision-language models for semantic understanding of individual frames and effectively reasons about the long-term evolution of events using state machines and temporal logic (TL) formulae that inherently capture memory. Our TL-based reasoning improves the F1 score of complex event identification by $9-15$\%, compared to benchmarks that use GPT-4 for reasoning, on state-of-the-art self-driving datasets such as Waymo and NuScenes.
BibTeX
@inproceedings{choi2024towards,
  author = {Minkyu Choi and Harsh Goel and Mohammad Omama and Yunhao Yang and Sahil Shah and Sandeep P. Chinchali},
  title = {{Towards Neuro-Symbolic Video Understanding}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-73229-4\_13},
}
Figure