A Neuro-Symbolic Approach for Real-World Event Recognition from Weak Supervision

Gianluca Apriceno, Andrea Passerini, Luciano SerafiniTIME 2022 · OpenReview

TL;DR

This paper introduces a novel neuro-symbolic approach for recognizing complex, structured events (like a high jump) from video, by combining deep learning with explicit domain knowledge. This method significantly outperforms purely neural networks, especially when training data for individual atomic actions is scarce or absent, demonstrating the power of symbolic reasoning to guide learning.

What Problem Does This Paper Address?

Recognizing complex events in videos, such as sports actions, is challenging. These events are often composed of a sequence of simpler, "atomic" events (e.g., a high jump consists of a run, a jump, and a fall). While deep learning models excel at learning patterns from data, they often struggle to ensure that their predictions are consistent with known relationships between these atomic events (e.g., a fall shouldn't happen before a jump). Furthermore, obtaining detailed, frame-by-frame annotations for all atomic events in large video datasets is extremely costly and time-consuming. This paper addresses the problem of accurately recognizing both structured and atomic events, particularly in scenarios with limited or noisy supervision for the atomic components, by leveraging explicit background knowledge.

Methods

The proposed approach is a neuro-symbolic architecture that integrates a neural network with a symbolic reasoning component.

  1. Neural Network (NN) Backbone: A standard neural network (like a pre-trained I3D model) processes video frames to extract features. This network has two output heads: one for predicting structured events and another for predicting atomic events. The structured event head outputs probabilities for different event classes, while the atomic event head outputs probabilities for each atomic event occurring at each time step.
  2. Symbolic Reasoning (MILP): The core novelty lies in using a Mixed Integer Linear Programming (MILP) solver. This solver takes the probabilities from the NN and incorporates explicit, human-defined background knowledge about event structures and temporal constraints. This knowledge includes:
  3. Hard Constraints: Strict rules about the order of atomic events within a structured event (e.g., "run" must precede "jump").
  4. Soft Constraints: Rules about durations of atomic events or their combinations (e.g., "run" should be longer than "throw").
  5. End-to-End Differentiable Training: The entire system is designed to be end-to-end trainable. The MILP solver's output is used to refine the neural network's predictions, and the loss function guides the learning process. This allows the neural network to learn to produce outputs that are more amenable to symbolic reasoning, even when direct supervision for atomic events is missing. The system leverages a cost function that balances the compatibility of the NN's predictions with the symbolic constraints.

Datasets

The paper uses the Multi-THUMOS untrimmed video dataset for training and evaluation. The authors note that this dataset does not explicitly distinguish between structured and atomic events, so they define these categories and split the events accordingly. They also mention creating a validation set of 10% of the training data.

Key Insights and Ideas

  • Knowledge as a Powerful Regularizer: Explicitly encoding domain knowledge (like temporal order and duration constraints) acts as a strong regularizer for neural networks, significantly improving performance, especially in low-supervision settings.
  • Bypassing Atomic Supervision: The neuro-symbolic approach can learn to detect atomic events with remarkable accuracy even when no direct supervision is provided for them during training, relying solely on structured event supervision and background knowledge. This is a major advantage for real-world applications where detailed atomic annotations are scarce.
  • Consistency is Key: Purely neural methods can produce predictions that are individually plausible but globally inconsistent (e.g., predicting events out of order). The symbolic component enforces consistency, leading to more reliable event recognition.
  • Bridging the Gap: The approach effectively bridges the gap between sub-symbolic (neural) and symbolic reasoning, leveraging the strengths of both.

Key Results

  • Atomic Event Recognition: The neuro-symbolic approach shows substantial improvements in F1 scores for atomic event recognition compared to a fully neural baseline, especially when atomic supervision is low. For instance, performance improvements of up to 60% and 50% are reported for "release" and "throw" atomic events, respectively.
  • Zero Atomic Supervision: In the scenario with 0% atomic supervision, the fully neural approach performs close to random guessing, while the neuro-symbolic approach achieves F1 scores comparable to fully supervised settings.
  • Structured Event Recognition: While the addition of knowledge doesn't significantly improve structured event recognition when full supervision is available for them, it is crucial for enabling atomic event recognition in low-supervision scenarios.
  • Qualitative Improvements: Figure 6 demonstrates qualitative improvements, showing how the neuro-symbolic approach produces more consistent event sequences and durations compared to the fully neural baseline, which can exhibit nonsensical predictions.

Limitations and Future Work

  • Complexity of Scenarios: The authors acknowledge that a major direction for future work is to increase the complexity of the scenarios considered. This includes dealing with structured events involving multiple actors and more intricate relationships between events.
  • Scalability: They note that current neuro-symbolic frameworks can struggle with scalability when dealing with complex reasoning problems, which is a challenge they aim to address.
  • Generalizability: While tested on sports events, extending the approach to other domains with different types of structured events and knowledge representations would be valuable.

Summary updated

Abstract

Events are structured entities involving different components (e.g, the participants, their roles etc.) and their relations. Structured events are typically defined in terms of (a subset of) simpler, atomic events and a set of temporal relation between them. Temporal Event Detection (TED) is the task of detecting structured and atomic events within data streams, most often text or video sequences, and has numerous applications, from video surveillance to sports analytics. Existing deep learning approaches solve TED task by implicitly learning the temporal correlations among events from data. As consequence, these approaches often fail in ensuring a consistent prediction in terms of the relationship between structured and atomic events. On the other hand, neuro-symbolic approaches have shown their capability to constrain the output of the neural networks to be consistent with respect to the background knowledge of the domain. In this paper, we propose a neuro-symbolic approach for TED in a real world scenario involving sports activities. We show how by incorporating simple knowledge involving the relative order of atomic events and constraints on their duration, the approach substantially outperforms a fully neural solution in terms of recognition accuracy, when little or even no supervision is available on the atomic events.
BibTeX
@inproceedings{apriceno2022neuro,
  author = {Gianluca Apriceno and Andrea Passerini and Luciano Serafini},
  title = {{A Neuro-Symbolic Approach for Real-World Event Recognition from Weak Supervision}},
  booktitle = {TIME},
  year = {2022},
  url = {https://openreview.net/forum?id=jW17c5bh5c},
}
Figure