A Scalable Reasoning and Learning Approach for Neural-Symbolic Stream Fusion

Danh Le Phuoc, Thomas Eiter, Anh Lê Tuán

Abstract

In autonomous driving and smart city applications, real-time perception must be augmented with logical reasoning to handle complex scenarios like occlusions and sensor noise. This paper presents Semantic Stream Reasoning (SSR), a neural-symbolic framework that combines the perception capabilities of Deep Neural Networks (DNNs) with the structured logic of Answer Set Programming (ASP). By introducing an incremental reasoning algorithm and a parallel learning approach, SSR overcomes the traditional "grounding bottleneck" of symbolic solvers. Experimental results on the KITTI and AI City Challenge datasets show that SSR significantly improves tracking accuracy (MOTA and IDF1) while maintaining real-time throughput of 15–30 FPS.

Introduction

Multi-object tracking (MOT) is a cornerstone of autonomous systems, yet traditional pipelines often rely on manually tuned, brittle heuristics. While symbolic logic offers explainability and flexibility, it has historically struggled with scalability, often failing to meet the 30-FPS requirement of real-time video streams due to the computational overhead of logic grounding. This research addresses this gap by proposing a scalable neural-symbolic approach that treats stream fusion as an incremental reasoning problem.

Methodology

The SSR framework bridges the gap between raw sensor data and high-level reasoning through several key innovations.

System architecture of the Semantic Stream Reasoning (SSR) framework.
  1. Semantic Stream Reasoning (SSR): The system replaces hard-coded logic with weighted logic rules. These rules, such as temporal consistency of object locations, are expressed within the Linear Analytic Reasoning over Streams (LARS) framework.
  2. Incremental Reasoning: To achieve real-time performance, the authors developed a "multi-shot" solving algorithm. Instead of re-evaluating the entire state for every frame, the solver reuses previous computations and updates only the changes in the semantic stream.
  3. Parallel Learning: Rule weights are optimized using Maximum Likelihood Estimation (MLE). A parallel learning algorithm processes data in mini-batches across multiple threads to accelerate the training phase.

Experimental Evaluation

The framework was evaluated on two major benchmarks: the KITTI autonomous driving dataset and the AI City Challenge (AIC) traffic surveillance dataset. These datasets provide diverse scenarios with varying object densities and occlusion levels.

Key Results

SSR demonstrates that symbolic logic can be both accurate and efficient enough for production environments.

Performance comparison of SSR against SORT and DSORT baselines.
  • Accuracy: SSR achieved a 10–15% improvement in MOTA and IDF1 scores over the SORT baseline and outperformed DeepSORT (DSORT) by 5–7%.
  • Throughput: The system maintained a processing speed of 15–30 FPS. Notably, SSR is 4–6 times faster than DeepSORT, which often struggles with computational latency.
  • Efficiency: Logic reasoning accounted for only 41% of total processing time, indicating that the incremental solver successfully mitigated the symbolic grounding bottleneck.

Conclusion

The SSR framework proves that neural-symbolic integration is a viable path for real-time stream fusion. By transforming heuristics into learnable logic rules and optimizing the reasoning process for incremental updates, the authors achieved state-of-the-art performance without sacrificing speed. Future work will focus on reducing the learning time for rule weights and exploring end-to-end training where symbolic feedback directly informs DNN perception models.

Summary updated

Abstract

Driven by deep neural networks (DNN), the recent development of computer vision makes vision sensors such as stereo cameras and Lidars ubiquitous in autonomous cars, robotics and traffic monitoring. However, a traditional DNN-based data fusion pipeline like object tracking has to hard-wire an engineered set of DNN models to a fixed processing logic, which makes it difficult to infuse new models to that pipeline. To overcome this, we propose a novel neural-symbolic stream reasoning approach realised by semantic stream reasoning programs which specify DNN-based data fusion pipelines via logic rules with learnable probabilistic degrees as weights. The reasoning task over this program is governed by a novel incremental reasoning algorithm, which lends itself also as a core building block for a scalable and parallel algorithm to learn the weights for such program. Extensive experiments with our first prototype on multi-object tracking benchmarks for autonomous driving and traffic monitoring show that our flexible approach can considerably improve both accuracy and processing throughput compared to the DNN-based counterparts.
BibTeX
@inproceedings{phuoc2021scalable,
  author = {Danh Le Phuoc and Thomas Eiter and Anh Lê Tuán},
  title = {{A Scalable Reasoning and Learning Approach for Neural-Symbolic Stream Fusion}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i6.16633},
}
Figure