Neuro-symbolic Architectures for Context Understanding

Alessandro Oltramari, Jonathan Francis, Cory A. Henson, Kaixin Ma, Ruwan Wickramarachchi

Introduction

Modern AI systems often struggle with "context understanding"—the human ability to fuse sensory data with background knowledge to interpret complex situations. Historically, AI has followed two divergent paths: data-driven deep learning, which excels at statistical pattern matching but remains an uninterpretable "black box," and symbolic AI, which is highly interpretable but brittle and difficult to scale. This paper explores neuro-symbolic architectures that combine the strengths of both, demonstrating that injecting explicit background knowledge into neural models improves both performance and interpretability in domains like autonomous driving and natural language processing.

Methodology

The researchers evaluated their neuro-symbolic framework across two distinct applications to test its versatility in handling context.

Autonomous Driving (AD) Scene Understanding

To help autonomous vehicles understand complex environments, the authors converted raw multimodal driving data from the NuScenes dataset into a structured "Scene Knowledge Graph" (KG). This KG maps entities (e.g., cars, pedestrians) and events (e.g., parked, moving). These discrete graphs were then translated into a continuous vector space using Knowledge Graph Embedding (KGE) algorithms like TransE and HolE, allowing neural networks to mathematically compute semantic similarities between different driving scenes.

Commonsense Question-Answering (NLP)

In the NLP domain, the authors enhanced a BERT-based Option Comparison Network (OCN) by injecting external commonsense knowledge from bases like ConceptNet and ATOMIC. Relevant relational triples (e.g., bank -> RelatedTo -> security) were extracted and converted into pseudo-sentences. These were then fused into the language model using an attention mechanism, forcing the model to "pay attention" to relevant facts before selecting an answer in the CommonsenseQA dataset.

Results and Key Insights

Semantic vs. Visual Similarity

A major finding in the autonomous driving experiments was that KGEs successfully identified scenes that were semantically similar, even if they were visually distinct. For instance, scenes containing stopped cars or specific barriers clustered together in the embedding space regardless of their pixel-level layout. This suggests a significant step toward vehicles understanding the functional "meaning" of a scene.

t-SNE visualization of scene embeddings showing semantic clustering of events (e.g., Parked Cars, Moving Cars) and features of interest.

Algorithm Performance and Domain Alignment

In the AD domain, TransE proved to be the most robust algorithm, consistently outperforming HolE across various semantic categories such as "Vehicle Event" and "Pedestrian Event."

Radar charts comparing the performance of TransE and HolE algorithms across various semantic categories and relations, highlighting TransE's superior robustness.

In the NLP task, the effectiveness of knowledge injection depended heavily on domain alignment. Injecting ConceptNet (general facts) boosted CommonsenseQA accuracy from a 64.1% baseline to 67.3%. However, injecting ATOMIC (procedural knowledge) actually degraded overall performance because its structure did not match the dataset's primary question types, though it did improve accuracy specifically for "Causes" and "Desires" questions.

Conclusion and Future Work

The study demonstrates that neuro-symbolic approaches provide inherent explainability; by observing which KG triples the model attends to, its decision-making process becomes transparent. However, limitations remain, particularly in handling logical negation and developing effective evaluation metrics for embeddings in specialized domains like AD.

Future research will focus on multi-task learning frameworks that can automatically identify the type of knowledge a question requires. Additionally, the authors propose the creation of a "global commonsense knowledge base" that aggregates multiple specialized ontologies to ensure stable performance across diverse real-world datasets.

Summary updated

Abstract

Computational context understanding refers to an agent's ability to fuse disparate sources of information for decision-making and is, therefore, generally regarded as a prerequisite for sophisticated machine reasoning capabilities, such as in artificial intelligence (AI). Data-driven and knowledge-driven methods are two classical techniques in the pursuit of such machine sense-making capability. However, while data-driven methods seek to model the statistical regularities of events by making observations in the real-world, they remain difficult to interpret and they lack mechanisms for naturally incorporating external knowledge. Conversely, knowledge-driven methods, combine structured knowledge bases, perform symbolic reasoning based on axiomatic principles, and are more interpretable in their inferential processing; however, they often lack the ability to estimate the statistical salience of an inference. To combat these issues, we propose the use of hybrid AI methodology as a general framework for combining the strengths of both approaches. Specifically, we inherit the concept of neuro-symbolism as a way of using knowledge-bases to guide the learning progress of deep neural networks. We further ground our discussion in two applications of neuro-symbolism and, in both cases, show that our systems maintain interpretability while achieving comparable performance, relative to the state-of-the-art.
BibTeX
@inproceedings{oltramari2020neuro,
  author = {Alessandro Oltramari and Jonathan Francis and Cory A. Henson and Kaixin Ma and Ruwan Wickramarachchi},
  title = {{Neuro-symbolic Architectures for Context Understanding}},
  booktitle = {Knowledge Graphs for eXplainable Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=2vouIrxsPc},
}
Figure