NeuPSL: Neural Probabilistic Soft Logic

Connor Pryor, Charles Dickens, Eriq Augustine, Alon Albalak, William Yang Wang, Lise Getoor

Abstract

In this paper, we introduce Neural Probabilistic Soft Logic (NeuPSL), a novel neuro-symbolic (NeSy) framework that unites state-of-the-art symbolic reasoning with the low-level perception of deep neural networks. To model the boundary between neural and symbolic representations, we propose a family of energy-based models, NeSy Energy-Based Models, and show that they are general enough to include NeuPSL and many other NeSy approaches. Using this framework, we show how to seamlessly integrate neural and symbolic parameter learning and inference in NeuPSL. Through an extensive empirical evaluation, we demonstrate the benefits of using NeSy methods, achieving upwards of 30% improvement over independent neural network models. On a well-established NeSy task, MNIST-Addition, NeuPSL demonstrates its joint reasoning capabilities by outperforming existing NeSy approaches by up to 10% in low-data settings. Furthermore, NeuPSL achieves a 5% boost in performance over state-of-the-art NeSy methods in a canonical citation network task with up to a 40 times speed up.

Introduction

Artificial Intelligence has traditionally been divided into two paradigms: Neural Networks, which excel at pattern recognition from raw data but lack logical reasoning and data efficiency, and Symbolic Logic, which provides robust reasoning and domain knowledge integration but struggles with raw sensory input. The "holy grail" of AI is a Neuro-Symbolic (NeSy) approach that combines both. However, existing NeSy methods face significant challenges in joint inference complexity—where reasoning over related examples becomes exponentially difficult—and joint learning, specifically the mathematical difficulty of passing gradients from symbolic engines back to neural networks.

Methods: Neural Probabilistic Soft Logic (NeuPSL)

To address these challenges, we propose two major contributions. First, we introduce NeSy Energy-Based Models (NeSy-EBMs), a unifying mathematical framework that treats neuro-symbolic systems as energy-based models. This allows for a common language to compare various NeSy approaches based on "energy functions," where lower energy signifies higher compatibility between data and logic.

Second, we present NeuPSL, a specific implementation that extends Probabilistic Soft Logic (PSL). NeuPSL utilizes "soft" logic, where truth values are continuous between 0 and 1.

NeuPSL Architecture: Integrating neural inference with symbolic reasoning through a Deep Hinge-Loss Markov Random Field.

Key features of NeuPSL include:

  • Neural Predicates: Deep neural network outputs are directly integrated into logical rules as predicates.
  • Convex Optimization: By employing Łukasiewicz logic, NeuPSL ensures the energy function is convex. This enables the use of fast optimization algorithms like ADMM (Alternating Direction Method of Multipliers) instead of slow combinatorial searches.
  • End-to-End Gradients: We demonstrate how to calculate gradients through the symbolic reasoning layer, allowing standard backpropagation to update neural network weights based on logical consistency.

Experimental Evaluation

We evaluated NeuPSL on three distinct tasks:

  1. MNIST-Addition (Overlap Variant): A benchmark requiring the model to sum digits from images without individual digit labels. We introduced "overlap," where images are reused across sums to test joint reasoning.
  2. Visual Sudoku Classification: 4x4 Sudoku puzzles composed of MNIST images. The model must classify puzzles as valid without pre-labeled digits.
  3. Citation Networks (Citeseer and Cora): Standard node classification benchmarks testing scalability on large relational graphs.

Results

NeuPSL demonstrates significant improvements across all benchmarks. It achieved upwards of a 30% improvement over independent neural networks. In low-data settings on the MNIST-Addition task, NeuPSL outperformed existing NeSy methods by up to 10%.

Categorical accuracy on MNIST Addition tasks 1 and 2, comparing NeuPSL against CNN, LTN, and DeepProbLog across varying numbers of puzzles and unique images.

In the Visual Sudoku task, while pure neural models failed to generalize, NeuPSL reached 70% accuracy with only 16 puzzles. Furthermore, on citation network tasks, NeuPSL was up to 40 times faster than DeepStochLog while maintaining a 5% lead in accuracy. These results highlight the power of logic as a "data multiplier" and the efficiency of convex optimization in NeSy systems.

Conclusion

NeuPSL provides a scalable and accurate framework for neuro-symbolic AI. By relaxing logic into a continuous, convex space, it transforms complex search problems into efficient optimization tasks. While it requires predefined logical rules, its ability to perform joint learning and inference makes it a powerful tool for data-scarce and relational domains. Future work will explore alternative loss functions and broader reasoning tasks like natural language understanding.

Summary updated

Abstract

In this paper, we introduce Neural Probabilistic Soft Logic (NeuPSL), a novel neuro-symbolic (NeSy) framework that unites state-of-the-art symbolic reasoning with the low-level perception of deep neural networks. To model the boundary between neural and symbolic representations, we propose a family of energy-based models, NeSy Energy-Based Models, and show that they are general enough to include NeuPSL and many other NeSy approaches. Using this framework, we show how to seamlessly integrate neural and symbolic parameter learning and inference in NeuPSL. Through an extensive empirical evaluation, we demonstrate the benefits of using NeSy methods, achieving upwards of 30% improvement over independent neural network models. On a well-established NeSy task, MNIST-Addition, NeuPSL demonstrates its joint reasoning capabilities by outperforming existing NeSy approaches by up to 10% in low-data settings. Furthermore, NeuPSL achieves a 5% boost in performance over state-of-the-art NeSy methods in a canonical citation network task with up to a 40 times speed up.
BibTeX
@inproceedings{pryor2023neupsl,
  author = {Connor Pryor and Charles Dickens and Eriq Augustine and Alon Albalak and William Yang Wang and Lise Getoor},
  title = {{NeuPSL: Neural Probabilistic Soft Logic}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=5gO71LRGncmB},
}
Figure