Introduction
Traditional machine learning models typically operate on "flat" data, where each observation is represented as a single row in a table. However, real-world data is inherently multi-relational—think of social networks, recommendation systems, or biological pathways. While Statistical Relational AI (StarAI) was developed to handle such data by merging first-order logic with probability, it often struggles with modeling complex latent features and suffers from an "over-confidence" problem as population sizes grow.
Relational Neural Networks (RelNN) address these challenges by introducing deep hidden layers into relational models. By bridging logic-based AI and deep learning, RelNN enables the learning of complex latent object properties from multi-relational data, achieving state-of-the-art performance while resolving the aggregation issues inherent in traditional StarAI models.
The Relational Modeling Challenge
StarAI models like Relational Logistic Regression (RLR) or Markov Logic Networks (MLNs) use weighted formulas to capture dependencies. For instance, a model might predict a person's happiness based on the kindness of their friends.
Despite their elegance, these models face two primary hurdles:
- Modeling Power: They often lack the "depth" to learn intermediate, non-linear features that aren't explicitly defined in the logic.
- The Aggregation Problem: As the number of related objects (the population) increases, the cumulative signal can cause probabilities to saturate at 0 or 1. This over-confidence leads to poor log-loss performance, as the model becomes too certain about noisy signals.
The RelNN Architecture
RelNNs extend Relational Logistic Regression by incorporating hidden layers, similar to standard deep neural networks.
Key Components
- Relational Linear Unit (RLL): This layer uses Weighted Formulas (WFs) to "count" occurrences of specific relational patterns. It acts as the relational equivalent of a dense layer.
- Relational Activation Layer (RAL): Non-linear activation functions (e.g., Sigmoid, ReLU) are applied to RLL outputs, allowing the model to learn complex, non-linear mappings.
- Latent Properties: RelNNs learn both rule-based hidden features (intermediate concepts) and numeric object-specific embeddings.
- Backpropagation: Training is performed using standard backpropagation, which is significantly more scalable than the Expectation-Maximization (EM) algorithms used in older StarAI frameworks.
By placing activation layers before the final output, RelNN can "squash" the influence of large populations, effectively mitigating the over-confidence problem.
Theoretical Connection to ConvNets
A key insight of the RelNN paper is its relationship to Convolutional Neural Networks (CNNs). Relational data can be viewed as a matrix where "filters" are vector-shaped and invariant to the order of rows and columns—a property known as exchangeability. This formalizes RelNN as a specialized form of CNN tailored for relational structures.
Experimental Evaluation
RelNN was benchmarked against three real-world datasets: MovieLens 1M (demographics prediction), PAKDD15 (gender prediction), and Yelp (restaurant classification).
Key Results
RelNN consistently outperformed baselines such as Matrix Factorization, RDN-Boost, and standard RLR/MLNs across multiple metrics:
- Accuracy: On MovieLens gender prediction, RelNN achieved 79.02%, significantly higher than the 70.73% achieved by traditional RLR.
- Mean Squared Error (MSE): For age prediction, RelNN reduced MSE from 156.05 to 62.7.
- Robustness: RelNN maintained stable performance as population sizes increased, whereas traditional models saw their log loss explode due to over-confidence.
Conclusion and Future Directions
RelNN represents a significant step forward in integrating deep learning with relational reasoning. By allowing for "depth" in logic-based models, it captures latent properties that shallow models miss. Future work includes automating structure learning to discover logic formulas directly from data and implementing the model in modern frameworks like PyTorch or TensorFlow to leverage GPU acceleration.
Summary updated