Inducing Symbolic Rules from Entity Embeddings using Auto-encoders

Thomas Ager, Ondrej Kuzelka, Steven Schockaert

Introduction

Vector space embeddings are powerful tools for capturing semantic relationships in natural language processing, yet they remain notoriously opaque. While these models can identify that two entities are semantically related, they lack the ability to explain the underlying logic in human-readable terms. This research addresses the "black box" nature of neural embeddings by proposing a pipeline to extract symbolic "If-Then" rules from unstructured text, bridging the gap between high-dimensional vector spaces and human-readable logic.

Methodology

The authors propose a multi-step framework to transition from raw text to interpretable symbolic logic:

  1. Initial Embedding: Using a collection of documents (e.g., IMDB movie reviews), the system generates an initial vector space via Positive Pointwise Mutual Information (PPMI) and Multidimensional Scaling (MDS).
  2. Identifying Interpretable Directions: Linear Support Vector Machines (SVMs) are trained to identify specific "directions" in the vector space that correspond to human-readable properties (nouns or adjectives). These directions allow for the ranking of entities based on the presence of a property, such as "Gore" or "Tannic."
  3. Hierarchical Abstraction via SDAEs: To move from specific terms to general concepts, the embeddings are processed through Stacked Denoising Auto-encoders (SDAEs). The denoising process—adding Gaussian noise to inputs—is critical as it forces the network to ignore local variances and capture broader, more stable semantic commonalities across layers.
  4. Rule Induction: Finally, the JRip propositional rule learner is employed to map relationships between layers. By treating properties in one layer as features to predict properties in a more abstract layer, the system induces rules like: IF Scares (Layer 2) AND Blood (Layer 2) THEN Gore (Layer 3). To facilitate this, continuous rankings are discretized into percentiles.

Key Results

The evaluation focused on a dataset of 15,000 IMDB movies and their associated reviews. The results demonstrate a clear progression of abstraction through the auto-encoder layers.

Comparison of keyword clusters across layers for standard and denoising auto-encoders.

As shown in the table above, Layer 1 typically captures specific, low-level terms (e.g., "terror," "horrific"). By Layer 4, the denoising auto-encoder successfully distills these into broader, more abstract concepts such as "society," "government," and "relationship." This progression indicates that the SDAE acts as an effective "generalizer" compared to standard auto-encoders, which tend to maintain more literal or superficial groupings.

Furthermore, the system successfully induced logical rules that align with human domain knowledge. For instance, the rule IF Emotions (Layer 3) AND Journey (Layer 3) THEN Adventure (Layer 4) demonstrates the model's ability to synthesize high-level genre classifications from constituent semantic themes. The authors noted that as layers become more compressed, the resulting rules become simpler and achieve higher scores, suggesting a successful distillation of domain knowledge.

Discussion and Limitations

While the method provides a significant step toward interpretable AI, it faces certain challenges. The discretization of continuous rankings into binary percentiles for rule induction inevitably leads to some information loss, potentially reducing the accuracy of the extracted rules compared to the original neural model. Additionally, label ambiguity remains a concern; automated label selection based on Kappa scores can occasionally produce confusing results if a term has multiple semantic meanings (e.g., "Mammoth" as an animal vs. a size descriptor).

Conclusion

This work demonstrates that denoising auto-encoders can be leveraged to uncover the latent hierarchical structure of entity embeddings. By mapping these structures to symbolic rules, the researchers provide a pathway for human experts to verify and explain the logic of complex neural models, balancing the power of vector representations with the necessity of interpretability.

Summary updated

BibTeX
@inproceedings{ager2016inducing,
  author = {Thomas Ager and Ondrej Kuzelka and Steven Schockaert},
  title = {{Inducing Symbolic Rules from Entity Embeddings using Auto-encoders}},
  booktitle = {NeSy@HLAI},
  year = {2016},
}
Figure