Neural Feature-Adaptation for Symbolic Predictions Using Pre-Training and Semantic Loss

Vedant Shah, Aditya Agrawal, Lovekesh Vig, Ashwin Srinivasan, Gautam Shroff, Tanmay T. VerlekararXiv 2022 · DOI · arXiv

TL;DR

This paper introduces a method to adapt pre-trained neural networks for symbolic prediction tasks, even when raw data is imperfect and lacks clear feature boundaries. By using a "semantic loss" and "abductive feedback," the system can learn to extract relevant symbols from noisy data, improving both prediction accuracy and the interpretability of the explanations. This is particularly useful for transferring symbolic knowledge to new domains without extensive re-annotation.

What Problem Does This Paper Address?

Modern AI systems often involve a duality: a "declarative" level for human understanding and interaction, and a "procedural" level for efficient computation. Neuro-symbolic systems aim to bridge this gap by combining high-level symbolic reasoning with low-level neural processing. A key challenge arises when the raw data used by the neural layer is imperfect or when the boundaries of relevant features are not obvious. Manually annotating such data for every new scenario is costly and time-consuming. Existing methods like NEUROLOG use a "semantic loss" to guide neural feature extraction using a symbolic theory, but they often rely on domain-specific pre-processing to delineate features. This paper tackles the problem of performing neuro-symbolic learning in domains where such pre-processing is difficult or impossible, and explores how pre-training can mitigate the lack of explicit feature delineation.

Methods

The core of the paper builds upon the NEUROLOG system, which uses a semantic loss function to train a neural feature extractor guided by a symbolic theory. The key innovation here is exploring scenarios without task-specific pre-processing.

  • Semantic Loss and Abductive Feedback: This mechanism allows the symbolic theory to "teach" the neural network which features are important. The neural network tries to extract symbols (feature values) from the raw data, and the semantic loss penalizes predictions that are inconsistent with the symbolic theory. Abductive feedback helps in finding explanations for the predictions.
  • Feature Adaptation: Instead of relying on pre-processing to define feature locations, the paper proposes "feature adaptation." This involves using pre-trained neural networks as a starting point for the feature extractor. The pre-training can be "task-independent," meaning it's not specific to the final symbolic task but provides a general understanding of the data.
  • Approximate Pre-training: To study the impact of pre-training quality, the authors introduce a method to create "approximate" pre-trained models by intentionally corrupting feature labels during pre-training. This allows them to systematically degrade the quality of the pre-trained feature extractor and observe its effect on the final neuro-symbolic model.

Datasets

The paper uses two main domains for experimentation:

  • Chess Problem: This is a benchmark problem from previous NEUROLOG work, involving predicting game states (safe, draw, mate) from images of a 3x3 chessboard with three pieces. The raw data consists of MNIST digits representing chess pieces.
  • Synthetic Time-Series Data: This dataset is generated to mimic aspects of the chess problem. It involves time-series data composed of combinations of 9 different shapes. The symbolic theory assigns class labels based on these shape combinations.

Key Insights and Ideas

  • Pre-processing is King, but Pre-training is a Strong Contender: When task-specific pre-processing is feasible, it leads to the best results. However, when it's not possible, using a pre-trained neural feature extractor (even an approximate one) significantly outperforms starting with a completely untrained network.
  • Semantic Loss Bridges the Gap: Even with imperfect feature extraction (i.e., correct predictions but wrong explanations), the semantic loss helps maintain high predictive accuracy. However, it doesn't guarantee correct feature identification without good initial feature representation.
  • Pre-training Mitigates Lack of Feature Delineation: The "feature adaptation" through pre-training can compensate for the absence of explicit feature boundaries in the raw data, allowing the system to learn more robust and interpretable representations.
  • Quality of Pre-training Matters: As the pre-trained feature extractor becomes more corrupted (higher 'alpha' value), the performance of the final neuro-symbolic model degrades. However, even poorly pre-trained models can still yield better explanatory fidelity than untrained models.

Key Results

  • Chess Domain: Models using task-specific pre-processing achieved the highest predictive accuracy and explanatory fidelity. For models without pre-processing, pre-training (N₁α) consistently outperformed untrained models (N₀), with performance decreasing as pre-training corruption increased.
  • Time-Series Domain: Similar trends were observed. Pre-processing yielded the best results. Without pre-processing, pre-trained models (N₁α) were superior to untrained models (N₀). The paper shows that the performance gap between pre-trained and untrained models can be significant, especially in terms of explanatory fidelity.
  • Improvement with Pre-training: For time-series data, the relative improvement from semantic loss after pre-training was most pronounced when the feature extractor quality was lower (around alpha=0.4-0.5), showing gains of 28-33% in both accuracy and fidelity.

Limitations and Future Work

  • Complexity of Theories: The paper notes that the complexity of the symbolic theory (e.g., number of features and conjuncts) can impact the effectiveness of pre-training. More complex theories might require better pre-trained feature extractors.
  • Domain Transfer: While the paper suggests that symbolic explanations could be reused across related domains through feature adaptation, this was not directly tested in the experiments.
  • Future Work: The authors suggest further exploring how pre-training can be used to transfer symbolic knowledge to new domains with varying data distributions. They also highlight the potential for improving models with even partially pre-trained neural models, especially when dealing with imperfect feature extraction.

Summary updated

Abstract

We are interested in neurosymbolic systems consisting of a high-level symbolic layer for explainable prediction in terms of human-intelligible concepts; and a low-level neural layer for extracting symbols required to generate the symbolic explanation. Real data is often imperfect meaning that even if the symbolic theory remains unchanged, we may still need to address the problem of mapping raw data to high-level symbols, each time there is a change in the data acquisition environment or equipment. Manual (re-)annotation of the raw data each time this happens is laborious and expensive; and automated labelling methods are often imperfect, especially for complex problems. NEUROLOG proposed the use of a semantic loss function that allows an existing feature-based symbolic model to guide the extraction of feature-values from raw data, using `abduction'. However, the experiments demonstrating the use of semantic loss through abduction appear to rely heavily on a domain-specific pre-processing step that enables a prior delineation of feature locations in the raw data. We examine the use of semantic loss in domains where such pre-processing is not possible, or is not obvious. We show that without any prior information about the features, the NEUROLOG approach can continue to predict accurately even with substantially incorrect feature predictions. We show also that prior information about the features in the form of even imperfect pre-training can help correct this situation. These findings are replicated on the original problem considered by NEUROLOG, without the use of feature-delineation. This suggests that symbolic explanations constructed for data in a domain could be re-used in a related domain, by `feature-adaptation' of pre-trained neural extractors using the semantic loss function constrained by abductive feedback.
BibTeX
@misc{shah2022neural,
  author = {Vedant Shah and Aditya Agrawal and Lovekesh Vig and Ashwin Srinivasan and Gautam Shroff and Tanmay T. Verlekar},
  title = {{Neural Feature-Adaptation for Symbolic Predictions Using Pre-Training and Semantic Loss}},
  year = {2022},
  eprint = {2211.16047},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.16047},
}
Figure