When Is Prior Knowledge Helpful? Exploring the Evaluation and Selection of Unsupervised Pretext Tasks from a Neuro-Symbolic Perspective

Lin-Han Jia, Si-Yu Han, Wen-Chao Hu, Jie-Jing Shao, Wen-Da Wei, Zhi Zhou, Lan-Zhe Guo, Yufeng Li

Introduction

In modern machine learning, leveraging unlabeled data through "pretext tasks" has become a standard practice. Whether through predicting image rotations in self-supervised learning (SSL), enforcing consistency in semi-supervised learning, or satisfying logical constraints in neuro-symbolic (NeSy) systems, the underlying goal remains the same: incorporating prior knowledge to enhance performance on a target objective. However, selecting an effective pretext task remains largely heuristic-driven and computationally expensive. This paper introduces a unified theoretical framework to evaluate the utility of prior knowledge before committing to full-scale training.

A Unified Theoretical Framework

The authors bridge the gap between NeSy and SSL by framing SSL assumptions as potentially "unreliable knowledge" within a neuro-symbolic context. They propose that the effectiveness of any pretext task is governed by three fundamental metrics:

  1. Knowledge Learnability: The degree to which a model can satisfy the pretext task, measured by training error on a small unlabeled subset.
  2. Knowledge Reliability: The alignment between the pretext task and the ground truth, estimated by comparing pretext predictions against a "proxy oracle" trained on a minimal labeled set.
  3. Knowledge Completeness: The extent to which the pretext task uniquely identifies the target task, preventing the model from learning "shortcuts" that satisfy the pretext objective without capturing the underlying concept.

By operationalizing these metrics, the framework allows researchers to estimate final model performance using as few as five labeled samples per class.

Experimental Methodology

The researchers validated their theory using CIFAR-10 and CIFAR-100 benchmarks. To test the framework across a diverse spectrum of prior knowledge, they constructed a suite of 115 candidate pretext tasks derived from 11 common image augmentations with varying intensities.

Table of the 11 base augmentations used to generate the 115 pretext tasks.

This extensive task suite allowed the authors to evaluate how variations in augmentation type and strength affect the learnability, reliability, and completeness of the resulting prior knowledge.

Results and Predictive Power

The core finding of the study is the remarkably high correlation between the framework's anticipated performance (calculated rapidly with minimal data) and the actual performance achieved after exhaustive training.

Correlation between anticipated performance and actual semi-supervised learning performance.

As shown in the figure above, the framework achieves a Pearson correlation coefficient of 0.785 for semi-supervised learning on CIFAR-10. Similar results were observed in self-supervised settings, with correlations reaching as high as 0.820. These results suggest that the three-pillar framework can effectively rank pretext tasks, allowing for informed selection without the need for expensive trial-and-error cycles.

Conclusion

This work provides a rigorous, quantitative foundation for understanding why certain pretext tasks succeed while others fail. By identifying learnability, reliability, and completeness as the key drivers of performance, the authors offer a path toward more efficient and principled model development. Future work may extend this framework to more complex symbolic knowledge bases or use these metrics to automatically discover optimal pretext tasks for novel datasets.

Summary updated

BibTeX
@inproceedings{jia2026when,
  author = {Lin-Han Jia and Si-Yu Han and Wen-Chao Hu and Jie-Jing Shao and Wen-Da Wei and Zhi Zhou and Lan-Zhe Guo and Yufeng Li},
  title = {{When Is Prior Knowledge Helpful? Exploring the Evaluation and Selection of Unsupervised Pretext Tasks from a Neuro-Symbolic Perspective}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=kt2CGsdTyt},
}
Figure