Automatic Rule Induction for Efficient Semi-Supervised Learning

Reid Pryzant, Ziyi Yang, Yichong Xu, Chenguang Zhu, Michael Zeng

Abstract

Large language models (LLMs) like BERT are powerful but notoriously data-hungry and opaque. Automatic Rule Induction (ARI) bridges the gap between deep learning and symbolic reasoning by automatically discovering human-readable rules and integrating them into pretrained transformers. This hybrid approach significantly boosts performance in data-scarce scenarios and provides a clear window into model decision-making processes.

Introduction: The Neuro-Symbolic Gap

Modern NLP faces two primary hurdles: data scarcity and a lack of interpretability. Pretrained models often struggle to generalize when labeled data is minimal (the "few-shot" setting). Furthermore, transformers are correlational engines that may adopt "spurious" patterns that fail in real-world applications.

While symbolic rules (e.g., "If the word 'free' appears, it is likely spam") are robust and explainable, manually crafting them is a tedious burden. ARI aims to remove the human from the loop, making rule-based augmentation as scalable as deep learning itself by automating the discovery and integration of these symbolic compasses.

Methodology: The ARI Framework

The ARI framework operates through a three-stage pipeline designed to extract and utilize symbolic knowledge without manual intervention.

  1. Rule Induction: The system trains "low-capacity" models, such as decision trees or simple linear regressions, on small labeled datasets. The simplicity of these models allows their learned artifacts—like high-weight keywords or specific decision paths—to be converted into symbolic rules. To ensure these rules capture meaningful signals rather than common "stop words," a PCA-based subtraction method filters out uninformative information.
  2. Rule Integration: Rather than hard-coding these rules, ARI employs an attention-based aggregation layer. This layer dynamically evaluates the input text to decide the relative trust placed in the backbone transformer versus the specific symbolic rules that "fired" for that input.
  3. Self-Training: The system utilizes a "teacher-student" loop where the rule-augmented model (the teacher) generates labels for a large pool of unlabeled data. these labels are then used to further refine the backbone transformer (the student).
The ARI framework pipeline: Rule induction from labeled data (top) and integration into the transformer backbone via an attention-based aggregator (bottom).

Experimental Evaluation

The authors evaluated ARI across nine diverse classification and relation extraction tasks to ensure broad applicability:

  • General/News: AGNews, TREC.
  • Biomedical: CDR (chemical-disease relations), ChemProt (protein-chemical relations).
  • Reviews/Web: IMDB, Youtube, and SMS spam detection.
  • Scientific: SciCite (citation intent).
  • Relation Extraction: SemEval.

Results and Key Insights

ARI consistently outperformed standard BERT fine-tuning and popular semi-supervised baselines such as PET and standard Self-Training. Across all nine tasks, the ARI variant utilizing PCA and decision trees achieved an average F1/Accuracy of 79.05, compared to BERT's 76.72.

Performance comparison on ChemProt and Youtube datasets, demonstrating that ARI (NSRI) provides the most significant gains when labeled data is extremely scarce (2-5%).

Key Findings:

  • Performance in Scarcity: ARI showed its greatest gains when labeled data was minimal (2-5% of the dataset). As labeled data increased to 40%, the gap narrowed as the transformer gained enough data to learn patterns internally.
  • Rules for "Hard" Examples: The model learns to rely more heavily on symbolic rules for cases where the transformer's internal confidence is low.
  • Automated Superiority: ARI-induced rules outperformed ASTRA, a state-of-the-art method using manually crafted expert rules, suggesting that automated induction can find higher-precision triggers than human annotators.
  • Interpretability: Because rules are integrated via attention, users can verify the specific rule (e.g., if 'claim' in x predict SPAM) that drove a prediction.

Limitations and Future Work

Despite its strengths, ARI's performance is sensitive to the hyperparameters of the "low-capacity" induction models. Additionally, while rules show some transferability between similar tasks, they are generally optimized for the specific data from which they were induced. Future research may explore "causal mechanisms" for rule induction to ensure rules reflect true logical relationships rather than sophisticated correlations.

By automating symbolic rule generation, ARI provides a blueprint for models that are more accurate in low-data settings and significantly easier for humans to trust and verify.

Summary updated

BibTeX
@inproceedings{pryzant2022automatic,
  author = {Reid Pryzant and Ziyi Yang and Yichong Xu and Chenguang Zhu and Michael Zeng},
  title = {{Automatic Rule Induction for Efficient Semi-Supervised Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.3},
}
Figure