Learning from Rules Generalizing Labeled Exemplars

Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, Sunita Sarawagi

In many machine learning applications, labeled data is not readily available and must be collected via painstaking human supervision. Researchers from IIT Bombay have developed a novel "rule-exemplar" framework that combines the speed of rule-based labeling with the precision of manual instance labeling. By treating rules as noisy generalizations of specific examples and using a unique "soft implication loss," their model (ImplyLoss) automatically denoises rules and significantly outperforms standard semi-supervised and weak-supervision methods.

The Problem: The Rock and the Hard Place

When building datasets, practitioners are often stuck between two suboptimal choices:

  1. Instance Labeling: High quality and precise, but incredibly slow, expensive, and difficult to scale.
  2. Rule-Based Labeling: Fast and scalable, but inherently "noisy." Humans tend to over-generalize. For example, a rule like Contains 'highly recommend' → Positive works most of the time, but fails on edge cases like, "I would highly recommend this phone if it weren't for the poor service."

The challenge lies in harnessing the efficiency of rules without letting their inherent noise sabotage the model. Current methods often treat rules and labels as independent sources of truth, failing to capture the systematic ways in which rules fail.

The Solution: Rule-Exemplar Pairs and ImplyLoss

To address this, the authors propose a unique supervision paradigm: Rule-Exemplar pairs. Instead of writing a rule in a vacuum, a human labels a specific instance (the exemplar) and then generalizes it into a rule.

Rules as generalizations around exemplars

The core "aha moment" is viewing rule noise not as random error, but as systematic over-generalization from a specific "safe" neighborhood (the exemplar) into noisier boundary regions.

To train on this coupled data, they introduce ImplyLoss, which utilizes three main components:

  • Classification Network ($P_\theta$): The standard model trying to predict the true label $y$.
  • Rule Network ($P_\phi$): A secondary network that learns a latent "coverage" variable ($r$). It predicts whether a specific rule is actually valid for a given instance.
  • Soft Implication Loss: This is the "secret sauce." It enforces a logical constraint: If the rule network says the rule is valid ($r=1$), then the classifier must agree with the rule's label.

Instead of a hard "if-then" statement, ImplyLoss uses a differentiable log-probability function that "withdraws" supervision if the rule network suspects a rule has over-generalized to a specific instance. This allows the model to jointly denoise the rules while training the classifier. The classifier provides a sanity check for the rules, and the denoised rules provide high-quality pseudo-labels to train the classifier on unlabeled data.

Key Results and Denoising Effectiveness

The authors evaluated their method across five diverse tasks, including Question Classification (TREC-6), MIT-R (a complex slot-filling task), SMS Spam, YouTube Spam, and Census Income. They demonstrated that even a small number of rules can cover thousands of unlabeled instances.

The results showed massive accuracy gains. On the Question Classification task, ImplyLoss provided an 11.7 point gain over a model trained only on labeled exemplars. For context, the next best existing method only managed a 0.3 point gain.

Furthermore, the model proved remarkably adept at identifying bad rule applications.

Denoised precision across datasets

As seen across multiple datasets, the latent denoising process drastically improved rule reliability. In the Question dataset, the precision of the rules jumped from 64% (raw) to 98% (denoised). The model also exhibited strong robustness; even when high-quality rules were intentionally removed to test the model on "messy" supervision, ImplyLoss remained more stable than meta-learning approaches like Learning to Reweight (L2R).

Limitations and Future Directions

While highly effective, the approach has limitations. Manual rule engineering, though more efficient than labeling thousands of individual points, still requires human expertise and time (e.g., it took 90 minutes to craft 68 rules for the Question task). Additionally, the current implementation relies heavily on text-based rules like regular expressions. Applying this algorithm to computer vision or audio would require developing new ways for humans to "generalize" exemplars in those domains.

Ultimately, this rule-exemplar framework presents a highly promising direction for low-resource applications where human supervision is the primary bottleneck, bridging the gap between human logic and machine learning labels.

Summary updated

Abstract

In many applications labeled data is not readily available, and needs to be collected via pain-staking human supervision. We propose a rule-exemplar method for collecting human supervision to combine the efficiency of rules with the quality of instance labels. The supervision is coupled such that it is both natural for humans and synergistic for learning. We propose a training algorithm that jointly denoises rules via latent coverage variables, and trains the model through a soft implication loss over the coverage and label variables. The denoised rules and trained model are used jointly for inference. Empirical evaluation on five different tasks shows that (1) our algorithm is more accurate than several existing methods of learning from a mix of clean and noisy supervision, and (2) the coupled rule-exemplar supervision is effective in denoising rules.
BibTeX
@inproceedings{awasthi2020learning,
  author = {Abhijeet Awasthi and Sabyasachi Ghosh and Rasna Goyal and Sunita Sarawagi},
  title = {{Learning from Rules Generalizing Labeled Exemplars}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=IRysN7XUFaS},
}
Figure