Robust Data Programming with Precision-guided Labeling Functions

Oishik Chatterjee, Ganesh Ramakrishnan, Sunita SarawagiAAAI 2020 · OpenReview

Introduction

Supervised machine learning models require vast amounts of labeled data, which often creates a significant bottleneck in development. Data Programming, popularized by systems like Snorkel, addresses this by allowing experts to write "Labeling Functions" (LFs)—short scripts that provide noisy, "weak" labels. A generative model then aggregates these noisy votes into probabilistic labels for training.

However, traditional data programming faces two major challenges. First, most LFs are discrete (e.g., keyword-based), which limits recall by missing semantic nuances. Second, training the generative model is often unsupervised and highly unstable. Without ground truth, models are sensitive to initialization and training duration, frequently converging to suboptimal states where they ignore expert intent.

Example Labeling Functions (LFs) and dictionaries for the Spouse relation extraction task.

Methodology: The CAGE Framework

Researchers from IIT Bombay introduced CAGE (Continuous And quality Guided labEling) to overcome these hurdles through two primary innovations:

  1. Continuous Labeling Functions: Unlike binary LFs, CAGE allows functions to output continuous scores (0 to 1) based on metrics like cosine similarity of word embeddings. This enables the model to recognize semantic similarities (e.g., "wed" vs. "married") that discrete rules miss. These scores are modeled using a Beta distribution within a graphical framework to capture match confidence.
  2. Quality Guides ($q$): To stabilize training, CAGE incorporates user-provided estimates of an LF's precision (e.g., "this rule is ~90% accurate"). These guides act as a data-independent regularizer using KL divergence, forcing the model's learned parameters to remain aligned with the expert's intuitive beliefs.

Experimental Results

CAGE was evaluated across several datasets, including Spouse (relation extraction), SMS Spam, and CDR (Chemical-Disease Relation). The framework demonstrated superior performance and, crucially, significantly higher training stability compared to the Snorkel baseline.

Performance and Stability

In the Spouse task, CAGE achieved an F1 score of 0.58, a substantial improvement over Snorkel's 0.41. The use of continuous LFs led to higher recall by generalizing beyond exact keyword matches.

F1 score performance and stability across training epochs for Spouse (a), SMS (b), and CDR (c) datasets. Note the significant fluctuations in the Snorkel baseline compared to CAGE.

As shown in the figures above, Snorkel's performance often fluctuates wildly or drops sharply as training progresses (notably in the Spouse and CDR tasks). In contrast, CAGE's quality guides act as a "north star," ensuring the model remains robust to initialization and prevents the unpredictable swings in accuracy often seen in unsupervised learning. This stability makes CAGE much more suitable for production environments where labeled validation data for early stopping is unavailable.

Conclusion

CAGE evolves the data programming paradigm by introducing semantic flexibility through continuous labeling and training stability through quality guides. Even crude user estimates of LF quality are sufficient to stabilize the model effectively. Future work may explore more complex dependency modeling between continuous LFs and automated methods for refining quality guides during training.

Summary updated

BibTeX
@inproceedings{chatterjee2020robust,
  author = {Oishik Chatterjee and Ganesh Ramakrishnan and Sunita Sarawagi},
  title = {{Robust Data Programming with Precision-guided Labeling Functions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=-eqpKpoagg},
}
Figure