Firebolt: Weak Supervision Under Weaker Assumptions

Zhaobin Kuang, Chidubem G. Arachie, Bangyong Liang, Pradyumna Narayana, Giulia DeSalvo, Michael S. Quinn, Bert Huang, Geoffrey Downs, Yang YangAISTATS 2022 · OpenReview

Abstract

Modern machine learning demands a large amount of training data. Weak supervision is a promising approach to meet this demand. It aggregates multiple labeling functions (LFs)–noisy, user-provided labeling heuristics—to rapidly and cheaply curate probabilistic labels for large-scale unlabeled data. However, standard assumptions in weak supervision—such as user-specified class balance, similar accuracy of an LF in classifying different classes, and full knowledge of LF dependency at inference time—might be undesirable in practice. In response, we present Firebolt, a new weak supervision framework that seeks to operate under weaker assumptions. In particular, Firebolt learns the class balance and class-specific accuracy of LFs jointly from unlabeled data. It carries out inference in an efficient and interpretable manner. We analyze the parameter estimation error of Firebolt and characterize its impact on downstream model performance. Furthermore, we show that on five publicly available datasets, Firebolt outperforms a state-of-the-art weak supervision method by up to 5.8 points in AUC. We also provide a case study in the production setting of a tech company, where a Firebolt-supervised model outperforms the existing weakly-supervised production model by 1.3 points in AUC and speedup label model training and inference from one hour to three minutes.

Introduction

Modern machine learning models are increasingly data-hungry, yet manual labeling remains a significant bottleneck. Weak Supervision (WS) has emerged as a powerful paradigm to address this by programmatically generating training data. In WS, users provide a set of "Labeling Functions" (LFs)—noisy, heuristic rules or patterns—which are then aggregated to produce probabilistic labels for large unlabeled datasets.

Labeling Functions serve as the noisy input heuristics for weak supervision.

Despite its success, existing WS frameworks like Snorkel often rely on restrictive assumptions. These include requiring users to specify the class balance of the unlabeled data, assuming that an LF's accuracy is uniform across different classes, and needing a complete map of LF dependencies for efficient inference. Firebolt is designed to relax these assumptions, making weak supervision more robust for real-world applications where data is often imbalanced and LF quality is asymmetric.

Methodology

Firebolt models the relationship between LFs and the latent ground truth labels using an Ising model. Unlike previous methods, Firebolt's innovation lies in its joint parameter estimation and efficient inference strategy.

Joint Parameter Estimation

Firebolt utilizes higher-order statistics, specifically two-way and three-way covariances between LFs, to solve a system of linear equations. This approach allows the framework to simultaneously "back out" the class balance and the class-specific accuracy (sensitivity and specificity) of each LF without requiring any ground truth labels. This is particularly effective for imbalanced datasets where an LF might be highly accurate for the minority class but less so for the majority.

Efficient and Interpretable Inference

Instead of relying on complex graphical model inference, Firebolt formulates the label aggregation as a logistic regression problem. This ensures polynomial-time complexity and provides clear interpretability: the regression coefficients directly represent the "weight" or contribution of each LF to the final probabilistic label.

Firebolt generates probabilistic labels that can be used to train downstream models.

Evaluation and Results

The authors evaluated Firebolt across benchmark datasets (e.g., spam, spouse, IMDb), zero-shot learning tasks (AwA2), and a large-scale industrial production environment at Google.

Performance Gains

On the highly imbalanced spouse dataset, Firebolt outperformed state-of-the-art methods by 5.8 AUC points and 10.8 Average Precision (AP) points. In the zero-shot AwA2 experiments, Firebolt achieved the best median AUC and AP across 45 classification tasks, demonstrating its versatility.

Industrial Impact

In a production case study involving hundreds of millions of examples, Firebolt improved model performance by 1.3 AUC points. More strikingly, it reduced the time required for label model training and inference from one hour to just three minutes, a 20x speedup.

Discussion and Future Work

A key insight of Firebolt is the "Power of Triplets"—using LF triplets to distinguish between natural data imbalance and LF bias. By modeling asymmetric LF quality, Firebolt handles imbalanced data far better than previous uniform-accuracy models.

However, some limitations remain. Firebolt still requires a dependency graph for initial parameter learning, and it currently assumes a single latent variable per data point. Future work may extend this to multi-task weak supervision and explore optimal convergence rates for covariance tensor estimation.

Conclusion

Firebolt represents a significant step forward in making weak supervision more practical. By operating under weaker assumptions and providing massive efficiency gains, it enables the rapid curation of high-quality training data for complex, real-world machine learning tasks.

Summary updated

Abstract

Modern machine learning demands a large amount of training data. Weak supervision is a promising approach to meet this demand. It aggregates multiple labeling functions (LFs)–noisy, user-provided labeling heuristics—to rapidly and cheaply curate probabilistic labels for large-scale unlabeled data. However, standard assumptions in weak supervision—such as user-specified class balance, similar accuracy of an LF in classifying different classes, and full knowledge of LF dependency at inference time—might be undesirable in practice. In response, we present Firebolt, a new weak supervision framework that seeks to operate under weaker assumptions. In particular, Firebolt learns the class balance and class-specific accuracy of LFs jointly from unlabeled data. It carries out inference in an efficient and interpretable manner. We analyze the parameter estimation error of Firebolt and characterize its impact on downstream model performance. Furthermore, we show that on five publicly available datasets, Firebolt outperforms a state-of-the-art weak supervision method by up to 5.8 points in AUC. We also provide a case study in the production setting of a tech company, where a Firebolt-supervised model outperforms the existing weakly-supervised production model by 1.3 points in AUC and speedup label model training and inference from one hour to three minutes.
BibTeX
@inproceedings{kuang2022firebolt,
  author = {Zhaobin Kuang and Chidubem G. Arachie and Bangyong Liang and Pradyumna Narayana and Giulia DeSalvo and Michael S. Quinn and Bert Huang and Geoffrey Downs and Yang Yang},
  title = {{Firebolt: Weak Supervision Under Weaker Assumptions}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
  url = {https://openreview.net/forum?id=-OAd_0ewQ1},
}
Figure