TL;DR
WeaNF (Weak Supervision with Normalizing Flows) shifts the focus of weak supervision from modeling label correlations to modeling the input feature space. By using Normalizing Flows to estimate the density of data covered by specific labeling functions, the authors create a generative model that generalizes better to unseen data and outperforms standard baselines like Snorkel on multiple NLP benchmarks.
What Problem Does This Paper Address?
In the era of massive datasets, manual annotation is a bottleneck. Weak supervision attempts to solve this by using "labeling functions" (LFs)—heuristics, keyword lists, or external databases—to programmatically generate noisy labels.
However, LFs are inherently flawed: they are noisy, often overlap, and only cover a small fraction of the data. Existing state-of-the-art methods, such as Snorkel, typically focus on modeling the outputs of these LFs (the labels themselves) to estimate their accuracy and correlations. The authors of WeaNF argue that this ignores a crucial piece of information: the feature space. By not modeling the actual distribution of the input data (e.g., text embeddings) that triggers an LF, existing models miss out on the semantic biases and generalization opportunities inherent in the data.
Methods
The core innovation of WeaNF is the use of Normalizing Flows (NF) to model the probability distribution of the input space induced by labeling functions. Specifically, they use the RealNVP architecture to learn a mapping between a simple base distribution and the complex distribution of text embeddings associated with an LF.
The authors propose four model variants:
- WeaNF-S (Standard): Learns the density $P(x|\lambda)$ for each LF $\lambda$. It predicts labels by choosing the class associated with the LF that yields the highest likelihood for a given input.
- WeaNF-I (Iterative): An EM-like variant that uses the model's own predictions on unlabeled data to iteratively refine the learned densities.
- WeaNF-N (Negative): Additionally models the "negative space"—the distribution of data where an LF does not match ($P(x|\neg\lambda)$). This allows the use of Bayes' rule to calculate the posterior $P(\lambda|x)$, enabling more robust aggregation schemes like NoisyOr.
- WeaNF-M (Mixed): Explicitly models the intersections of multiple LFs by sampling from a simplex of LF embeddings, aiming to capture how different rules interact in the feature space.
Datasets
The models were evaluated on five diverse classification tasks, using text embeddings generated by SentenceTransformers (specifically bert-base-nli-mean-tokens):
- IMDb: Sentiment analysis (Positive/Negative).
- Spouse: Relation extraction (identifying spouse relations in text).
- YouTube Spam: Classifying YouTube comments as spam or not.
- SMS Spam: Detecting spam in private messages.
- TREC-6: A 6-class question classification dataset (e.g., Location, Human, Entity).
The authors performed specific preprocessing to remove duplicates and filtered out LFs that occurred too infrequently to allow for stable density estimation.
Key Insights and Ideas
- Generalization Beyond Keywords: Because Normalizing Flows learn a smooth density in the embedding space, the model can generalize. For example, if an LF is triggered by the keyword "hilarious," the flow might assign a high density to a sentence containing "funny," even if the word "hilarious" is absent.
- Modeling Bias Directly: By modeling $P(x|\lambda)$, the model directly captures the "biases" of the weak supervision sources in the feature space, rather than just treating them as black-box label generators.
- The Power of Iteration: The iterative WeaNF-I model was consistently the strongest, demonstrating that even noisy pseudo-labels from a density estimator can effectively guide the learning process on unlabeled data.
Key Results
WeaNF-I emerged as the top-performing model across almost all benchmarks, frequently outperforming both Majority Voting (MV) and the Snorkel paradigm (DP + MLP).
| Dataset | MV | DP + MLP (Snorkel) | WeaNF-I (Ours) |
|---|---|---|---|
| IMDb (Acc) | 56.84 | 67.79 | 74.08 |
| Spouse (F1) | 49.87 | 57.05 | 57.96 |
| YouTube (Acc) | 81.66 | 88.79 | 89.08 |
| SMS (F1) | 56.10 | 84.40 | 93.54 |
| Trec (Acc) | 61.20 | 43.00 | 67.80 |
Qualitative analysis (Table 4 in the paper) showed that the model successfully assigned high likelihoods to semantically related examples that were not explicitly caught by the hard-coded labeling functions.
Limitations and Future Work
- Text Length Sensitivity: The authors noted that WeaNF-N struggled more with datasets containing longer texts (like IMDb and Spouse). Modeling high-dimensional densities for long sequences remains a significant challenge for Normalizing Flows.
- Intersection Sparsity: While WeaNF-M attempted to model LF intersections, many intersections occur too rarely in real-world data to build a reliable density model.
- Architecture Exploration: The study used RealNVP as a backbone; the authors suggest that exploring more modern or expressive flow architectures could yield even better results in capturing complex text distributions.
Summary updated