TL;DR
This paper introduces WeaNF, a novel approach to weak supervision that uses normalizing flows to model the underlying data distributions associated with labeling functions. By learning these distributions, WeaNF can better handle overlapping and noisy weak labels, leading to improved classification performance compared to existing methods.
What Problem Does This Paper Address?
Traditional supervised learning requires large amounts of manually labeled data, which is expensive and time-consuming to obtain. Weak supervision offers a solution by leveraging noisy, imperfect labeling sources like keywords, heuristics, or external databases. However, these weak labels are often noisy, incomplete, and can have complex interdependencies. Existing weak supervision methods, particularly those based on generative modeling (like Snorkel), focus on modeling the relationships between labeling functions and true labels. This paper identifies a gap: these methods don't directly model the biases and characteristics of weak supervision within the feature space itself. The authors propose to address this by using normalizing flows to model the input data distributions induced by each labeling function.
Methods
The core of WeaNF lies in using normalizing flows to learn probability distributions over the input feature space for each weak labeling function. Normalizing flows are a class of generative models that can learn complex, high-dimensional distributions by composing simple, invertible transformations.
The paper proposes four variants:
- WeaNF-S (Standard): Each labeling function is modeled as a multivariate normal distribution.
- WeaNF-I (Iterative): An iterative EM-like approach to leverage unlabeled data, where predictions from previous iterations are used to label unseen data.
- WeaNF-N (Negative Model): Extends WeaNF-S/I by explicitly modeling the "negative space" – the distribution of data not matched by a labeling function. This allows for direct access to posterior probabilities of labeling functions given an input ($P(\lambda|x)$).
- WeaNF-M (Mixed Model): Aims to directly model the intersections of multiple labeling functions by learning a common space for their embeddings.
The learned densities are then aggregated using various schemes (Maximum, Union, NoisyOr, Simplex) to make final class predictions.
Datasets
The authors evaluate their models on five standard weakly supervised classification datasets:
- IMDb: Movie review sentiment analysis.
- Spouse: Classifying spouse relations in text.
- YouTube Spam: Detecting spam comments on YouTube.
- SMS Spam: Detecting spam in private messaging.
- TREC-6: Classifying questions into six categories.
The datasets vary in size, class balance, and the number of labeling functions provided.
Key Insights and Ideas
- Modeling the Feature Space is Crucial: Directly modeling the input data distributions associated with labeling functions (as done by WeaNF) offers a complementary and effective approach to existing methods that focus on labeling function correlations.
- Normalizing Flows are Well-Suited: The invertible nature and ability to model complex distributions make normalizing flows a powerful tool for learning these input-side densities.
- Handling Overlap and Negative Space: WeaNF-N's explicit modeling of the negative space is key to accessing posterior probabilities, which can be leveraged for more nuanced aggregation strategies.
- Iterative Refinement Helps: The WeaNF-I variant, which iteratively uses unlabeled data, shows significant improvements, highlighting the benefit of incorporating unlabeled data.
- Aggregation Matters: Different aggregation schemes (e.g., NoisyOr, Union) yield varying results, suggesting that the choice of how to combine learned densities is important.
Key Results
- Competitive Performance: WeaNF models generally perform competitively with, and often outperform, standard weak supervision baselines (Majority Voting, MLP, Snorkel+MLP) across the tested datasets.
- WeaNF-I is a Strong Performer: The iterative version, WeaNF-I, is highlighted as the best-performing model on most datasets, achieving accuracies up to 93.54% (SMS dataset).
- WeaNF-N Benefits from Many LFs: WeaNF-N shows improvements on datasets with a large number of labeling functions, suggesting its ability to handle complex interactions.
- Generalization: Qualitative analysis shows that WeaNF models can generalize to examples not directly matched by labeling functions, detecting semantic meanings beyond exact keyword matches.
Limitations and Future Work
- Dataset Specificity: While competitive, WeaNF-N sometimes struggles to predict the correct labeling function on longer text datasets (IMDb, Spouse), potentially due to the complexity of modeling these longer sequences.
- Simplex Aggregation Sensitivity: The simplex aggregation scheme appears sensitive to the smoothness of the probability density, sometimes underperforming simpler methods like maximum aggregation.
- Future Work: The authors suggest further exploration of more expressive normalizing flow architectures and advanced aggregation techniques. They also note the potential for WeaNF to be used in conjunction with other weak supervision methods.
Summary updated