TL;DR
This paper argues that generative crowdsourcing models, which model the data generation process, outperform discriminative models (which only model the relationship between data and labels) in typical crowdsourcing scenarios. They introduce a novel variational inference algorithm that significantly improves the performance of their generative model, MOMRESP, and demonstrate its superiority on various text classification tasks.
What Problem Does This Paper Address?
Crowdsourcing is a popular method for generating labeled datasets for machine learning. However, crowdsourced labels are often noisy, meaning annotators make mistakes. To address this, researchers develop probabilistic models to aggregate these noisy labels and infer the true underlying labels. While many existing "data-aware" models leverage features of the data itself (like words in a document) to improve label inference, there's a debate about whether generative models (which model how data and labels are generated) or discriminative models (which directly model the relationship between data and labels) are better for this task. Previous work has largely favored discriminative models, but this paper challenges that assumption.
Methods
The authors compare two data-aware crowdsourcing models: LOGRESP (a discriminative log-linear model) and MOMRESP (a generative multinomial model). Both models are extensions of the ITEMRESP framework, which models annotator reliability and true labels. The key innovation is the development of a novel mean-field variational inference algorithm for MOMRESP. This algorithm allows the generative model to be trained and inferred efficiently, making it directly comparable to discriminative models. They also present a similar mean-field variational inference algorithm for LOGRESP to ensure a fair comparison. The paper also explores the relationship between generative and discriminative models, showing how they can be seen as different ways of fitting the same underlying model.
Datasets
The paper uses six text classification datasets for evaluation:
- 20 Newsgroups
- WebKB
- Cade12
- Reuters8
- Reuters52
- LDC-labeled Enron emails
These datasets were preprocessed by stemming, removing stopwords, and discarding features that appeared fewer than 5 times. Features were also fractionally scaled. For evaluation, they simulated annotators with varying accuracy levels and error patterns, creating different "annotator pools" (HIGH, MED, LOW, CONFLICT). They also conducted an experiment with human annotators from CrowdFlower on a subset of the 20 Newsgroups dataset.
Key Insights and Ideas
- Generative models offer an "early gain" advantage: In typical crowdsourcing settings, generative models like MOMRESP tend to achieve higher accuracy with fewer annotations compared to discriminative models. This is crucial because acquiring many annotations can be expensive.
- Robustness to noise: Generative models appear to be more robust to noisy annotations, which is a common characteristic of crowdsourced data.
- Semi-supervised learning is inherent: Because MOMRESP is fully generative, it can naturally leverage unannotated data to improve label inference, a capability that discriminative models like LOGRESP lack without explicit extensions.
- Fair comparison is key: The development of comparable mean-field variational inference algorithms for both models was essential for a rigorous empirical comparison.
- Majority vote is surprisingly competitive: In many scenarios, especially when annotator quality is high, a simple majority vote baseline becomes very competitive with sophisticated models like LOGRESP once a sufficient number of annotations are available.
Key Results
- MOMRESP consistently outperforms LOGRESP, especially when annotation data is scarce. This advantage is observed across various datasets and simulated annotator pools.
- The new mean-field variational inference algorithm significantly improves MOMRESP's performance compared to previous inference methods (e.g., Gibbs sampling).
- While LOGRESP may eventually surpass MOMRESP with a very large number of annotations, the "early gains" of MOMRESP make it more practical for many real-world crowdsourcing tasks.
- The CONFLICT annotator pool, designed to simulate annotators with fundamentally different interpretations, highlights the limitations of majority vote and shows MOMRESP's resilience.
Limitations and Future Work
- Model assumptions: The paper acknowledges that the generative multinomial model's assumption of inter-feature independence might not be suitable for all datasets, particularly those with highly correlated features.
- Hyperparameter optimization: The authors suggest that further hyperparameter optimization could potentially reduce the performance disparity between their MF algorithm and other inference methods.
- Future directions: The authors propose exploring more sophisticated generative models, such as document topic models, or transforming data for better compatibility with simpler models. They also suggest extending these experiments to other crowdsourcing tasks beyond text classification, like sequence labeling.
Summary updated