Abstract
Classification problems with a very large or unbounded set of output categories are common in many areas such as natural language and image processing. In order to improve accuracy on these tasks, this paper introduces an unsupervised generative model to aggregate predictions from multiple classifiers. By modeling "confusion sets" using Dirichlet processes and accounting for the varying expertise of judges across different types of categories, the approach significantly outperforms standard majority voting.
Introduction
Ensemble methods—combining predictions from multiple "experts" or classifiers—are a staple in machine learning for boosting accuracy. However, most existing aggregation techniques rely on two assumptions that often fail in real-world applications: they assume a small, fixed output space (e.g., binary classification) and require supervised training data to learn which experts are trustworthy.
In complex tasks like natural language parsing or image segmentation, the number of potential categories can be enormous or even unbounded. Furthermore, standard unsupervised models assume that when experts make mistakes, their errors are independent. In reality, experts usually get confused by the same small subset of plausible wrong answers, forming "confusion sets." Failing to account for these sets in large output spaces causes standard aggregation models to degrade into simple majority voting.
Methods
The authors propose a novel generative model for unsupervised aggregation that relaxes the strict conditional independence assumption by explicitly modeling confusion sets.
- Modeling Confusion Sets (Type-Agnostic Model): Instead of assuming mistakes are independent, the model assumes they are drawn from a shared underlying "confusion distribution." To capture the intuition that experts tend to make the same reasonable mistakes without requiring labeled data, the authors use a Dirichlet Process (DP). This mathematically enforces a preference for smaller, more concentrated confusion sets.
- Incorporating Category Types (Typed Model): An expert's accuracy is rarely uniform; a parser might be excellent at identifying nouns but poor at verbs. The authors extend their model to group categories into types, assigning each expert a recall parameter and a false-positive rate per type.
- Learning and Inference: The model learns expert reliability purely by observing agreement patterns among experts using the Expectation-Maximization (EM) algorithm, with the Chinese Restaurant Process (CRP) handling the Dirichlet Process mathematics.
Results
The models were evaluated on both synthetic and real-world datasets to test the impact of confusion modeling and category typing.
Synthetic Experiments
A custom dataset with 150 output categories split into three types was used to simulate experts with specific type-based strengths. As shown in Figure 1, the typed model significantly outperformed both the majority vote baseline and the type-agnostic model, validating the utility of exploiting type-specific expertise.
Dependency Parsing
The model aggregated outputs from 20 to 23 parsers from the CoNLL-2007 Shared Task across 10 languages. The aggregation model successfully outperformed both the majority vote baseline and the single best expert. This performance boost was particularly pronounced in practical scenarios where only a small number of experts (e.g., 3 to 8 parsers) were available.
Discussion and Limitations
The core insight of this work is that assuming independent errors is fatal in large output spaces. By assuming mistakes are exchangeable via Dirichlet processes rather than independent, the model can naturally explain high agreement between incorrect experts without needing supervised training data.
However, the typed model requires sufficient data to estimate parameters for specific category types. In the dependency parsing experiments, the test sets were too small to reliably estimate these parameters, forcing a reliance on the type-agnostic version. Future work includes leveraging larger unlabeled datasets and exploring automated ways to discover category types rather than relying on manual definitions.
Summary updated