TL;DR
Active learning often fails in noisy environments because ensemble committees get confused by poorly performing or redundant models. This paper introduces UNREAL (UNique Rashomon Ensembled Active Learning), a method that builds a query committee exclusively from unique, high-performing models within the Rashomon set. By filtering out redundant decision trees that yield identical predictions, UNREAL cuts through label noise, boosts predictive accuracy by up to 20%, and significantly improves model interpretability.
What Problem Does This Paper Address?
Active learning (AL) aims to minimize data labeling costs by smartly querying the most informative unlabeled points. A popular framework for this is Query-by-Committee (QBC), where an ensemble of models votes on unlabeled data, and the points with the highest disagreement (uncertainty) are selected for labeling.
However, QBC faces a major hurdle when dealing with limited initial data or label noise. Standard ensemble methods like Random Forests aggregate weak learners indiscriminately, meaning poor or highly redundant models get a vote. In tree-based models, this manifests as the "multiplicity of trees"—multiple distinct tree structures can map to the exact same data partition. This redundancy artificially inflates committee agreement, causing the active learner to chase noise and overfit rather than targeting true regions of uncertainty.
Methods
The authors present UNREAL to inject sanity into the committee selection process:
- Enumerate the Rashomon Set: Instead of randomly sampling features and data (like a Random Forest), UNREAL uses the
TreeFarmsalgorithm to exhaustively find the Rashomon set—the collection of all sparse decision trees that perform within a small error threshold ($\epsilon$) of the empirical risk minimizer. - Deduplicate Explanations: To eliminate structural redundancy, UNREAL groups these trees by their actual prediction patterns on the candidate dataset. It then selects just one representative tree from each unique pattern group to form the active learning committee.
- Query via Vote Entropy: The deduplicated committee uses vote entropy to identify the most genuinely uncertain data points for oracle labeling.
The authors compare UNREAL against standard Passive Learning, Random Forest QBC, and DUREAL (Duplicate Rashomon Ensemble Active Learning), a variant that retains all redundant trees in the Rashomon set.
Datasets
The methodology was evaluated on five benchmark datasets that provide a gradient of structural complexity and noise:
- Iris and MONK-1: Represent relatively simple, clean underlying data structures.
- MONK-3: Introduces a more intricate generative scheme.
- COMPAS (recidivism risk prediction) and Bar7 (restaurant coupon acceptance): Real-world datasets characterized by notable noise and complexity. Due to the computational intensity of full tree enumeration, these two datasets were downsampled to a fixed 200 observations for the simulations.
Key Insights and Ideas
- Noise as a Diversity Generator: Counterintuitively, increasing label noise expands the size of the Rashomon set and increases "predictive multiplicity" (the number of diverse, equally valid explanations). UNREAL turns this ambiguity into an advantage by using the expanded set to capture a comprehensive view of true uncertainty.
- Trees $\neq$ Explanations: A key realization is that distinct tree structures are often just different geometric manifestations of the identical classification pattern. Deduplicating them prevents the committee's uncertainty metrics from being skewed by complex, redundant trees.
- The Combinatorial Bias of DUREAL: The authors discovered that keeping duplicate trees (DUREAL) creates an implicit weighted voting system favoring complex trees, simply because complex partitions combinatorially allow for more tree variations. UNREAL achieves comparable accuracy to DUREAL but uses a vastly smaller, more parsimonious, and interpretable committee.
Key Results
- Up to 20% Accuracy Gains: Ensembles drawn from the Rashomon set consistently outperformed traditional QBC with Random Forests and passive learning. In the MONK-1 dataset, the Rashomon-based approaches demonstrated an improvement of up to 20% in predictive accuracy (F1 score) over the Random Forest baseline.
- Faster Convergence: With the exception of the highly noisy Bar7 dataset, the performance gap between Rashomon ensembling and traditional methods widened progressively as more labeled data was added.
- Natural Complexity Collapse: As active learning progressed and more data points were labeled, the number of unique classification patterns in UNREAL naturally collapsed toward a single explanation, demonstrating clear algorithmic convergence.
Limitations and Future Work
- Static Thresholding: The Rashomon threshold ($\epsilon$) is selected based on initial training data and held constant. However, the optimal threshold size changes as the dataset evolves. The authors note that dynamically recalibrating $\epsilon$ at each iteration would be ideal but introduces massive computational overhead.
- Computational Intensity: Exhaustively enumerating the Rashomon set via
TreeFarmsis highly resource-intensive, which is why the real-world datasets had to be scaled down to 200 observations. It may not be suitable for applications prioritizing raw computational speed over predictive accuracy. - Future Directions: The authors point to Rashomon Partition Sets (RPS) as a promising alternative framework to comprehensively enumerate the Rashomon set without being constrained by the rigid geometric structure of decision trees.
Summary updated