Unique Rashomon Sets for Robust Active Learning

Simon Nguyen, Kentaro Hoffman, Tyler McCormick

TL;DR

Active learning often fails in noisy environments because ensemble committees get confused by poorly performing or redundant models. This paper introduces UNREAL (UNique Rashomon Ensembled Active Learning), a method that builds a query committee exclusively from unique, high-performing models within the Rashomon set. By filtering out redundant decision trees that yield identical predictions, UNREAL cuts through label noise, boosts predictive accuracy by up to 20%, and significantly improves model interpretability.

What Problem Does This Paper Address?

Active learning (AL) aims to minimize data labeling costs by smartly querying the most informative unlabeled points. A popular framework for this is Query-by-Committee (QBC), where an ensemble of models votes on unlabeled data, and the points with the highest disagreement (uncertainty) are selected for labeling.

However, QBC faces a major hurdle when dealing with limited initial data or label noise. Standard ensemble methods like Random Forests aggregate weak learners indiscriminately, meaning poor or highly redundant models get a vote. In tree-based models, this manifests as the "multiplicity of trees"—multiple distinct tree structures can map to the exact same data partition. This redundancy artificially inflates committee agreement, causing the active learner to chase noise and overfit rather than targeting true regions of uncertainty.

Methods

The authors present UNREAL to inject sanity into the committee selection process:

  1. Enumerate the Rashomon Set: Instead of randomly sampling features and data (like a Random Forest), UNREAL uses the TreeFarms algorithm to exhaustively find the Rashomon set—the collection of all sparse decision trees that perform within a small error threshold ($\epsilon$) of the empirical risk minimizer.
  2. Deduplicate Explanations: To eliminate structural redundancy, UNREAL groups these trees by their actual prediction patterns on the candidate dataset. It then selects just one representative tree from each unique pattern group to form the active learning committee.
  3. Query via Vote Entropy: The deduplicated committee uses vote entropy to identify the most genuinely uncertain data points for oracle labeling.

The authors compare UNREAL against standard Passive Learning, Random Forest QBC, and DUREAL (Duplicate Rashomon Ensemble Active Learning), a variant that retains all redundant trees in the Rashomon set.

Datasets

The methodology was evaluated on five benchmark datasets that provide a gradient of structural complexity and noise:

  • Iris and MONK-1: Represent relatively simple, clean underlying data structures.
  • MONK-3: Introduces a more intricate generative scheme.
  • COMPAS (recidivism risk prediction) and Bar7 (restaurant coupon acceptance): Real-world datasets characterized by notable noise and complexity. Due to the computational intensity of full tree enumeration, these two datasets were downsampled to a fixed 200 observations for the simulations.

Key Insights and Ideas

  • Noise as a Diversity Generator: Counterintuitively, increasing label noise expands the size of the Rashomon set and increases "predictive multiplicity" (the number of diverse, equally valid explanations). UNREAL turns this ambiguity into an advantage by using the expanded set to capture a comprehensive view of true uncertainty.
  • Trees $\neq$ Explanations: A key realization is that distinct tree structures are often just different geometric manifestations of the identical classification pattern. Deduplicating them prevents the committee's uncertainty metrics from being skewed by complex, redundant trees.
  • The Combinatorial Bias of DUREAL: The authors discovered that keeping duplicate trees (DUREAL) creates an implicit weighted voting system favoring complex trees, simply because complex partitions combinatorially allow for more tree variations. UNREAL achieves comparable accuracy to DUREAL but uses a vastly smaller, more parsimonious, and interpretable committee.

Key Results

  • Up to 20% Accuracy Gains: Ensembles drawn from the Rashomon set consistently outperformed traditional QBC with Random Forests and passive learning. In the MONK-1 dataset, the Rashomon-based approaches demonstrated an improvement of up to 20% in predictive accuracy (F1 score) over the Random Forest baseline.
  • Faster Convergence: With the exception of the highly noisy Bar7 dataset, the performance gap between Rashomon ensembling and traditional methods widened progressively as more labeled data was added.
  • Natural Complexity Collapse: As active learning progressed and more data points were labeled, the number of unique classification patterns in UNREAL naturally collapsed toward a single explanation, demonstrating clear algorithmic convergence.

Limitations and Future Work

  • Static Thresholding: The Rashomon threshold ($\epsilon$) is selected based on initial training data and held constant. However, the optimal threshold size changes as the dataset evolves. The authors note that dynamically recalibrating $\epsilon$ at each iteration would be ideal but introduces massive computational overhead.
  • Computational Intensity: Exhaustively enumerating the Rashomon set via TreeFarms is highly resource-intensive, which is why the real-world datasets had to be scaled down to 200 observations. It may not be suitable for applications prioritizing raw computational speed over predictive accuracy.
  • Future Directions: The authors point to Rashomon Partition Sets (RPS) as a promising alternative framework to comprehensively enumerate the Rashomon set without being constrained by the rigid geometric structure of decision trees.

Summary updated

Abstract

Collecting labeled data for machine learning models is often expensive and time-consuming. Active learning addresses this challenge by selectively labeling the most informative observations, but when initial labeled data is limited, it becomes difficult to distinguish genuinely informative points from those appearing uncertain primarily due to noise. Ensemble methods like random forests are a powerful approach to quantifying this uncertainty but do so by aggregating all models indiscriminately. This includes poor performing models and redundant models, a problem that worsens in the presence of noisy data. We introduce UNique Rashomon Ensembled Active Learning (UNREAL), which selectively ensembles only distinct models from the Rashomon set, which is the set of nearly optimal models. Restricting ensemble membership to high-performing models with different explanations helps distinguish genuine uncertainty from noise-induced variation. We show that UNREAL achieves faster theoretical convergence rates than traditional active learning approaches and demonstrates empirical improvements of up to 20% in predictive accuracy across five benchmark datasets, while simultaneously enhancing model interpretability.
BibTeX
@misc{nguyen2025unique,
  author = {Simon Nguyen and Kentaro Hoffman and Tyler McCormick},
  title = {{Unique Rashomon Sets for Robust Active Learning}},
  year = {2025},
  eprint = {2503.06770},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.06770},
}
Figure