Conformal Selection for Hierarchical Streaming Data

Zhenyuan Sun, Chengyao Yu, Ruixing Ming, Bingyi JingarXiv 2026 · arXiv

TL;DR

This paper introduces Online Hierarchical Selection (OHS), a framework for making sequential "select/reject" decisions on streaming data that is grouped or hierarchical (e.g., patients in different hospitals). By leveraging rank-based e-values and a specialized online Benjamini-Hochberg procedure, the authors provide a way to control the False Discovery Rate (FDR) at any point in time, even when data violates standard i.i.d. assumptions.

What Problem Does This Paper Address?

In many real-world scenarios—like clinical trials or drug discovery—we must decide whether to "select" a candidate (e.g., enroll a patient or test a chemical compound) based on a predictive score before the actual outcome is known.

Existing "conformal selection" methods provide a principled way to do this while controlling the False Discovery Rate (FDR). However, they typically rely on the assumption that data points are i.i.d. (independent and identically distributed) or at least exchangeable.

Real-world data is rarely that simple; it is often hierarchical. For example:

  • Multi-center studies: Patients are grouped by hospitals with different population distributions.
  • Repeated assessments: One unit might have multiple annotations.
  • Cluster trials: Individuals are nested within specific villages or schools.

If you ignore this group structure, your statistical tests become invalid, leading to a loss of power or, worse, a failure to control false discoveries. This paper fills that gap by extending conformal selection to hierarchical streams.

Methods

The authors propose two main algorithms: Online Hierarchical Selection (OHS) and its feedback-enabled variant, OHSF.

  1. Rank-based E-values: Instead of traditional p-values, the authors use e-values, which are more robust for testing under complex data structures. They construct two types of evidence for each arriving candidate:
  2. Within-group e-value: Compares the candidate to previous "null" (unsuccessful) samples from the same group.
  3. Between-group e-value: Compares the candidate to a representative sample from every other group.
  4. Weighted Averaging: These two e-values are combined using a deterministic weight that balances local group information with global population trends.
  5. Online e-BH Procedure: The combined e-values are fed into an online version of the Benjamini-Hochberg procedure. This allows the system to expand the "selection set" as new data arrives without ever revoking a previous decision.
  6. Feedback (OHSF): In settings where the true outcome is revealed shortly after a decision is made, OHSF incorporates that new "ground truth" back into the reference pools, significantly boosting the statistical power of future decisions.

Datasets

The researchers evaluated their methods on three primary sources:

  • Synthetic Simulations: Controlled environments with varying signal strengths and group offsets to verify FDR control.
  • Beijing Multi-Site Air Quality (PRSA): Hourly pollutant data from 12 monitoring stations. The goal was to identify instances of high $PM_{2.5}$ concentrations.
  • Drug–Target Affinity (KIBA): A dataset of over 118,000 affinity measurements. Here, groups are defined by different kinase targets, and the task is to select drug-target pairs with high binding affinity.

Key Insights and Ideas

  • The Power of "Between-Group" Evidence: A major "aha" moment is that even if a specific group has very few samples (making within-group comparison weak), we can still make a confident decision by comparing the candidate to the "average" behavior across all other groups.
  • Distribution-Free Guarantees: The method doesn't care what the underlying distribution of the data is. As long as the data is "hierarchically exchangeable" (meaning the labels within a group can be swapped without changing the joint distribution), the FDR control holds.
  • Handling "Unseen" Groups: The framework is robust enough to handle the sudden arrival of a brand-new group that wasn't present in the initial calibration data—a common occurrence in streaming applications.

Key Results

  • Rigorous FDR Control: In all experiments (simulated and real), OHS and OHSF successfully kept the empirical FDR below the target threshold (e.g., $\alpha = 0.05$).
  • Superior Power: Compared to "Ungrouped" baselines (which ignore hierarchy) or "Within-only" baselines (which ignore other groups), OHS variants recovered significantly more "true" signals.
  • Stability with Feedback: In the air quality and drug-target tasks, OHSF showed that incorporating feedback allows the model to maintain high performance even as the data stream evolves, outperforming fixed-pool versions.

Limitations and Future Work

  • Delayed Feedback: The current OHSF model assumes feedback is revealed almost immediately (at the next time step). In many fields, outcomes might take weeks or months to arrive. Extending this to delayed feedback is a noted area for future research.
  • Selection Bias in Feedback: Currently, the feedback mechanism assumes we eventually see the labels for all candidates. In some cases, we only see the outcome for candidates we selected.
  • Data-Driven Weights: The weights used to combine within- and between-group evidence are currently deterministic. The authors suggest that developing data-driven weights that adapt to the variance of specific groups could further increase power while maintaining validity.

Summary updated

Abstract

Many real-world tasks involve making sequential decisions about whether to select candidates before their responses of interest are revealed. In this work, we address the online selection problem in the context of hierarchical data, where candidates arrive sequentially from different populations. This extends existing online conformal selection frameworks beyond the standard i.i.d./exchangeable data assumption. The primary challenge lies in constructing valid statistics for each candidate sequentially under the hierarchical data structure. To this end, we first develop Online Hierarchical Selection (OHS), a distribution-free method that enables principled candidate selection for hierarchical data streams. Specifically, we introduce methods to construct rank-based e-values for both between-group and within-group samples, and apply averaged e-values to the online e-BH procedure. We then develop OHS with feedback (OHSF), which boosts the power of OHS by leveraging potential response feedback. Theoretically, the proposed methods achieve false discovery rate control under mild hierarchical exchangeability and data arrival assumptions at any time. We also generalize methods to broader settings, such as data with covariate shift and selection under effect-related responses. Extensive experiments on both simulated and real-world datasets demonstrate the effectiveness of the proposed methods.
BibTeX
@misc{sun2026conformal,
  author = {Zhenyuan Sun and Chengyao Yu and Ruixing Ming and Bingyi Jing},
  title = {{Conformal Selection for Hierarchical Streaming Data}},
  year = {2026},
  eprint = {2611.26328},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26328},
}
Figure