TL;DR
This paper introduces Online Hierarchical Selection (OHS), a framework for making sequential "select/reject" decisions on streaming data that is grouped or hierarchical (e.g., patients in different hospitals). By leveraging rank-based e-values and a specialized online Benjamini-Hochberg procedure, the authors provide a way to control the False Discovery Rate (FDR) at any point in time, even when data violates standard i.i.d. assumptions.
What Problem Does This Paper Address?
In many real-world scenarios—like clinical trials or drug discovery—we must decide whether to "select" a candidate (e.g., enroll a patient or test a chemical compound) based on a predictive score before the actual outcome is known.
Existing "conformal selection" methods provide a principled way to do this while controlling the False Discovery Rate (FDR). However, they typically rely on the assumption that data points are i.i.d. (independent and identically distributed) or at least exchangeable.
Real-world data is rarely that simple; it is often hierarchical. For example:
- Multi-center studies: Patients are grouped by hospitals with different population distributions.
- Repeated assessments: One unit might have multiple annotations.
- Cluster trials: Individuals are nested within specific villages or schools.
If you ignore this group structure, your statistical tests become invalid, leading to a loss of power or, worse, a failure to control false discoveries. This paper fills that gap by extending conformal selection to hierarchical streams.
Methods
The authors propose two main algorithms: Online Hierarchical Selection (OHS) and its feedback-enabled variant, OHSF.
- Rank-based E-values: Instead of traditional p-values, the authors use e-values, which are more robust for testing under complex data structures. They construct two types of evidence for each arriving candidate:
- Within-group e-value: Compares the candidate to previous "null" (unsuccessful) samples from the same group.
- Between-group e-value: Compares the candidate to a representative sample from every other group.
- Weighted Averaging: These two e-values are combined using a deterministic weight that balances local group information with global population trends.
- Online e-BH Procedure: The combined e-values are fed into an online version of the Benjamini-Hochberg procedure. This allows the system to expand the "selection set" as new data arrives without ever revoking a previous decision.
- Feedback (OHSF): In settings where the true outcome is revealed shortly after a decision is made, OHSF incorporates that new "ground truth" back into the reference pools, significantly boosting the statistical power of future decisions.
Datasets
The researchers evaluated their methods on three primary sources:
- Synthetic Simulations: Controlled environments with varying signal strengths and group offsets to verify FDR control.
- Beijing Multi-Site Air Quality (PRSA): Hourly pollutant data from 12 monitoring stations. The goal was to identify instances of high $PM_{2.5}$ concentrations.
- Drug–Target Affinity (KIBA): A dataset of over 118,000 affinity measurements. Here, groups are defined by different kinase targets, and the task is to select drug-target pairs with high binding affinity.
Key Insights and Ideas
- The Power of "Between-Group" Evidence: A major "aha" moment is that even if a specific group has very few samples (making within-group comparison weak), we can still make a confident decision by comparing the candidate to the "average" behavior across all other groups.
- Distribution-Free Guarantees: The method doesn't care what the underlying distribution of the data is. As long as the data is "hierarchically exchangeable" (meaning the labels within a group can be swapped without changing the joint distribution), the FDR control holds.
- Handling "Unseen" Groups: The framework is robust enough to handle the sudden arrival of a brand-new group that wasn't present in the initial calibration data—a common occurrence in streaming applications.
Key Results
- Rigorous FDR Control: In all experiments (simulated and real), OHS and OHSF successfully kept the empirical FDR below the target threshold (e.g., $\alpha = 0.05$).
- Superior Power: Compared to "Ungrouped" baselines (which ignore hierarchy) or "Within-only" baselines (which ignore other groups), OHS variants recovered significantly more "true" signals.
- Stability with Feedback: In the air quality and drug-target tasks, OHSF showed that incorporating feedback allows the model to maintain high performance even as the data stream evolves, outperforming fixed-pool versions.
Limitations and Future Work
- Delayed Feedback: The current OHSF model assumes feedback is revealed almost immediately (at the next time step). In many fields, outcomes might take weeks or months to arrive. Extending this to delayed feedback is a noted area for future research.
- Selection Bias in Feedback: Currently, the feedback mechanism assumes we eventually see the labels for all candidates. In some cases, we only see the outcome for candidates we selected.
- Data-Driven Weights: The weights used to combine within- and between-group evidence are currently deterministic. The authors suggest that developing data-driven weights that adapt to the variance of specific groups could further increase power while maintaining validity.
Summary updated