Introduction
Standard online hypothesis testing requires making permanent "reject" or "not reject" decisions the moment data arrives. This paper introduces TOAD (Significance Thresholds based on Active Discoveries), a framework that allows researchers to update their decisions until a pre-set "decision deadline." By blending the flexibility of offline testing with the flow of online streams, TOAD provides higher statistical power and guaranteed error control even when data points are correlated.
The Challenge of Streaming Data
In the era of "Big Data," hypotheses are often tested in a continuous stream—such as monitoring credit card transactions for fraud or running sequential experiments where early results inform later ones. To avoid being misled by random noise, researchers use False Discovery Rate (FDR) control to ensure that the proportion of "false alarms" stays below a certain threshold (usually 5% or 10%).
The challenge is that existing methods are often too rigid. Traditional "online" methods typically require immediate, irreversible decisions. Conversely, "offline" methods, like the Benjamini-Hochberg (BH) procedure, require all data to be available upfront. However, many real-world scenarios feature decision deadlines: a window of time exists before a final decision is required (e.g., before a transaction clears or a clinical trial phase ends). TOAD leverages this window to improve discovery rates without compromising error control.
The TOAD Framework
TOAD stands for Significance Thresholds based on Active Discoveries. It introduces several key innovations:
- Active Set Management: TOAD tracks a set of "active" hypotheses whose deadlines have not yet passed.
- Hybrid Online-Offline Logic: It combines the LOND procedure (an online method based on past discoveries) with the Benjamini-Hochberg (BH) procedure (the gold standard for offline testing).
- Retroactive Rejection: Unlike traditional online methods, TOAD can retroactively reject a hypothesis if subsequent data within the active window provides stronger context for significance.
- Dependency Robustness: TOAD is mathematically proven to control FDR under arbitrary p-value dependencies or positive correlations, addressing a major weakness in many online methods that assume independence.
Experimental Evaluation
The framework was evaluated using:
- Synthetic Simulations: 3,000 test statistics with varying correlation ($\rho = 0, 0.5$) and batch sizes (10, 100, 1000) representing different deadline lengths.
- Real-World Fraud Detection: A Kaggle dataset of 48 hours of credit card transactions, simulating a stream where fraud decisions must be made within 10 seconds to 3 hours.
Key Results
TOAD demonstrates a significant advantage in statistical power while maintaining strict error control.
As shown in the power analysis, TOAD consistently outperforms its closest rigorous competitor, Batch-PRDS-BH. While some "naive" batch methods (like Batch-BH) might appear to have higher power, they often fail to control the FDR, especially under correlation.
In the credit card fraud dataset, TOAD produced the highest number of discoveries among all methods that successfully controlled the FDR at the target 0.1 level. As the decision deadline increases, TOAD’s performance improves, approaching the efficiency of full offline testing.
Conclusion and Future Directions
TOAD bridges the gap between online and offline testing. It is a true generalization: with an immediate deadline, it collapses into the LOND procedure; with an infinite deadline, it becomes the standard BH procedure. Future work includes addressing "adaptive stopping" (where experiments end early due to strong results) and implementing "decaying memory" to manage error budgets over very long streams without losing statistical guarantees.
Summary updated