Abstract
We present STAR: a novel system aimed at solving the complex issue of "p-hacking" and false discoveries in scientific studies. STAR provides a concrete way for ensuring the application of false discovery control procedures in hypothesis testing, using mathematically provable guarantees, with the goal of reducing the risk of data dredging. STAR generates an efficiently auditable certificate which attests to the validity of each statistical test performed on a dataset. STAR achieves this by using several cryptographic techniques which are combined specifically for this purpose. Under-the-hood, STAR uses a decentralized set of authorities (e.g., research institutions), secure computation techniques, and an append-only ledger which together enable auditing of scientific claims by 3rd parties and matches real world trust assumptions. We implement and evaluate a construction of STAR using the Microsoft SEAL encryption library and SPDZ multi-party computation protocol. Our experimental evaluation demonstrates the practicality of STAR in multiple real world scenarios as a system for certifying scientific discoveries in a tamper-proof way.
1. Introduction: Addressing the Replication Crisis
The scientific community is currently grappling with a "Replication Crisis," largely driven by the prevalence of p-hacking or data dredging. In these scenarios, researchers may perform numerous statistical tests on a single dataset but selectively report only those that yield "significant" results. This phenomenon is a core component of the Multiple Comparisons Problem (MCP): as the number of tests increases, so does the likelihood of encountering a false positive by pure chance.
While statistical safeguards such as the Bonferroni correction or $\alpha$-investing exist to mitigate these risks, there is currently no mechanism to guarantee their application. Existing solutions rely heavily on institutional trust or the pre-registration of hypotheses, which can be restrictive or easily bypassed. STAR (Statistical Tests with Auditable Results) addresses this gap by creating a framework where data remains encrypted and every test is logged against a predefined "budget" for false discoveries.
2. The STAR System Architecture
STAR facilitates auditable hypothesis testing through a decentralized workflow involving data owners, researchers, and independent auditors. The system ensures that researchers can compute necessary statistics without ever accessing raw, unencrypted data.
The process begins with data owners providing encrypted datasets. Researchers then evaluate statistical tests on this encrypted data. The results are processed through a multi-party computation (MPC) layer, which reveals only the final p-value. Crucially, this p-value is automatically recorded on a public, append-only ledger. This ledger allows third-party auditors to validate the results by checking the sequence of tests against false discovery control procedures.
3. Cryptographic Building Blocks
The robustness of STAR stems from the integration of several advanced cryptographic techniques. These blocks work in tandem to provide access control, hypothesis tracking, and auditability.
3.1 Somewhat Homomorphic Encryption (SHE)
Researchers perform the majority of statistical calculations—such as the sums and products required for t-tests or ANOVA—locally on encrypted data using SHE. This maintains data privacy while allowing for complex mathematical operations.
3.2 Multi-party Computation (MPC)
Because division is computationally expensive in homomorphic encryption, STAR employs a "division trick." A set of independent servers uses MPC to perform only the final division step of a test statistic. This reveals the p-value to the researcher while ensuring that no single server can reconstruct the underlying raw data.
3.3 Append-only Ledger and $\alpha$-Investing
Every revealed p-value is committed to a tamper-proof TestLog. STAR implements $\alpha$-investing, a "streaming" false discovery control procedure. Researchers are allocated an initial "alpha-wealth." Each test consumes a portion of this wealth; significant results "earn" wealth back. Since every test is logged, an auditor can mathematically prove that a researcher did not exceed their allocated budget, thereby verifying the validity of the published claims.
4. Experimental Evaluation
The practicality of STAR was evaluated using both synthetic datasets (ranging from 1,000 to 10,000 rows) and the real-world Abalone dataset from the UCI Machine Learning Repository.
4.1 Performance and Scalability
Computation times for standard tests (t-test, Pearson’s correlation, ANOVA) ranged from a few seconds to approximately nine minutes for the largest datasets. While slower than unencrypted computation, this overhead is negligible in the context of scientific studies that typically span months. Furthermore, the system demonstrated scalability across global server configurations (USA, Germany, Brazil, Japan), proving its viability for international research collaborations.
4.2 Effective FDR Control
In simulations where the data consisted entirely of random noise (null hypotheses), STAR successfully maintained the False Discovery Rate (FDR) below the 0.05 threshold. In contrast, a system lacking STAR’s integrated controls would have produced numerous false discoveries.
5. Discussion and Future Work
While STAR presents a significant step toward tamper-proof science, certain limitations remain. The computational overhead, though manageable for many studies, may still pose challenges for "big data" applications like large-scale genomics. Additionally, the system assumes that data owners do not collude with researchers to bypass cryptographic protections.
Future development will focus on expanding the supported test suite beyond the current four major tests to include more complex models, such as multi-variable regressions and machine learning algorithms. We also aim to address potential "data poisoning" by malicious data owners at the source. By providing a decentralized, auditable framework, STAR offers a path toward restoring trust and rigor in scientific discovery.
Summary updated