STAR: Statistical Tests with Auditable Results

Sacha Servan-Schreiber, Olga Ohrimenko, Tim Kraska, Emanuel Zgraggen

Abstract: We present STAR: a novel system aimed at solving the complex issue of "p-hacking" and false discoveries in scientific studies. STAR provides a concrete way for ensuring the application of false discovery control procedures in hypothesis testing, using mathematically provable guarantees, with the goal of reducing the risk of data dredging. STAR generates an efficiently auditable certificate which attests to the validity of each statistical test performed on a dataset. STAR achieves this by using several cryptographic techniques which are combined specifically for this purpose. Under-the-hood, STAR uses a decentralized set of authorities (e.g., research institutions), secure computation techniques, and an append-only ledger which together enable auditing of scientific claims by 3rd parties and matches real world trust assumptions. We implement and evaluate a construction of STAR using the Microsoft SEAL encryption library and SPDZ multi-party computation protocol. Our experimental evaluation demonstrates the practicality of STAR in multiple real world scenarios as a system for certifying scientific discoveries in a tamper-proof way.

Introduction: Addressing the Replication Crisis

The scientific community is currently grappling with a "Replication Crisis." A 2016 survey revealed that over 70% of researchers have failed to reproduce another scientist's results. A primary driver of this crisis is the Multiple Comparisons Problem (MCP), often manifesting as "p-hacking" or "data dredging."

When a researcher tests numerous hypotheses on a single dataset, the laws of probability suggest they will eventually find a "significant" result purely by chance. While statistical corrections like the Bonferroni method exist, there is currently no mechanism to prove a researcher actually applied them. The current scientific model relies on an "honor system" where we trust that a published p-value was not the result of dozens of undisclosed, failed attempts. STAR aims to replace this trust with a "trust-but-verify" cryptographic architecture.

System Architecture

STAR creates a tamper-proof environment for hypothesis testing by ensuring that every interaction with a dataset is recorded and validated against strict statistical rules. The workflow involves data owners, researchers, a decentralized set of computing authorities, and a public ledger.

The STAR system workflow from data ownership to public auditing.

In this pipeline, data remains encrypted at all times. Researchers submit statistical tests to a multi-party computation (MPC) cloud. The results are then recorded in a public log, allowing any third-party auditor to validate that the researchers followed proper false discovery control procedures.

Cryptographic Foundations

STAR achieves its goals through a hybrid of four core technologies, each serving a specific security or statistical property:

  1. $\\alpha$-investing: A sequential discovery control procedure. Unlike traditional methods, $\\alpha$-investing treats significance as a "wealth" resource. If a test fails, wealth is lost; if it succeeds, wealth is earned. If a researcher runs out of wealth, they can no longer claim significance for further tests.
  2. Somewhat Homomorphic Encryption (SHE): Using the Microsoft SEAL library, data owners encrypt their datasets. Researchers can perform arithmetic operations on this encrypted data locally, allowing for the "heavy lifting" of test statistics (like sums and squares) without seeing raw data.
  3. Multi-party Computation (MPC): STAR uses the SPDZ protocol to handle operations that are inefficient in SHE, such as division. This process is distributed across decentralized authorities (e.g., different universities), ensuring no single entity can forge a result.
  4. The TestLog: Every test is recorded on an append-only, tamper-proof ledger, creating an immutable trail of every hypothesis tested against a dataset.
Mapping of STAR's core properties to their cryptographic building blocks.

Evaluation and Results

The authors evaluated STAR using both synthetic datasets (ranging from 1,000 to 10,000 rows) and the real-world Abalone Dataset from the UCI Machine Learning Repository.

Key Insights

  • Hybrid Efficiency: By performing bulk arithmetic via SHE and only using MPC for the final "division gate" and decryption, the system is orders of magnitude faster than pure MPC approaches.
  • Decentralized Trust: The system remains secure as long as at least one server in the decentralized network is honest, matching real-world institutional trust assumptions.
  • Practical Performance: While slower than unencrypted computation, STAR is practical for scientific publishing. Runtimes for common tests (t-test, ANOVA) ranged from seconds to approximately 9 minutes for large datasets. Even in a "worst-case" global network scenario (servers in the USA, Germany, Brazil, and Japan), the system remained feasible.

Limitations and Future Work

While STAR provides a robust framework, some challenges remain. The system assumes the data owner does not collude with researchers to run "pre-tests" in secret. Additionally, while the data values are hidden, metadata (attribute types and ranges) could potentially leak information. Future work aims to expand the test suite beyond the current four major tests and reduce the initial setup time required for encrypting and uploading large datasets.

Summary updated

Abstract

We present STAR: a novel system aimed at solving the complex issue of "p-hacking" and false discoveries in scientific studies. STAR provides a concrete way for ensuring the application of false discovery control procedures in hypothesis testing, using mathematically provable guarantees, with the goal of reducing the risk of data dredging. STAR generates an efficiently auditable certificate which attests to the validity of each statistical test performed on a dataset. STAR achieves this by using several cryptographic techniques which are combined specifically for this purpose. Under-the-hood, STAR uses a decentralized set of authorities (e.g., research institutions), secure computation techniques, and an append-only ledger which together enable auditing of scientific claims by 3rd parties and matches real world trust assumptions. We implement and evaluate a construction of STAR using the Microsoft SEAL encryption library and SPDZ multi-party computation protocol. Our experimental evaluation demonstrates the practicality of STAR in multiple real world scenarios as a system for certifying scientific discoveries in a tamper-proof way.
BibTeX
@misc{servanschreiber2019star,
  author = {Sacha Servan-Schreiber and Olga Ohrimenko and Tim Kraska and Emanuel Zgraggen},
  title = {{STAR: Statistical Tests with Auditable Results}},
  year = {2019},
  eprint = {1901.10875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1901.10875},
}
Figure