Private Randomized Controlled Trials: A Protocol for Industry Scale Deployment

Mahnush Movahedi, Benjamin M. Case, Andrew Knox, Li Li, Sanjay Saravanan, Shubho Sengupta, Erik Taubeneck

Introduction

Randomized Controlled Trials (RCTs) are the gold standard for determining causality, yet in modern digital ecosystems, the data required for these trials is often siloed across different organizations. For instance, in digital advertising, a publisher may hold exposure data (treatment), while an advertiser holds purchase data (outcome). Privacy regulations and competitive interests prevent the direct merging of these datasets.

This paper introduces Private RCT, a high-performance protocol that enables two parties to conduct RCTs without sharing raw user data. By integrating Private Set Intersection (PSI), Multi-Party Computation (MPC), and Differential Privacy (DP), the authors present a system capable of scaling to industry-level datasets exceeding 500 million rows.

Methodology: The Private RCT Pipeline

The proposed pipeline consists of three distinct stages designed to balance privacy, utility, and computational efficiency:

  1. Private Identity Matching: Utilizing the Private-ID (PID) protocol, parties identify the intersection or union of their users via pseudorandom identifiers. This aligns records without revealing specific individual identities.
  2. Scalable Secure Computation (MPC): Aligned data is processed using a sharded version of the EMP-toolkit. By partitioning data across parallel containers, the system overcomes the memory constraints that typically hinder MPC at scale.
  3. Differentially Private Outputs: To prevent information leakage from aggregate results, calibrated noise is added to the final lift estimates and confidence intervals within the secure MPC environment.
High-level overview of the Private RCT pipeline, illustrating the flow from identity matching to differentially private output.

System Architecture and Scalability

A core innovation of this work is the sharded architecture. MPC is traditionally a monolithic bottleneck; however, by deterministically partitioning data into shards and running parallel two-party computation (2PC) games, the authors achieved an 81x speedup over single-threaded baselines.

The sharded MPC architecture, showing how a coordinator distributes data to workers for parallel processing before final aggregation.

The system employs a hybrid cryptographic approach, combining the efficiency of Decisional Diffie-Hellman (DDH) for identity matching with Garbled Circuits for the complex logic of the RCT (e.g., comparing timestamps and values). Furthermore, the authors developed a two-sided DP calibration method to ensure formal privacy guarantees for both participants, even when data sensitivities differ.

Evaluation and Key Results

The protocol was evaluated using synthetic datasets ranging up to 500 million rows and modeled after real-world "Conversion Lift" use cases at Meta. Key findings include:

  • Industry-Scale Performance: The system processed 500 million rows using 50 AWS Fargate containers per party in approximately 4 hours and 12 minutes, at a cost of roughly $1,600.
  • Framework Benchmarking: The EMP-toolkit was found to be the most robust framework, performing 14x faster than SCALE-MAMBA and significantly outperforming ABY, which failed due to memory errors at 100k rows.
  • Linear Scaling: The architecture demonstrated near-linear scaling, where doubling the number of containers roughly halved the execution time, indicating minimal coordination overhead.

Discussion and Future Work

While the protocol demonstrates significant practical utility, it remains sensitive to network latency; a move from 1ms to 60ms latency resulted in a 7-10x slowdown. The current implementation assumes a semi-honest adversarial model, and while "covert security" is proposed via cut-and-choose methods, fully malicious-secure versions at this scale remain a challenge. Future research will also address complex "many-to-many" identity matching to improve the validity of experimental designs in messy real-world data environments.

Summary updated

BibTeX
@inproceedings{movahedi2021private,
  author = {Mahnush Movahedi and Benjamin M. Case and Andrew Knox and Li Li and Yiming Paul Li and Sanjay Saravanan and Shubho Sengupta and Erik Taubeneck},
  title = {{Private Randomized Controlled Trials: A Protocol for Industry Scale Deployment}},
  booktitle = {Ccsw},
  year = {2021},
  doi = {10.1145/3474123.3486764},
}
Figure