TL;DR
This paper introduces "Private RCT," a novel protocol for conducting Randomized Controlled Trials (RCTs) while preserving the privacy of sensitive user data. It achieves this by combining three key privacy technologies—private set intersection, multiparty computation, and differential privacy—to enable collaborative analysis between parties who don't want to share their raw data. This allows for large-scale, privacy-preserving RCTs, demonstrated by a deployment capable of handling hundreds of millions of users.
What Problem Does This Paper Address?
Randomized Controlled Trials (RCTs) are the gold standard for establishing causal relationships in fields like medicine, social science, and marketing. However, many real-world RCTs involve highly sensitive data, creating a significant privacy challenge. Often, one party controls the treatment assignment (e.g., which users see an ad), while another party measures the outcomes (e.g., purchases). These parties may not want to share their individual-level data with each other due to privacy concerns or competitive reasons. This prevents them from collectively learning robust causal effects. Existing privacy-preserving computation methods, when used in isolation, struggle to scale to industry-level datasets or are too restrictive for the complex computations needed for RCTs.
Methods
The paper proposes a three-stage protocol called "Private RCT" to address these challenges:
- Private Identity Matching: This stage privately identifies and joins users across the two parties' datasets. It utilizes protocols like Private-ID or Private Secret Shared Set Intersection (PS³I) to create an encrypted join of users without revealing individual identities or the extent of overlap. This is crucial for linking treatment and outcome data.
- Scalable Secure Computation: Here, a general-purpose Multi-Party Computation (MPC) framework is used to perform the actual RCT analysis on the joined, encrypted data. The paper highlights the use of sharding techniques to scale this computation to massive datasets (up to 500 million rows) by distributing the workload across multiple containers.
- Differential Privacy (DP) for Output: To prevent any leakage of information from the final aggregated results, differential privacy is applied. This involves adding carefully calibrated noise to the statistical estimates of causal effects and their uncertainty measures, ensuring that neither party can infer sensitive information about the other's private inputs.
The protocol combines Decisional Diffie-Hellman (DDH) based Private Set Intersection (PSI), Secure Multi-Party Computation (MPC), and Differential Privacy (DP).
Datasets
The paper does not explicitly mention specific real-world datasets used for training. However, it extensively uses synthetic data for evaluation, particularly for the "Conversion Lift" application, simulating datasets with up to 500 million rows. The evaluation framework itself is designed to be applicable to various RCT scenarios.
Key Insights and Ideas
- Synergy of Privacy Techniques: The core insight is that combining PSI, MPC, and DP in a layered approach is more effective and scalable than relying on any single technique. Each stage addresses a specific privacy leakage point: the join, the computation, and the output release.
- Scalability through Sharding: A significant contribution is the development of a privacy-preserving sharding mechanism that allows MPC frameworks to handle massive datasets (500 million rows) by distributing computation across multiple containers without compromising privacy.
- Practical Deployment Focus: The paper emphasizes building a practical, end-to-end protocol that can be deployed in real-world industry settings, demonstrated by their "Conversion Lift" product.
- Two-Sided Privacy Guarantees: The protocol is designed to provide privacy for both parties, ensuring that neither can learn the other's sensitive input data, even from the aggregated output. This is achieved through careful calibration of DP noise.
Key Results
- Scalability: The protocol successfully scales to handle datasets of 500 million rows and over a billion gates, demonstrating its suitability for industry-scale deployments.
- Performance: The sharded EMP-toolkit implementation achieved significant speedups, with an 81x improvement over single-threaded versions for large datasets. Performance tests showed that running multiple containers linearly decreases running time with negligible coordination overhead.
- Practical Application: The paper details the implementation of "Private Conversion Lift," a product used by Facebook to measure advertising effectiveness, showcasing the real-world applicability of their protocol.
- Framework Evaluation: The authors conducted a thorough evaluation of various MPC frameworks, ultimately selecting EMP-toolkit for its performance and scalability, and providing a valuable comparison for the community.
Limitations and Future Work
- Semi-Honest Adversary Model: The primary adversarial model considered is "semi-honest," meaning parties follow the protocol but try to learn extra information. The paper acknowledges that malicious adversaries (who might deviate from the protocol) are a more complex challenge.
- Complexity of Many-to-Many Matching: While the paper extends identity matching to handle many-to-many relationships, it notes that this can introduce "contaminated" users that need to be handled carefully, especially in the context of RCT validity.
- Future Directions: The authors suggest future work could explore more robust handling of malicious adversaries, further optimization of MPC frameworks, and extending the protocol to support more than two parties for more complex collaborative studies. They also highlight the need for continued development in MPC libraries to address memory limits and compiler-protocol coupling.
Summary updated