Abstract
Difference-in-Differences (DID) is a widely used tool for causal impact evaluation but is constrained by data privacy regulations when applied to sensitive personal information, such as individual-level performance records or healthcare data, that must not be shared with data analysts. Obtaining consent can reduce sample sizes or exclude treated/untreated groups, diminishing statistical power or making estimation impossible. Federated Learning, which shares aggregated statistics to ensure privacy, can address these concerns, but advanced federated DID software packages remain scarce. We derived and developed a federated version of the Callaway and Sant’Anna DID, implemented within the DataSHIELD platform. Our package adheres to DataSHIELD’s security measures and adds extra protections, enhancing data privacy and confidentiality. It reproduces point estimates, asymptotic standard errors, and bootstrapped standard errors equivalent to the non-federated implementation. We demonstrate this functionality on simulated data and real-world data from a malaria intervention in Mozambique. By leveraging federated estimates, we increase effective sample sizes leading to reduced estimation uncertainty, and enable estimation when single data owners cannot share the data but only have access to the treated or untreated group.
The Privacy Bottleneck in Impact Evaluation
Evaluating the impact of a policy or treatment—such as a new healthcare protocol or an educational intervention—often relies on Difference-in-Differences (DID). This method compares the changes in outcomes over time between a treated group and a control group.
However, modern DID requires "individual-level" data, which is often highly sensitive (e.g., patient records). Privacy regulations like GDPR make it difficult or impossible to move this data to a central server for analysis. If a researcher can only access data from one hospital that only has "treated" patients, they cannot perform a DID analysis at all. Furthermore, if only a small subset of people consents to share their data centrally, the resulting "selection bias" can ruin the study's accuracy.
Methods: Federated CSDID
To address these challenges, researchers developed a federated version of the Callaway and Sant’Anna DID (CSDID), an advanced framework that handles multiple time periods and varying treatment timings.
Technical Implementation
The tool is implemented within the DataSHIELD ecosystem, where the analysis is "taken to the data." Instead of pooling raw data, the analyst sends R commands to remote servers; the servers compute local summary statistics and send only those aggregates back.
Key technical components include:
- Doubly Robust (DR) Estimation: The tool supports DR, Inverse-Probability Weighting (IPW), and Regression Outcome (RO) methods to ensure results are valid even if parts of the statistical model are slightly misspecified.
- Privacy Enhancements: To prevent "covariance-based attacks"—where a malicious analyst might try to reverse-engineer individual data—the authors implemented row-wise shuffling of data frames and restricted the transmission of known numbers to the servers.
- Federated Multiplier Bootstrap: The team successfully adapted the bootstrap method to work across distributed servers, ensuring that the "uncertainty" of the estimate is as accurate as if the data were pooled.
Key Results and Insights
The tool was validated using simulated data (801 individuals across six servers) and real-world data from a malaria intervention study in Mozambique.
- Mathematical Equivalence: The federated CSDID reproduces the exact point estimates and standard errors of a traditional pooled analysis. The federated package reproduced the average treatment effect with a Maximal Absolute Error of $5.35 \times 10^{-14}$ compared to the non-federated version.
- Enabling the "Impossible": Federated learning allows for "cross-site" DID. If Server A has only treated individuals and Server B has only controls, neither can perform an impact evaluation alone. The federated approach bridges this gap without either side seeing the other's raw data.
- Power in Numbers: By leveraging data from multiple sites, the federated model significantly reduces estimation uncertainty (narrower confidence intervals) compared to "study-level" analyses where each site works in isolation.
Conclusion and Future Work
This implementation proves that federated learning isn't just an approximation; it can be a mathematically exact and secure alternative to centralized data pooling. While currently optimized for panel data (tracking the same individuals over time), future work aims to extend this to repeated cross-sections and incorporate synthetic control methods into the federated toolkit.
Summary updated