TL;DR
This paper introduces a novel framework for privacy-preserving unsupervised domain adaptation (UDA) in clinical text analysis. It addresses the critical issue of data privacy by sharing only differentially private source feature distributions, rather than raw data, to adapt models to new clinical datasets, achieving strong privacy guarantees with minimal performance impact.
What Problem Does This Paper Address?
Unsupervised Domain Adaptation (UDA) is a powerful technique for transferring knowledge from a labeled source domain to an unlabeled target domain, especially useful when obtaining labeled data for the target domain is difficult or expensive. In clinical text analysis, this could mean adapting a model trained on one hospital's electronic health records (EHRs) to another hospital's records, which often have different data distributions (e.g., variations in terminology, patient demographics, or reporting styles).
However, standard UDA methods typically require sharing the source data with the target client for training. This poses a significant privacy risk, particularly with sensitive clinical data, where sharing raw patient information is often prohibited by regulations. Even sharing just the source model's feature distributions can be vulnerable to membership inference attacks, where an adversary tries to determine if a specific individual's data was part of the training set. This paper tackles the challenge of enabling effective UDA while rigorously protecting the privacy of the source data.
Methods
The authors propose a multi-pronged approach to achieve privacy-preserving UDA:
- Source Feature Distribution Sharing: Instead of sharing the raw source data, they propose sharing the distribution of source features. This is modeled using Gaussian Mixture Models (GMMs), which capture the statistical properties of the features without revealing individual data points.
- Differential Privacy (DP) Pretraining: To further enhance privacy against membership inference attacks on the GMM itself, they introduce a novel differential privacy training strategy for the source feature extractor. This involves using DP-SGD (Differentially Private Stochastic Gradient Descent) during the initial pretraining of the source model. DP guarantees that the output of the model is statistically indistinguishable whether or not a specific individual's data was included in the training set, within a defined privacy budget (ε, δ).
- Knowledge Distillation (KD) for Overfitting Mitigation: Clinical text data, especially when using powerful models like BERT, can be prone to overfitting to the source domain. To combat this, they employ Knowledge Distillation (KD) loss. This technique helps to "smooth" the outputs of the target model by encouraging it to mimic the softened predictions of a teacher model (in this case, the source model), thereby improving generalization to the target domain.
- Information Maximization (IM) for Target Adaptation: Since the target data is unlabeled, supervised signals are unavailable. To guide the adaptation process, they incorporate an Information Maximization (IM) loss, which encourages the model to learn features that are both discriminative within classes and distinct between classes.
The overall framework involves pretraining the source model with DP, extracting DP-GMMs from its features, and then using these GMMs to resample source features for adaptation on the target client, combined with KD and IM losses.
Datasets
The paper evaluates their proposed method on two clinical text datasets for pulmonary disease label extraction:
- MIMIC-CXR Database: A large, publicly available dataset of chest radiographs with free-text reports.
- Open-i collection: Another dataset of chest radiographs and associated reports.
These datasets are used to infer findings from clinical reports, treated as a multi-label classification task. The authors manually select common disease labels to mitigate differences between the datasets.
Key Insights and Ideas
- GMMs as a Privacy-Friendly Representation: Using GMMs to represent source feature distributions is a clever way to abstract away individual data points while retaining useful statistical information for adaptation.
- DP for Robust Privacy: Applying differential privacy during the initial pretraining phase is crucial for defending against sophisticated membership inference attacks that could exploit even statistical summaries of the source data.
- KD for Clinical Text UDA: Knowledge Distillation proves effective in mitigating BERT's overfitting tendencies in UDA for clinical text, a domain known for its complexity and potential for noisy labels.
- The "Noisy Source Data" Hypothesis: The paper suggests that UDA can sometimes outperform direct source-to-target evaluation when the source data is noisy. The GMM-based regularization in their method helps to smooth out these noisy distributions, leading to better adaptation.
Key Results
- Privacy Preservation: The proposed method successfully preserves source data privacy, as demonstrated by the use of differential privacy with a tunable privacy budget (ε).
- Performance Utility: The framework achieves competitive performance on the pulmonary disease label extraction task, with only a minor drop in F1-score compared to methods that use the full source data (when privacy is not a concern). For instance, in Table I, the "w/o source+DP" (their method) often performs comparably or even better than "w source" in certain classes, especially when the source data is noisy.
- Robustness to Noise: The method shows promising results even on noisy clinical text datasets, outperforming baseline "no adapt" and "with source" scenarios in several cases.
- Impact of Privacy Budget: Table II and Figure 6 show that as the privacy budget (ε) increases (meaning weaker privacy guarantees), the performance of the DP-pretraining method approaches that of the non-DP version, indicating a trade-off between privacy and utility that can be managed.
- GMM Cluster Number: Figure 7 suggests that the number of clusters (K) in the GMM is an important hyperparameter, with an optimal value around 5-6 for the tested scenario.
Limitations and Future Work
- Minor Performance Drop: While the performance drop is minor, it is still present, especially with stronger privacy guarantees (smaller ε). The authors acknowledge this trade-off.
- Hyperparameter Tuning: The optimal number of GMM clusters (K) needs to be determined, and its impact can vary across tasks.
- Future Work: The paper suggests exploring the same K value in other clinical text datasets and investigating the impact of the number of GMM clusters further. They also mention that their method is compatible with various UDA backbones, implying potential for integration with other advanced UDA techniques.
Summary updated