Enhancing Robustness of Foundation Model Representations under Provenance-related Distribution Shifts

Xiruo Ding, Zhecheng Sheng, Brian Hur, Feng Chen, Serguei Pakhomov, Trevor Cohen

Abstract

Foundation models are a current focus of attention in both industry and academia. While they have shown their capabilities in a variety of tasks, in-depth research is required to determine their robustness to distribution shift when used as a basis for supervised machine learning. This is especially important in the context of clinical data, with particular limitations related to data accessibility, lack of pretraining materials, and limited availability of high-quality annotations. In this work, we examine the stability of models based on representations from foundation models under distribution shift. We focus on confounding by provenance, a form of distribution shift that emerges in the context of multi-institutional datasets when there are differences in source-specific language use and class distributions. Using a sampling strategy that synthetically induces varying degrees of distribution shift, we evaluate the extent to which representations from foundation models result in predictions that are inherently robust to confounding by provenance. Additionally, we examine the effectiveness of a straightforward confounding adjustment method inspired by Pearl's conception of backdoor adjustment. Results indicate that while foundation models do show some out-of-the-box robustness to confounding-by-provenance related distribution shifts, this can be considerably improved through adjustment. These findings suggest a need for deliberate adjustment of predictive models using representations from foundation models in the context of source-specific distributional differences.

Introduction

In high-stakes domains such as clinical medicine, researchers frequently aggregate data from multiple institutions to train robust machine learning models. However, this multi-source approach introduces a significant challenge: confounding by provenance. This occurs when a model learns to associate site-specific linguistic patterns or administrative jargon with clinical outcomes, rather than learning the underlying medical indicators. For example, if Hospital A serves a high-risk population and Hospital B serves a low-risk one, a model might use Hospital A's unique note-taking style as a shortcut for predicting high risk. When deployed to a new environment with different patient distributions, such models often fail.

This study investigates whether the rich, general-purpose representations provided by Foundation Models (FMs) like Llama 2 and BERT are inherently resistant to these site-specific shortcuts, or if explicit statistical interventions are still required to ensure reliable generalization.

Methodology

Representation Extraction and Classification

To evaluate the models, the researchers extracted "contextual embeddings"—frozen numerical representations of text—from various foundation models. These embeddings served as inputs for a logistic regression classifier. This approach avoids the high computational cost of end-to-end fine-tuning while allowing for an assessment of the information encoded within the FM's latent space.

Causal Backdoor Adjustment

To mitigate site-specific bias, the study implemented a method inspired by Judea Pearl’s causal inference framework. By treating the data source (provenance) as a confounding variable during training and applying a "backdoor adjustment" during prediction, the researchers aimed to decouple the clinical predictions from site-specific linguistic cues.

Synthetic Distribution Shift

The researchers developed a sampling strategy to synthetically induce varying degrees of distribution shift. This allowed them to measure the "instability coefficient"—the slope of performance (AUPRC) relative to the degree of shift ($\alpha_{test}$). A perfectly robust model would maintain a slope of zero.

Visualization of the sampling strategy used to synthetically induce distribution shifts by varying class and provenance distributions.

Experimental Results

Innate Robustness of Foundation Models

The results demonstrate that larger foundation models offer superior innate robustness compared to simple baselines. For instance, Llama 2-7B embeddings exhibited an instability coefficient of 0.013 in certain high-prevalence scenarios, significantly outperforming Sentence-BERT’s coefficient of 0.028. However, even these advanced models were not immune to performance degradation as the test-site distribution diverged from the training data.

Impact of Causal Adjustment

Applying backdoor adjustment acted as a significant "force multiplier" for stability across all models. As shown in the comparative analysis, adjustment flattened the performance slope dramatically. For Sentence-BERT on the SHAC clinical dataset, the instability coefficient dropped from 0.035 to a nearly stable 0.005.

Comparison of model performance (AUPRC) across varying distribution shifts for Binary Unigram, Sentence-BERT, and Llama 2-7B. 'Adjusted' plots show the impact of Pearl's backdoor adjustment.

While the simple Binary Unigram baseline benefited the most from adjustment in terms of relative stability, the foundation models (Sentence-BERT and Llama 2) maintained higher absolute performance (AUPRC) across all test conditions. This suggests that while FMs provide better features, they still require explicit causal modeling to resist the temptation of "easy" site-specific shortcuts.

Conclusion

This research highlights that even state-of-the-art foundation models are susceptible to provenance-related bias. While their general-purpose representations provide a stronger foundation for robustness than traditional methods, they are not a substitute for sound statistical practices. The findings advocate for the deliberate use of causal adjustment techniques when deploying FM-based models in multi-institutional settings to ensure reliable and equitable performance across diverse clinical environments.

Summary updated

Abstract

Foundation models are a current focus of attention in both industry and academia. While they have shown their capabilities in a variety of tasks, in-depth research is required to determine their robustness to distribution shift when used as a basis for supervised machine learning. This is especially important in the context of clinical data, with particular limitations related to data accessibility, lack of pretraining materials, and limited availability of high-quality annotations. In this work, we examine the stability of models based on representations from foundation models under distribution shift. We focus on confounding by provenance, a form of distribution shift that emerges in the context of multi-institutional datasets when there are differences in source-specific language use and class distributions. Using a sampling strategy that synthetically induces varying degrees of distribution shift, we evaluate the extent to which representations from foundation models result in predictions that are inherently robust to confounding by provenance. Additionally, we examine the effectiveness of a straightforward confounding adjustment method inspired by Pearl's conception of backdoor adjustment. Results indicate that while foundation models do show some out-of-the-box robustness to confounding-by-provenance related distribution shifts, this can be considerably improved through adjustment. These findings suggest a need for deliberate adjustment of predictive models using representations from foundation models in the context of source-specific distributional differences.
BibTeX
@misc{ding2023enhancing,
  author = {Xiruo Ding and Zhecheng Sheng and Brian Hur and Feng Chen and Serguei Pakhomov and Trevor Cohen},
  title = {{Enhancing Robustness of Foundation Model Representations under Provenance-related Distribution Shifts}},
  year = {2023},
  eprint = {2312.05435},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.05435},
}
Figure