Evaluating the Utility of Model Configurations and Data Augmentation on Clinical Semantic Textual Similarity

Yuxia Wang, Fei Liu, Karin Verspoor, Timothy BaldwinBioNLP 2020 · DOI · OpenReview

TL;DR

This paper explores how to best adapt pre-trained language models like BERT for clinical Semantic Textual Similarity (STS) tasks, which are often hampered by limited data. They introduce a novel hierarchical convolutional architecture and two data augmentation techniques (segment reordering and back-translation) that significantly improve performance on clinical STS benchmarks.

What Problem Does This Paper Address?

Semantic Textual Similarity (STS) is crucial for understanding the degree of meaning overlap between two text snippets. While powerful pre-trained language models (LMs) like BERT have revolutionized NLP, applying them to specialized domains like clinical text presents challenges. Clinical STS datasets are typically small ("low-resource"), making standard fine-tuning prone to overfitting and impractical due to computational constraints. The paper aims to find model configurations and data augmentation strategies that effectively leverage LMs for clinical STS without requiring massive datasets or extensive computational resources.

Methods

The authors investigate several key aspects of adapting BERT for clinical STS:

  • Hierarchical Convolutional Architecture (HConv-BERT): Instead of relying solely on the CLS token for sentence representation (CLS-pooling), they propose incorporating hierarchical convolutional filters. These filters are applied to the BERT encoder's output to capture local interactions and build richer feature representations, making the model less prone to overfitting on small datasets.
  • Model Flexibility (Fine-tuning Strategies): They explore different degrees of flexibility during fine-tuning by varying the number of transformer blocks that are allowed to update their weights. This helps identify the optimal balance between adapting to the task and avoiding overfitting.
  • Data Augmentation: To combat data sparsity, they introduce two methods:
  • Segment Reordering (SR): This technique permutes segments within a sentence (delimited by commas or semicolons) to create new training examples.
  • Back Translation (BT): Sentences are translated into a distant language (Chinese) and then back into English using machine translation tools. This aims to introduce linguistic diversity.

Datasets

  • N2C2-STS: This is the primary dataset used for training and evaluation. It's an extension of the MedSTS dataset, specifically designed for clinical STS, and contains a limited number of annotated sentence pairs (1.6K).
  • STS-B: Used for general-domain STS evaluation and comparison.
  • SICK-R: Another general-domain dataset used for evaluating the impact of training data size.
  • BIOSSES: A biomedical literature STS dataset used for cross-domain generalization evaluation.

Key Insights and Ideas

  • Domain-Specific Fine-tuning is Less Impactful: Surprisingly, domain-specific fine-tuning on clinical STS yielded less improvement than expected compared to general-domain STS. The authors hypothesize this is due to the inherent richness and complexity of clinical language.
  • Hierarchical Convolutions Offer a Better Representation: The HConv-BERT architecture, by using convolutional filters to process BERT outputs, significantly outperforms standard CLS-pooling for STS tasks, especially in low-resource settings. This suggests that capturing local textual patterns is crucial for STS.
  • Optimal Model Flexibility is Data-Dependent: The ideal number of trainable transformer layers for fine-tuning varies with dataset size. Smaller datasets require more layers to be trainable to achieve peak performance, but this also increases the risk of overfitting.
  • Data Augmentation is Crucial for Clinical STS: Both SR and BT methods provided substantial improvements, with BT showing larger gains. This highlights the critical need for data augmentation in low-resource clinical NLP tasks.

Key Results

  • The HConv-BERT architecture achieved substantial performance gains over feature-based approaches, with Pearson correlations reaching up to 79.4/74.4 on N2C2-STS (r/p x 100).
  • When fine-tuned, HConv-BERT and CLS-BERT showed comparable performance, but HConv-BERT often reached its peak with fewer trainable layers, making it more efficient.
  • Data augmentation, particularly back-translation (BT), consistently boosted performance across all tested models on N2C2-STS. The best-performing model, ConvBERTSTS-B + BT, achieved results comparable to the state-of-the-art IBM-N2C2 approach.
  • The proposed method demonstrated strong generalizability on the BIOSSES dataset, outperforming the baseline by significant margins.

Limitations and Future Work

  • Segment Reordering Naivety: The authors acknowledge that their segment reordering implementation is "naïve" and can sometimes result in unnatural sentence sequences. They suggest future work could involve filtering these irregular statements using a language model pre-trained on clinical corpora.
  • Back Translation Language Choice: They propose exploring the impact of using target languages other than Chinese for back-translation in future work.
  • Cross-Domain Generalization: While the method showed good generalization, further investigation into cross-domain transfer learning for clinical STS could be beneficial.

Summary updated

Abstract

In this paper, we apply pre-trained language models to the Semantic Textual Similarity (STS) task, with a specific focus on the clinical domain. In low-resource setting of clinical STS, these large models tend to be impractical and prone to overfitting. Building on BERT, we study the impact of a number of model design choices, namely different fine-tuning and pooling strategies. We observe that the impact of domain-specific fine-tuning on clinical STS is much less than that in the general domain, likely due to the concept richness of the domain. Based on this, we propose two data augmentation techniques. Experimental results on N2C2-STS 1 demonstrate substantial improvements, validating the utility of the proposed methods.
BibTeX
@inproceedings{wang2020evaluating,
  author = {Yuxia Wang and Fei Liu and Karin Verspoor and Timothy Baldwin},
  title = {{Evaluating the Utility of Model Configurations and Data Augmentation on Clinical Semantic Textual Similarity}},
  booktitle = {BioNLP},
  year = {2020},
  doi = {10.18653/v1/2020.bionlp-1.11},
}
Figure