Self-Supervised Knowledge Assimilation for Expert-Layman Text Style Transfer

Wenda Xu, Michael Saxon, Misha Sra, William Yang Wang

Introduction

Effective communication between medical professionals and the public is often obstructed by a significant "knowledge gap." Patients typically describe symptoms using vague layman terms, while medical reports are filled with technical jargon that is incomprehensible to non-experts. Bridging this gap through automated text style transfer is a complex NLP challenge. Unlike standard style transfer (e.g., sentiment shifts), medical transfer requires precise domain-specific knowledge—mapping "renal" to "kidney-related"—and suffers from a severe lack of high-quality parallel datasets where the same meaning is expressed in both styles.

Methodology

To address these hurdles, researchers developed a two-stage strategy involving a novel pretraining task and a data-mining pipeline.

Knowledge Base Assimilation (KBA)

The core innovation is Knowledge Base Assimilation (KBA), a self-supervised pretraining task. Using the Unified Medical Language System (UMLS), the authors constructed a "Terminology Bijective Graph" containing pairs of expert and layman terms sharing the same Concept Unique Identifier (CUI). During pretraining, the model is presented with sentences where medical terms have been swapped for their counterparts in the opposite style (e.g., "dyspnea" replaced with "shortness of breath"). The model must then reconstruct the original sentence, forcing it to internalize dictionary-like mappings directly into its weights.

Figure 1: Overview of the KBA pretraining framework and the Terminology Bijective Graph used to bridge expert and layman styles.

Multi-Task Learning and Data Mining

KBA is integrated with standard self-supervised tasks—Mask, Switch, and Delete—to ensure the model maintains general linguistic fluency while learning medical specifics. To facilitate fine-tuning, the authors employed a margin-based criterion using LASER embeddings to extract 11,512 parallel sentence pairs from the non-parallel MSD (Medical Style Dataset). This process turned an unaligned dataset into a supervised training resource.

Key Results

The proposed KBA+SSL model significantly outperformed unsupervised baselines such as DAR and Style Transformer across multiple metrics.

  • Overall Success Rate (OSR): The percentage of sentences succeeding in content preservation, style strength, and fluency improved by 106% relative to the best baseline.
  • Understanding Success Rate (USR): This metric, measuring how easily laypeople could understand the output, reached 0.37 for the KBA+SSL Large model, compared to 0.28 for the Style Transformer.
  • Qualitative Improvements: While baselines often produced nonsensical outputs, the KBA model generated accurate, human-like translations, such as correctly mapping "crackles on auscultation" to "airway narrowing."
Figure 6: Performance metrics across different data percentages, highlighting the Overall Success Rate (OSR) and Understanding Success Rate (USR).

Discussion and Future Work

The research demonstrates that combining terminology-focused pretraining (KBA) with context-focused self-supervision (SSL) is essential. KBA provides the "medical literacy," while SSL ensures grammatical fluency.

However, limitations remain. The current KBA task relies on structured knowledge bases like UMLS, which may not exist for other specialized fields like law. Additionally, the current graph uses 1-to-1 relations; future work could incorporate more complex, multi-node knowledge structures to capture nuanced clinical relationships. Despite these challenges, this approach represents a significant step toward making medical information more accessible to the general public.

Summary updated

Abstract

Expert-layman text style transfer technologies have the potential to improve communication between members of scientific communities and the general public. High-quality information produced by experts is often filled with difficult jargon laypeople struggle to understand. This is a particularly notable issue in the medical domain, where layman are often confused by medical text online. At present, two bottlenecks interfere with the goal of building high-quality medical expert-layman style transfer systems: a dearth of pretrained medical-domain language models spanning both expert and layman terminologies and a lack of parallel corpora for training the transfer task itself. To mitigate the first issue, we propose a novel language model (LM) pretraining task, Knowledge Base Assimilation, to synthesize pretraining data from the edges of a graph of expert- and layman-style medical terminology terms into an LM during self-supervised learning. To mitigate the second issue, we build a large-scale parallel corpus in the medical expert-layman domain using a margin-based criterion. Our experiments show that transformer-based models pretrained on knowledge base assimilation and other well-established pretraining tasks fine-tuning on our new parallel corpus leads to considerable improvement against expert-layman transfer benchmarks, gaining an average relative improvement of our human evaluation, the Overall Success Rate (OSR), by 106%. We release our code and parallel corpus for future research.
BibTeX
@inproceedings{xu2022self,
  author = {Wenda Xu and Michael Saxon and Misha Sra and William Yang Wang},
  title = {{Self-Supervised Knowledge Assimilation for Expert-Layman Text Style Transfer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=zfrjorGtiU5},
}
Figure