Generating Scientific Definitions with Controllable Complexity

Tal August, Katharina Reinecke, Noah A. SmithACL 2022 · OpenReview

Abstract

Scientific communication often suffers from a "jargon barrier" that limits accessibility for non-experts. This paper introduces a novel natural language processing task: generating scientific definitions that can be automatically adjusted for complexity based on the reader's background. By framing definition generation as an abstractive question-answering task and employing a novel reranking approach, the researchers successfully produced definitions ranging from "science news" simplicity to "journal abstract" density. While the system achieves high fluency and relevance, the study also highlights significant challenges regarding factual accuracy in technical domains.

Introduction

The ability to understand scientific concepts is frequently hindered by specialized terminology. A definition suitable for a researcher may be unintelligible to a student, and standard dictionary entries often lack the necessary context or flexibility. To address this, we explore the challenge of controllable complexity in scientific definition modeling. Unlike general-purpose words, scientific terms are often esoteric and context-dependent, requiring systems that can adapt explanations to varying levels of technical expertise.

Overview of the controllable definition generation task, where a question and journal abstracts are used to produce definitions of varying complexity.

Methodology

The task is framed as an abstractive Question Answering (QA) problem: "What is (are) X?"

Basic Generation and Support Documents

Initial experiments involved fine-tuning architectures like BART and GPT-2, alongside out-of-the-box GPT-3. A key finding was that providing "support documents"—specifically, the top 10 related scientific abstracts retrieved from the S2ORC corpus—significantly enhanced model performance. This context allows models to define rare or emerging terms not well-represented in their pre-training data.

Controllable Complexity via Reranking

To adjust the difficulty of the output, the authors compared standard controllable generation techniques (e.g., PPLM, DExperts) against a novel Reranking approach. In this method, a fine-tuned BART model generates 100 candidate definitions. A discriminator (either a SciBERT model or a Linear SVM) then scores these candidates based on their stylistic resemblance to either complex scientific journals or simpler science news articles. The highest-scoring candidate for the target complexity level is selected.

Datasets

The study introduces a benchmark dataset of 8,263 terms and definitions sourced from MedQuAD and Wikipedia science glossaries. To train complexity discriminators, a corpus of scientific journal abstracts (high complexity) and science news articles (low complexity) was utilized.

Results

Performance and Complexity Control

The BART model with support documents (BART SD) emerged as the strongest baseline, achieving a ROUGE-L of 0.30 and a BERTscore of 0.89. Human evaluations favored the SVM-Reranker for its ability to control complexity while maintaining high fluency (3.71/4) and relevance (3.51/4).

Comparison of complexity metrics across different control methods, highlighting the effectiveness of the Reranking approach.

As shown in the quantitative analysis, the reranking methods successfully shifted Academic Vocabulary List (AVL) usage by approximately 10% between modes, mirroring the difference between professional journals and news media.

The Factuality Hurdle

Despite successes in style control, factual accuracy remains a significant concern. Models frequently "hallucinated" incorrect information, such as misidentifying benign lesions as cancerous. The reranking approach proved safer than more aggressive steering methods like DExperts, which exhibited error rates as high as 86%. Even with reranking, approximately 38% of generations contained factual inaccuracies.

Discussion and Conclusion

Reranking offers a "best of both worlds" solution by leveraging the base model's fluency while using a discriminator as a stylistic filter. However, the high rate of hallucinations (60% across all models in human evaluation) indicates that these systems are not yet ready for high-stakes deployment in medical or scientific settings. Future work should focus on improving factual grounding and exploring a continuous spectrum of complexity beyond binary "high/low" classifications.

Summary updated

BibTeX
@inproceedings{august2022generating,
  author = {Tal August and Katharina Reinecke and Noah A. Smith},
  title = {{Generating Scientific Definitions with Controllable Complexity}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=fU_OyUtMBjL},
}
Figure