Sparks: Inspiration for Science Writing using Language Models

Katy Gero, Vivian Liu, Lydia ChiltonACL 2022

TL;DR

This paper introduces "Sparks," a system that generates inspiring sentences for science writers using large language models (LLMs). A user study with STEM graduate students revealed three key ways sparks are used: for inspiration, translation, and perspective-taking, each linked to distinct interaction patterns. The research highlights how LLMs can effectively support creative yet constrained writing tasks like science communication.

What Problem Does This Paper Address?

Science writing, while crucial for disseminating knowledge, is a challenging task. It requires writers to be both creative in explaining complex ideas and constrained by scientific accuracy and clarity. While large language models (LLMs) have shown impressive capabilities in various text generation tasks, their application to specialized domains like science writing, especially in a way that genuinely aids human creativity, remains an open question. Existing LLMs can sometimes produce generic, repetitive, or even factually incorrect outputs, posing challenges for their use in high-stakes communication. This paper investigates how LLMs can be leveraged to specifically support science writers by providing novel ideas and perspectives, addressing the need for tools that can enhance, rather than replace, human expertise in this domain.

Methods

The core of this work is the "Sparks" system, which generates short, inspiring sentences related to a scientific concept. The system utilizes a pre-trained large language model. A key methodological contribution is a custom decoding method designed to produce sparks that are both coherent and diverse. This method aims to go beyond standard LLM output generation to specifically foster creative inspiration. The effectiveness of this system was evaluated through a user study.

Datasets

Not discussed in the paper. The paper focuses on the system's design and a user study, rather than training LLMs on specific science writing datasets.

Key Insights and Ideas

  • "Sparks" as a Creative Catalyst: The system doesn't aim to write for the user but to provide "sparks" – short, evocative sentences that can jumpstart a writer's thinking process. This positions LLMs as collaborators rather than replacements.
  • Three Distinct Use Cases Emerge: The user study revealed that participants used sparks in three primary ways:
  • Inspiration: Generating new ideas or angles for their writing.
  • Translation: Helping to rephrase complex scientific jargon into more accessible language.
  • Perspective: Offering different ways to frame or view a scientific concept.
  • Interaction Patterns Mirror Use Cases: Each of these use cases correlated with unique ways participants interacted with the system, suggesting that the utility of sparks is tied to the writer's specific need at that moment.
  • Quality vs. Satisfaction Nuance: While participants tended to select higher-quality sparks, the overall quality of sparks presented to a user didn't directly correlate with their satisfaction with the tool. This suggests that the potential for inspiration and the ability to find useful sparks might be more important than the average quality of all generated sparks.

Key Results

The paper reports on a user study with 13 STEM graduate students who used the "Sparks" system to write "tweetorials" (short, explanatory Twitter threads on scientific topics). The study found:

  • The "Sparks" system generates outputs that are more coherent and diverse than a baseline method and approach a "human gold standard" in quality.
  • Three main use cases for sparks were identified: inspiration, translation, and perspective.
  • These use cases were associated with distinct interaction patterns among the users.
  • Participant satisfaction was not directly correlated with the overall quality of sparks presented, but rather with their ability to find useful sparks for their specific needs.

Limitations and Future Work

  • Limited Scope of "Sparks": The current system generates short sentences. Future work could explore generating longer passages or different types of content to support writers.
  • Generalizability: The study focused on "tweetorials," a specific science writing format. Further research is needed to understand how "Sparks" might support other forms of science communication.
  • User Satisfaction Nuance: The finding that overall spark quality didn't correlate with satisfaction suggests a need to explore how to better tailor spark generation or presentation to individual user needs and preferences. Future work could investigate personalized spark generation or more interactive ways for users to refine the sparks they receive.

Summary updated

BibTeX
@inproceedings{gero2022sparks,
  author = {Katy Gero and Vivian Liu and Lydia Chilton},
  title = {{Sparks: Inspiration for Science Writing using Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure