Abstract
Ehud Reiter, Albert Gatt, François Portet, Marian van der Meulen. Proceedings of the Fifth International Natural Language Generation Conference.
2008.
Introduction
In a Neonatal Intensive Care Unit (NICU), medical staff are inundated with a constant stream of high-frequency sensor data and discrete clinical logs. While traditional graphical visualizations are the standard for monitoring, research suggests that textual summaries can often lead to faster and more accurate clinical decisions. However, generating these summaries automatically remains a significant challenge. This study evaluates BT-45, a Natural Language Generation (NLG) system designed to summarize 45-minute windows of complex neonatal data, comparing its performance against both human experts and standard visualizations.
The BT-45 System and Methodology
The BT-45 system processes data through a four-stage pipeline: signal analysis (identifying patterns), data abstraction (grouping events into clinical concepts), document planning (selecting medically significant events), and microplanning/realisation (converting plans into English).
The evaluation involved 35 medical professionals who reviewed 24 real-world NICU scenarios. Participants were presented with either a graphical visualization, a human-written summary, or a BT-45 generated summary. Their task was to identify appropriate clinical actions based on the data provided.
Key Results: The Performance Gap
The study found that while BT-45 performed comparably to graphical visualizations, it significantly lagged behind human-written summaries in facilitating correct clinical decisions.
As shown in the results, human summaries led to a mean decision accuracy score of 0.39, while BT-45 scored 0.34—nearly identical to the 0.33 achieved by graphical displays. Notably, there was a low content overlap (MASI score of 0.21) between the events chosen by the system and those selected by human experts, suggesting that automated "importance" algorithms do not yet fully capture clinical intuition.
Discussion: The Importance of Narrative
The primary deficiency identified in BT-45 was the lack of a coherent narrative. While the system could report abstracted facts, it failed to weave them into a story that helped clinicians build a mental model of the patient's state.
The Continuity Problem
A major issue was "continuity errors." BT-45 might report a dangerous drop in oxygen but fail to mention its subsequent recovery, leading to unnecessary alarm. Humans, by contrast, ensure the "story" of an event is complete, providing closure to clinical observations.
Relational and Contextual Awareness
Human experts naturally describe related data channels in tandem (e.g., comparing central vs. peripheral temperature). BT-45 treated these channels independently, missing the clinical significance of their relationship. Furthermore, humans often include "sensor noise" if it provides context for a physical examination, whereas the system was programmed to filter it out as irrelevant.
Conclusion
The evaluation of BT-45 demonstrates that for NLG to serve as an effective clinical decision-support tool, it must move beyond simple data abstraction. Future systems must employ top-down document planning to create narrative arcs and utilize temporal ontologies to better explain the causal and chronological relationships between events. A summary is not merely a list of facts; it is a discourse that explains why events occurred.
Summary updated