On-Body Visualization of Patient Data for Cooperative Tasks

Dmitri Presnov, Julia Kurz, Judith Willkomm, Daniel Alt, Johannes Dillmann, Robert Zilke, Veit Braun, Cornelius Schubert, Andreas Kolb

Introduction

Electronic Health Records (EHRs) are essential for modern medical documentation, yet their current text-intensive interfaces often hinder rather than help in active clinical settings. In high-pressure environments like neurosurgical wards, the ability to quickly share and interpret a patient's status is vital. This paper introduces a novel approach to address this bottleneck: an anatomically integrated, in-place visualization system that maps abstract EHR data directly onto a 3D virtual patient body.

The Problem: Information Overload in Clinical Workflows

During "cooperative tasks"—such as ward rounds or shift handovers—physicians must rapidly synthesize a patient's neurological status, including pain locations, muscle weakness, and reflex responses. Traditional EHRs require navigating complex menu structures and reading through pages of text, a process that is inefficient and prone to error. The authors argue that since medical education and practice are deeply rooted in anatomy, the human body should serve as the primary spatial map for this information.

System Design and Implementation

The researchers followed a participatory design process with neurosurgeons to establish a framework for "anatomical proxies." The resulting mobile prototype, built using the Vulkan API on Android, features several key innovations:

  • Visual Encoding: Abstract data properties (e.g., symptom intensity) are mapped to visual attributes like color hue, saturation, and texture. For instance, purple is used to denote muscle weakness (paresis), while red indicates radicular pain.
  • Occlusion and Visibility: To ensure internal symptoms are visible, the system uses dynamic transparency. This allows users to see "through" the skin to affected muscles or nerves while maintaining the necessary anatomical context.
  • Anatomical Proxies: For structures too small to be easily seen on a full-body model (such as specific tendons), the system uses projective textures to display proxy icons on the skin surface.
  • Temporal Visualization: A temporal slider enables clinicians to scroll through a patient's history, providing a visual representation of how symptoms have evolved or responded to treatment over time.

Evaluation and Results

The prototype was evaluated by 10 neurosurgeons using seven real-life clinical descriptions of spinal disc herniation. The evaluation focused on three primary areas:

  1. Spontaneous Interpretation: Without access to a legend, surgeons correctly identified muscle weakness 100% of the time and pain 80% of the time, demonstrating the system's high level of intuitiveness.
  2. Diagnostic Matching: Participants successfully matched 3D visualizations to textual case descriptions. While complex cases required the use of zoom and filter tools, the visual approach significantly reduced the cognitive effort needed to identify specific patient statuses.
  3. Trend Analysis: All 10 participants correctly interpreted the patient's healing progress (improvement vs. deterioration) using the temporal slider, often without needing to consult any textual data.

Discussion and Future Work

The study highlights the potential of the "body as an interface" to reduce cognitive load and improve information transfer. However, several limitations were identified. Visualizing very small or adjacent structures on a mobile screen remains challenging, and the current system does not yet integrate non-anatomical data, such as laboratory results (e.g., hemograms). Additionally, while the mobile platform was well-received, surgeons noted the practical difficulty of carrying additional devices during their rounds.

Future research will focus on integrating a wider range of medical data and adapting the visualization to accommodate the varying clinical examination styles of individual physicians. By moving toward an anatomically-anchored data model, clinical interfaces can become more aligned with the natural mental models of medical professionals.

Summary updated

Abstract

Electronic health records (EHR) systematically represent patient data in digital form. However, text and visualization based EHR systems are poorly integrated in the hospital workflow due to their complex and rather non-intuitive access structure. This is especially disadvantageous in clinical cooperative situations that require an efficient, task specific information transfer. In this paper we introduce a novel concept of anatomically integrated in-place visualization designed to engage with cooperative tasks on a neurosurgical ward. Based on the findings of our field studies and the derived design goals, we propose an approach that follows a visual tradition in medicine, which is tightly related with anatomy, by using a virtual patient's body as spatial representation of visually encoded abstract medical data. More specifically, we provide a generic set of formal requirements for these kinds of in-place visualizations, we apply these requirements in order to achieve a specific visualization of neurological symptoms related to the differential diagnosis of spinal disc herniation, and we present a prototypical implementation of the visualization concept on a mobile device. Moreover, we discuss various challenges related to visual encoding and visibility of the body model components. Finally, the prototype is evaluated by 10 neurosurgeons, who assess the validity and the further potential of the proposed approach.
BibTeX
@misc{presnov2019body,
  author = {Dmitri Presnov and Julia Kurz and Judith Willkomm and Daniel Alt and Johannes Dillmann and Robert Zilke and Veit Braun and Cornelius Schubert and Andreas Kolb},
  title = {{On-Body Visualization of Patient Data for Cooperative Tasks}},
  year = {2019},
  eprint = {1906.07720},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.07720},
}

Loading related content...

Figure