HealthLens: A Natural Language Querying System for Interactive Visualization of Electronic Health Records

Haodi Zhang, Siqi Ning, Qiyong Zheng, Yuanfeng Song, Liang-Jie ZhangIJCAI 2025 · DOI

TL;DR

This paper introduces HealthLens, a novel system that allows healthcare professionals to generate complex visualizations of Electronic Health Records (EMRs) using natural language queries. By leveraging a specialized language model (MedCodeT5) and a bilevel optimization approach, HealthLens eliminates the need for users to have prior knowledge of declarative visualization languages, making EMR data analysis more accessible.

What Problem Does This Paper Address?

Extracting actionable insights from Electronic Health Records (EMRs) is crucial for modern healthcare, but it's hindered by the complexity of both structured and unstructured data. While data visualization is a powerful tool for understanding this data, creating effective visualizations requires significant technical expertise. This includes understanding domain-specific data, mastering declarative visualization languages (DVLs) like Vega-Lite, and knowing how to transform data and choose appropriate visual encodings. This steep learning curve creates a barrier for many healthcare professionals, limiting their ability to leverage visualization for clinical decision-making. Existing solutions often focus on niche applications or still require manual coding, failing to address the broader need for accessible, interactive EMR visualization.

Methods

HealthLens employs a two-stage approach to translate natural language queries into visualizations:

  • Stage 1: MedCodeT5 for DVL Generation: The core of HealthLens is MedCodeT5, a language model specifically fine-tuned on medical coding schemas. This model takes a natural language query (e.g., "Show how blood glucose levels relate to insulin doses over the past year") and generates a corresponding Declarative Visualization Language (DVL) specification. This is a key innovation, as it bridges the gap between user intent and the technical requirements of visualization tools.
  • Stage 2: Bilevel Optimization with LLM for Refinement: Once a DVL specification is generated, HealthLens attempts to render the visualization. If errors occur during this process (e.g., invalid field references, incorrect chart types), an advanced Large Language Model (LLM) like GPT-4 is employed. This LLM analyzes the error information and iteratively corrects the DVL specification. Beyond error correction, the LLM also handles chart optimization, enhancing the visualization with elements like raster graphics, color, and tooltips, and can even generate textual explanations of the chart. This bilevel optimization ensures robust and visually appealing outputs.

Datasets

  • MedicalVis Dataset: This dataset is central to MedCodeT5's capabilities. It comprises 35,374 instances covering various visualization types (bar charts, pie charts, line charts, scatter plots) and is designed for medical text-to-visualization tasks. The dataset was created using a pipeline that leverages LLMs to generate high-quality DVLs from structured SQL queries. This dataset allows MedCodeT5 to learn medical data semantics, query structures, and visualization aesthetics specific to EMRs.

Key Insights and Ideas

  • Democratizing EMR Visualization: The most significant insight is that sophisticated EMR data visualization can be made accessible to non-technical users. By abstracting away the complexities of DVLs, HealthLens empowers healthcare professionals to focus on data interpretation rather than technical implementation.
  • Domain-Specific LLMs are Crucial: The success of MedCodeT5 highlights the importance of domain-specific language models for specialized tasks like EMR analysis. Fine-tuning on medical coding schemas allows the model to understand the nuances of medical data and queries more effectively than general-purpose models.
  • Iterative Refinement is Key: The bilevel optimization approach, where an LLM corrects and enhances DVLs, is critical for achieving accurate and aesthetically pleasing visualizations. This iterative process handles the inherent complexities and potential errors in both user queries and initial model outputs.

Key Results

The paper demonstrates the feasibility of creating sophisticated EMR visualizations with minimal technical expertise. While specific quantitative results like accuracy percentages are not detailed in the provided text, the demonstration scenario showcases:

  • Successful Natural Language to Visualization Translation: Users can input plain language queries and receive interactive visualizations within seconds.
  • Interactive Refinement: Users can further refine visualizations with follow-up natural language queries, demonstrating dynamic data exploration.
  • Error Handling and Correction: The system effectively handles initial errors and provides corrected, improved visualizations.

Limitations and Future Work

  • Limitations: The paper mentions that the auto-refinement and correction mechanism is currently implemented using GPT-4 and prompt engineering. The full potential of the bilevel optimization for chart optimization (e.g., applying raster graphics, adding color, inserting textual explanations) is still being explored.
  • Future Work: The authors suggest that HealthLens has the potential to significantly reduce the technical barrier to advanced EMR analysis, enabling healthcare professionals to focus on clinical reasoning. Future work would likely involve further enhancing the LLM's capabilities in chart optimization and explanation generation, and potentially expanding the range of visualization types and EMR data complexities that HealthLens can handle.

Summary updated

BibTeX
@inproceedings{zhang2025healthlens,
  author = {Haodi Zhang and Siqi Ning and Qiyong Zheng and Yuanfeng Song and Liang-Jie Zhang},
  title = {{HealthLens: A Natural Language Querying System for Interactive Visualization of Electronic Health Records}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/1276},
}
Figure