A Visual Information Extraction System Using Multimodal Large Language Models for Scanned Cognitive and Behavioral Assessments

Enshuo Hsu, Linh Nguyen, Sunyang Fu, Heath Goodrum, Sandra Stansberry, Kirk Roberts, Hongfang Liu, Yen-Chi L. LearXiv 2026 · arXiv

This study presents a significant step forward in clinical data science, moving from manual transcription to automated, multimodal extraction of "trapped" data in handwritten medical forms.

TL;DR

Researchers developed a modular system using multimodal Large Language Models (LLMs) to extract structured data from scanned, handwritten cognitive and behavioral assessments. By implementing a "human-in-the-loop" dual-model validation design, they achieved 97.5% accuracy while reducing manual labor by nearly two-thirds and cutting costs by over 12x.

What Problem Does This Paper Address?

In modern healthcare, a staggering amount of critical patient data is effectively "invisible" to automated systems. While Electronic Health Records (EHRs) are standard, many specialized assessments—like the Mini-Mental State Examination (MMSE) for dementia or the Vanderbilt scale for ADHD—are still completed on paper.

These forms are then scanned into the EHR as images. Because they contain messy handwriting, checkboxes, and even hand-drawn diagrams (like the Mini-Cog clock-drawing task), traditional Optical Character Recognition (OCR) often fails. Currently, retrieving this data for research or clinical decision support requires expensive, slow, and unscalable manual review. The authors aim to bridge this gap using the visual reasoning capabilities of modern multimodal LLMs.

Methods

The authors proposed a modular, two-layer architecture:

  1. Form Classifier: A scanned page is first sent to an LLM to identify the form type (e.g., is this an MMSE or a MoCA?).
  2. Score Extractor: Once classified, the image is passed to a specialized "extractor" module tailored to that specific form's layout. This module prompts the LLM to return structured data (JSON) based on a predefined schema.
The system uses a modular flowchart to classify forms before extracting specific fields.

A key innovation is the Human-in-the-loop (HITL) design. To ensure clinical-grade reliability, the system uses two different models (GPT-4.1-mini and GPT-5.4-mini). If both models agree on the extracted scores, the result is accepted automatically. If they disagree, the page is flagged for human review. This "dual-model validation" focuses human effort only on the most ambiguous cases.

Datasets

The study utilized a real-world dataset of 6,699 scanned document files from the EHRs of UT Physicians.

  • Cognitive Assessment Cohort: 3,669 pages (MMSE, MoCA, Mini-Cog).
  • ADHD Assessment Cohort: 3,030 pages (NICHQ Vanderbilt scales).
  • Ground Truth: Three research assistants manually annotated 1,000 pages to create a gold-standard benchmark for evaluation.

Key Insights and Ideas

  • Spatial Reasoning over Raw OCR: Unlike traditional OCR that tries to turn pixels into text first, multimodal LLMs "look" at the form holistically. This allows them to interpret checkboxes and hand-drawn clocks with surprising sophistication.
  • The "Agreement" Proxy: The researchers found that model agreement is a highly reliable proxy for accuracy. By only reviewing "disagreements," they maintained near-perfect accuracy while automating the vast majority of the work.
  • Handling Ambiguity: In the Mini-Cog clock-drawing task, 62.3% of errors were due to genuinely ambiguous drawings (e.g., clock arms of equal length). Interestingly, the LLMs often provided "reasonable" interpretations that, while different from the human annotator, were clinically plausible.
Examples of errors, including rotated scans and ambiguous marks that challenge the LLM.

Key Results

The system's performance represents a massive leap over traditional methods:

  • Accuracy: End-to-end accuracy for cognitive assessments reached 97.5% with the HITL design. For the Vanderbilt ADHD forms, accuracy was naturally high at 97.5% even without human intervention.
  • Efficiency: The system is 10 to 20 times faster than manual data entry.
  • Cost: Processing costs were reduced by over 12 times compared to human transcriptionists.
  • Labor Reduction: The dual-model validation reduced the manual workload by 63.8%, allowing staff to focus only on the most difficult-to-read scans.
Quantitative performance comparison showing LLMs dramatically outperforming the Tesseract OCR baseline.

Limitations and Future Work

While promising, the authors note several areas for further development:

  • Model Dependency: The study relied on proprietary models (GPT-4/5 variants via Azure). Future work should evaluate open-weight models to increase accessibility and privacy control.
  • Layout Variations: The system was developed using static versions of forms. In practice, hospitals often use slightly different versions of the same assessment, which could impact extraction logic.
  • Generalizability: As a single-organization study, the results reflect the specific scanning artifacts and handwriting styles of one health system. Testing across diverse clinical environments is a necessary next step.

Ultimately, this work demonstrates that multimodal LLMs are no longer just "chatbots"—they are powerful visual engines capable of digitizing the "dark data" of the medical world.

Summary updated

Abstract

Objective. Paper-based assessments are common in patient care, but handwriting and scanning artifacts make these forms difficult to process at scale. This study proposes a modular, multimodal LLM-based system with human oversight for extracting fields and scores. We conducted a qualitative analysis of the use of LLMs for hand-drawing analysis. Materials and Methods. The system processes cognitive assessments (i.e., MMSE, MoCA, Mini-Cog) and the NICHQ Vanderbilt Assessment using two modules: 1) a form classifier that assigns a scanned page to a form type, followed by a corresponding 2) score extractor that extracts fields and scores into a schema. A human-in-the-loop design utilizing dual-model validation engages human annotators when two LLMs disagree, limiting manual review to challenging pages only. Results. End-to-end accuracy for cognitive assessments was 88%, with most form types exceeding 90%. For the NICHQ Vanderbilt, overall accuracy reached 97.5%, with all form types above 96%. The human-in-the-loop design further improved cognitive assessment accuracy to 97.5% while reducing manual work by 63.8%. In the Mini-Cog Clock Drawing task, qualitative analysis showed that 62.3% of all extraction errors stemmed from ambiguous drawing, in which cases the LLM provided reasonable readings. Discussion. Compared to current manual data extraction, our LLM-based system reduces processing time by 10 to 20 times and costs by over 12 times. The modular design allows seamless expansion to additional form types. Conclusion. Our system provides an accurate, scalable, and cost-efficient solution for structuring data trapped in handwritten paper forms for clinical decision support.
BibTeX
@misc{hsu2026visual,
  author = {Enshuo Hsu and Linh Nguyen and Sunyang Fu and Heath Goodrum and Sandra Stansberry and Kirk Roberts and Hongfang Liu and Yen-Chi L. Le},
  title = {{A Visual Information Extraction System Using Multimodal Large Language Models for Scanned Cognitive and Behavioral Assessments}},
  year = {2026},
  eprint = {2611.22252},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22252},
}
Figure