VITAL: Visual-Semantic Dual Supervision for Enhanced and Interpretable Latent Reasoning in Medical MLLMs

Qiaoru Li, Shaotian Liang, Jintao Chen, Haoran Sun, Yuxiang Cai, Jianwei Yin, Yankai JiangarXiv 2026 · arXiv

TL;DR

VITAL introduces a latent-space reasoning framework for medical Multimodal Large Language Models (MLLMs) that utilizes visual-semantic dual supervision during training. By forcing continuous hidden states to encode both textual logic and high-resolution visual evidence, VITAL eliminates the massive inference latency and language bottlenecks of explicit Chain-of-Thought (CoT) reasoning. The framework achieves state-of-the-art performance competitive with trillion-parameter proprietary models while offering post-hoc visual and textual interpretability at zero additional inference cost.

What Problem Does This Paper Address?

Medical Visual Question Answering (VQA) demands highly precise, fine-grained visual understanding. While explicit Chain-of-Thought (CoT) prompting helps models decompose complex diagnoses into structured steps, it suffers from two major flaws in clinical settings:

  1. The Language Bottleneck: Forcing a model to compress rich visual features (like subtle lesion textures or boundary gradients) into discrete text tokens inevitably discards critical perceptual details.
  2. Inference Overhead: Generating verbose text reasoning chains significantly increases latency and computational cost, which is highly impractical for time-sensitive clinical environments.

"Latent reasoning"—where a model processes continuous hidden states instead of text tokens—offers a faster alternative. However, existing latent reasoning methods suffer from modality collapse (where hidden states drift into text-only representations), self-referential visual anchors, and a complete lack of interpretability. Clinicians cannot trust a black-box model that provides no explanation for its diagnostic path.

Figure 1: Comparison of reasoning paradigms.

Methods

The authors propose VITAL (Visual-Implicit Thinking with Aligned Latents), which implements a recurrent latent reasoning loop with strict train-inference parity. To prevent modality collapse and ensure clinical transparency, VITAL introduces Visual-Semantic Dual Supervision using temporary training-time "scaffolding":

  • Semantic Supervision: An auxiliary text decoder (a lightweight, independent language model) is attached to the latent states to reconstruct human-readable reasoning chains step-by-step.
  • Visual Supervision: A visual projector maps the latent states into the feature space of a frozen, independent expert medical vision encoder (MedSigLIP), regressing Region of Interest (ROI) features.
Figure 2: Overview of VITAL. The multimodal backbone encodes the input into a prefix KV-cache. A recurrent latent loop iterates K steps (zk = fθ(zk−1; C)) with identical paths at training and inference. Latent states are supervised by Ltask (answer), Ltext (auxiliary text decoder), and Lvisual (visual projector regressing ROI features). Both auxiliary modules are discarded at inference with zero overhead.

Crucially, both the text decoder and visual projector are entirely discarded at inference time, resulting in zero extra latency, memory, or parameter overhead. However, they can be re-attached post-hoc to act as a "textual window" (showing what the model reasons) and a "visual window" (showing where the model looks), providing dual interpretability.

To stabilize the optimization of these continuous chains, the authors utilize a three-phase curriculum learning strategy that progressively scales the reasoning depth from $K=1$ to $K=4$ steps.

Datasets

The authors constructed a large-scale dataset of ~61K five-tuple training samples (comprising an image, question, answer, $K$-step reasoning chain, and ROI visual feature). The dataset was automatically compiled from two public medical segmentation sources:

  • Medical Segmentation Decathlon (MSD)
  • BiomedParse

The dataset spans 9 imaging modalities (including CT, MRI, X-ray, ultrasound, and pathology). For evaluation, VITAL was tested across 7 distinct benchmarks: the MSD and BiomedParse test splits, three public medical VQA datasets (VQA-RAD, PathVQA, PMC-VQA), a visual grounding benchmark (GEMeX-RMCoT), and a private, held-out in-house neuroimaging dataset of 1,000 VQA pairs to ensure robust generalization.

Key Insights and Ideas

  • Naïve Latent Reasoning Can Degrade Performance: The authors discovered that unconstrained latent reasoning baselines actually perform worse than the zero-shot base model on medical tasks. Without explicit dual supervision, latent states rapidly collapse into text-dominant subspaces, losing all spatial grounding.
  • Independent Expert Anchors Matter: Aligning latent states to an independent, high-resolution medical vision encoder (MedSigLIP) yields massive accuracy gains compared to self-referential alignment (using the MLLM's own native vision tokens). Spatial resolution is a critical bottleneck for latent visual grounding.
  • The Power of Discardable Scaffolding: VITAL proves that interpretability and efficiency are not mutually exclusive. By treating explanation generation as a training constraint rather than an operational requirement, models can remain lean at runtime while retaining auditability.
  • Coarse-to-Fine Convergence: Qualitative analysis reveals that over successive latent steps ($z_1 \rightarrow z_4$), the visual projector's attention heatmaps naturally sharpen from broad anatomical regions down to tight, precise focuses on specific pathologies.

Key Results

  • Massive Speedups over Explicit CoT: VITAL achieves a ~97× inference speedup compared to explicit CoT models (353ms vs. 34.1s) while delivering substantially higher accuracy.
  • State-of-the-Art Performance: VITAL achieved an absolute accuracy improvement over its base model (Qwen3-VL-8B) of +47.52% on MSD and +46.67% on BiomedParse.
  • Superior Visual Grounding: On the GEMeX-RMCoT visual grounding benchmark, VITAL scored 86.67%, outperforming trillion-parameter proprietary models like Claude-Opus-4.6 by an absolute margin of 28.34%.
  • Strong Generalization: On the held-out, private in-house neuroimaging testset, VITAL achieved 80.50% accuracy, comfortably beating GPT-5.4 (73.80%).
In-domain accuracy vs. latency, showcasing VITAL's Pareto efficiency.

Limitations and Future Work

  • Fixed Reasoning Depth: VITAL currently enforces a fixed maximum of $K=4$ latent steps for every question. The authors note that implementing an adaptive early-exit mechanism (allowing the model to stop "thinking" early for simple questions) is a vital next step.
  • Dependence on Teacher Quality: The textual reasoning chains used during training are distilled from a proprietary MLLM teacher, meaning VITAL's reasoning ceiling is fundamentally bounded by the teacher's medical accuracy.
  • Conciseness vs. Verbosity Metrics: Because latent reasoning prioritizes concise correctness, VITAL scores lower on token-overlap metrics (like Token-F1) in open-ended generation setups that inherently reward verbose text outputs.

Summary updated

Abstract

Latent reasoning enables reasoning over continuous hidden states rather than explicit tokens, avoiding the language bottleneck and inference overhead of chain-of-thought for medical VQA. However, existing methods suffer from modality collapse, insufficient visual supervision, and train-inference mismatch. Moreover, their opaque latent states offer no interpretability, which is critical in clinical applications. We propose VITAL, a latent-space reasoning framework for medical MLLMs with visual-semantic dual supervision: an auxiliary text decoder reconstructs reasoning chains from latent states, while a visual projector regresses ROI features from a frozen, independent medical vision encoder. Both modules are discarded at inference with zero overhead, yet can be re-attached post-hoc for dual interpretability, providing textual and visual explanations of the reasoning process without sacrificing efficiency. We construct a 61K dataset spanning 9 imaging modalities, exceeding prior medical visual latent reasoning datasets by an order of magnitude. Experiments on 7 benchmarks show that VITAL consistently and substantially outperforms the backbone, all latent reasoning baselines, and medical MLLMs trained on far larger data, achieving state-of-the-art results competitive with trillion-parameter proprietary models.
BibTeX
@misc{li2026vital,
  author = {Qiaoru Li and Shaotian Liang and Jintao Chen and Haoran Sun and Yuxiang Cai and Jianwei Yin and Yankai Jiang},
  title = {{VITAL: Visual-Semantic Dual Supervision for Enhanced and Interpretable Latent Reasoning in Medical MLLMs}},
  year = {2026},
  eprint = {2605.28422},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.28422},
}
Figure