TL;DR
MedJev is a "System One" vision-language model that replaces slow, token-by-token text generation with a direct decision-making head for medical image understanding. By truncating the language backbone and using a novel residual decision head, it achieves a 7.93% absolute accuracy gain and a 6.04× speedup over traditional generative models like Qwen3.5-4B.
What Problem Does This Paper Address?
Most modern Vision-Language Models (VLMs) operate like Daniel Kahneman’s "System Two": they are slow, deliberate, and "think" by generating long strings of text. While this is great for general chat, it’s often a bottleneck in medical diagnostics.
When a VLM is asked to classify a medical image or answer a multiple-choice question, it typically generates a reasoning trace before providing an answer. This autoregressive process is computationally expensive—latency scales with the length of the response. Furthermore, these models often fail to follow formatting constraints, making it difficult to parse a clean answer from free-form text. Most importantly, the authors note that for medical tasks, long reasoning traces often don't help if the initial visual grounding is weak; in fact, they can introduce new errors.
Methods
The authors introduce MedJev, a model designed for "System One" thinking—fast, intuitive, and direct. Instead of generating tokens, MedJev maps images, questions, and candidate answers to a decision in a single multimodal forward pass.
Key architectural innovations include:
- Language-Side Depth Reduction: The researchers discovered that the deepest layers of a language model are often redundant for discriminative tasks. MedJev keeps the full vision pathway but removes the top language layers (e.g., keeping only 23 out of 32 layers in a 4B model). This significantly reduces the computational footprint without sacrificing visual detail.
- Residual Decision Head: Instead of a vocabulary projection layer, MedJev uses a specialized head that scores candidate answers. It reuses pretrained Query-Key (QK) matching logic from the backbone to see how strongly a "decision token" attends to "candidate marker tokens." To handle complex medical nuances, they add a lightweight, zero-initialized Multi-Layer Perceptron (MLP) as a residual to refine these scores.
- Single-Pass Inference: By appending marker tokens to each candidate answer in the prompt, the model evaluates all options simultaneously. There is no decoding loop, no KV cache management, and no need for a text parser.
Datasets
MedJev was trained and evaluated on a diverse mixture of four medical benchmarks:
- HAM10000: Dermoscopic images for skin lesion classification (7 categories).
- PneumoniaMNIST: Pediatric chest X-rays for pneumonia detection.
- MedThinkVQA: A challenging benchmark involving clinical cases with 2 to 47 images per case and multiple-choice diagnoses.
- PMC-VQA: Visual question answering based on figures from biomedical literature.
Key Insights and Ideas
- The "Aha" Moment on Depth: A "depth probe" analysis (Figure 3) revealed that the model's ability to discriminate between answers plateaus well before the final layer. This justifies cutting the "head" off the language model to save time.
- Visual Tokens are King: Unlike other efficiency methods that prune visual tokens (which might delete a tiny but critical lesion), MedJev keeps every single visual token and only prunes the language-side computation.
- Pretrained Matching is a Strong Prior: The attention mechanism in pretrained LLMs is already a world-class "matcher." By framing multiple-choice as an attention task between a question and candidates, MedJev leverages years of pretraining more effectively than free-form generation does.
Key Results
MedJev-4B was compared against 20 cutting-edge models in the 4B–9B parameter range.
- Accuracy: It achieved a state-of-the-art average accuracy of 69.06%, outperforming its base model (Qwen3.5-4B) by nearly 8%.
- Speed: The end-to-end speedup is dramatic, averaging 6.04×. On the multi-image MedThinkVQA benchmark, where generative models get bogged down in long responses, MedJev was 7.35× faster.
- Reliability: Because it outputs a probability distribution over a fixed set of candidates, it never suffers from "parsing failures" where a model provides an answer that isn't one of the options.
Limitations and Future Work
The primary limitation of MedJev is its focus on closed-set tasks. Because it relies on scoring predefined candidates, it cannot currently be used for open-ended medical reporting or free-form dialogue.
The authors suggest that future work will involve extending this "System One" approach to more general medical scenarios and complex visual perception tasks that go beyond simple classification, potentially bridging the gap between fast intuition and slow, detailed reasoning.
Summary updated