This paper explores a "System 1" approach to medical multiple-choice questions (MCQs) by using frozen language models as feature extractors. By training only two small projection heads with a contrastive loss, the author creates a fast, low-compute baseline that achieves up to 46% accuracy on the MedMCQA dataset, highlighting the surprising utility of prompt-engineered frozen embeddings for complex domain-specific tasks.
What Problem Does This Paper Address?
Most state-of-the-art medical question-answering systems rely on "System 2" reasoning—slow, deliberate, generative processes like Chain-of-Thought (CoT) prompting. While accurate, these are computationally expensive.
The author investigates whether a "System 1" model—one that makes fast, intuitive decisions based on vector similarity—can handle the nuances of medical exams. The challenge is that in medical MCQs, the correct answer isn't always the most "similar" to the question in a surface-level semantic sense. This paper aims to establish a cheap, fast baseline for this architecture on a difficult benchmark.
Methods
The core architecture is a two-tower decision model built on top of frozen, decoder-only language models (Qwen and Granite).
- Feature Extraction: A question and its options are passed separately through a frozen LLM. Instead of using the raw last-token hidden state, the author uses a "PromptEOL" technique, wrapping the text in a summary prompt (e.g.,
This question: "[text]" is answered in one word: ") to force the model to compress the meaning into the final token. - Multi-Depth Reading: The model concatenates hidden states from multiple layers (depths) of the LLM rather than just the final layer.
- Projection Heads: Two small, trainable MLP "heads" (a State head for questions and an Action head for options) map these high-dimensional vectors into a shared 512-dimensional space.
- Training: The heads are trained using a contrastive loss (InfoNCE). The goal is to maximize the cosine similarity between the question vector (s) and the correct answer vector ($a_k$) while minimizing similarity to "negatives" (wrong options and other answers in the batch).
Datasets
The study uses MedMCQA, a large-scale dataset of over 190,000 questions from Indian medical entrance exams.
The author performed specific pre-processing to fit the "System 1" paradigm:
- Filtering: Questions with context-dependent options like "all of the above" or "both A and B" were removed because they cannot be embedded meaningfully in isolation.
- Splits: The official development set was used as the test set (3,842 questions), and a custom validation set was carved out of the training data.
Key Insights and Ideas
- Prompting is a Force Multiplier: Simply wrapping the input in a "one-word-summary" prompt boosted accuracy by ~10 percentage points compared to using raw text. This suggests that even frozen decoders can be "steered" to produce much better embeddings for downstream tasks.
- Learning Rate Sensitivity: For these small projection heads, a lower learning rate (0.0003 vs 0.001) was crucial. It provided the single largest gain among hyperparameter tweaks, likely preventing the heads from over-fitting to the training distribution too quickly.
- Negation is the Achilles' Heel: The model significantly underperforms on questions containing words like "except," "not," or "false." Similarity-based models naturally gravitate toward the most related concepts; they struggle to understand that the task is to find the least related one.
- Real Reasoning vs. Frequency Bias: By shuffling questions against options (which dropped accuracy to chance), the author proved the model is actually "reading" the question rather than just picking options that were frequently correct during training.
Key Results
The "System 1" approach provides a respectable, albeit non-competitive, baseline:
- Top Performance: The best configuration (using Qwen3-8B) reached 45.99% ± 0.20% accuracy on the test set.
- Baselines: This significantly beats random chance (25%) and a "question-blind" baseline (28.1%) that simply picks the most frequent correct answers.
- Efficiency: Because the LLM is frozen, training the heads takes only a few minutes on a single GPU. Embedding the entire dataset is the only "heavy" lift, and those vectors can be cached.
Limitations and Future Work
- The Performance Gap: While fast, the model is far behind generative giants like Med-PaLM 2 (72.3%). This suggests that "System 1" similarity alone is insufficient for the highest levels of medical expertise.
- Validation vs. Test: The author noted that gains on the validation set were nearly double the gains seen on the test set, indicating a struggle with distribution shift (specifically, the test set had a much higher proportion of dental questions).
- Future Directions:
- LoRA: Allowing the encoder to learn slightly via Low-Rank Adaptation might close the gap.
- Negation Handling: Explicitly detecting negated questions and flipping the similarity ranking.
- Light Interaction: Allowing a small amount of cross-attention between the question and option towers before the final decision.
Summary updated