A System 1 Decision Model for Medical Multiple-Choice Questions Built on Frozen Language Models

Pavan Kumar ManthaarXiv 2026 · arXiv

This paper explores a "System 1" approach to medical multiple-choice questions (MCQs) by using frozen language models as feature extractors. By training only two small projection heads with a contrastive loss, the author creates a fast, low-compute baseline that achieves up to 46% accuracy on the MedMCQA dataset, highlighting the surprising utility of prompt-engineered frozen embeddings for complex domain-specific tasks.

What Problem Does This Paper Address?

Most state-of-the-art medical question-answering systems rely on "System 2" reasoning—slow, deliberate, generative processes like Chain-of-Thought (CoT) prompting. While accurate, these are computationally expensive.

The author investigates whether a "System 1" model—one that makes fast, intuitive decisions based on vector similarity—can handle the nuances of medical exams. The challenge is that in medical MCQs, the correct answer isn't always the most "similar" to the question in a surface-level semantic sense. This paper aims to establish a cheap, fast baseline for this architecture on a difficult benchmark.

Methods

The core architecture is a two-tower decision model built on top of frozen, decoder-only language models (Qwen and Granite).

  1. Feature Extraction: A question and its options are passed separately through a frozen LLM. Instead of using the raw last-token hidden state, the author uses a "PromptEOL" technique, wrapping the text in a summary prompt (e.g., This question: "[text]" is answered in one word: ") to force the model to compress the meaning into the final token.
  2. Multi-Depth Reading: The model concatenates hidden states from multiple layers (depths) of the LLM rather than just the final layer.
  3. Projection Heads: Two small, trainable MLP "heads" (a State head for questions and an Action head for options) map these high-dimensional vectors into a shared 512-dimensional space.
  4. Training: The heads are trained using a contrastive loss (InfoNCE). The goal is to maximize the cosine similarity between the question vector (s) and the correct answer vector ($a_k$) while minimizing similarity to "negatives" (wrong options and other answers in the batch).
Figure 1: The model. Only the two heads and one scale value are trained; the language model is frozen. All options of a question share one action head, and their vectors can be cached.

Datasets

The study uses MedMCQA, a large-scale dataset of over 190,000 questions from Indian medical entrance exams.

The author performed specific pre-processing to fit the "System 1" paradigm:

  • Filtering: Questions with context-dependent options like "all of the above" or "both A and B" were removed because they cannot be embedded meaningfully in isolation.
  • Splits: The official development set was used as the test set (3,842 questions), and a custom validation set was carved out of the training data.
Table 1: Statistics of the processed dataset. “Negated” is the share of questions containing except, not, false, incorrect or least.

Key Insights and Ideas

  • Prompting is a Force Multiplier: Simply wrapping the input in a "one-word-summary" prompt boosted accuracy by ~10 percentage points compared to using raw text. This suggests that even frozen decoders can be "steered" to produce much better embeddings for downstream tasks.
  • Learning Rate Sensitivity: For these small projection heads, a lower learning rate (0.0003 vs 0.001) was crucial. It provided the single largest gain among hyperparameter tweaks, likely preventing the heads from over-fitting to the training distribution too quickly.
  • Negation is the Achilles' Heel: The model significantly underperforms on questions containing words like "except," "not," or "false." Similarity-based models naturally gravitate toward the most related concepts; they struggle to understand that the task is to find the least related one.
  • Real Reasoning vs. Frequency Bias: By shuffling questions against options (which dropped accuracy to chance), the author proved the model is actually "reading" the question rather than just picking options that were frequently correct during training.

Key Results

The "System 1" approach provides a respectable, albeit non-competitive, baseline:

  • Top Performance: The best configuration (using Qwen3-8B) reached 45.99% ± 0.20% accuracy on the test set.
  • Baselines: This significantly beats random chance (25%) and a "question-blind" baseline (28.1%) that simply picks the most frequent correct answers.
  • Efficiency: Because the LLM is frozen, training the heads takes only a few minutes on a single GPU. Embedding the entire dataset is the only "heavy" lift, and those vectors can be cached.
Figure 2: Test accuracy by configuration. Error bars show the standard deviation over three seeds. “Plain” rows use the initial recipe.

Limitations and Future Work

  • The Performance Gap: While fast, the model is far behind generative giants like Med-PaLM 2 (72.3%). This suggests that "System 1" similarity alone is insufficient for the highest levels of medical expertise.
  • Validation vs. Test: The author noted that gains on the validation set were nearly double the gains seen on the test set, indicating a struggle with distribution shift (specifically, the test set had a much higher proportion of dental questions).
  • Future Directions:
    • LoRA: Allowing the encoder to learn slightly via Low-Rank Adaptation might close the gap.
  • Negation Handling: Explicitly detecting negated questions and flipping the similarity ranking.
  • Light Interaction: Allowing a small amount of cross-attention between the question and option towers before the final decision.

Summary updated

Abstract

A System 1 model answers a multiple-choice question in a single pass. A frozen language model turns the question and each option into vectors, two small, trained heads place them in a shared space, and the closest option wins. The design has been shown on agent decisions, but we are not aware of a study of how far it goes on exam questions that need medical knowledge, where the right option is often no more similar to the question than the wrong ones. We rebuilt the recipe from scratch for MedMCQA and trained only two projection heads and one temperature on top of frozen Qwen3.5-0.8B, Granite-4.2-3B and Qwen3-8B, using a contrastive loss with in-batch and hard negatives and a mask for repeated answers. With plain last-token vectors, both small encoders answer about one question in three correctly (33.1% and 33.0% on 3,842 held-out questions, where chance is 25.0%). Writing each text inside a one-word-summary prompt and lowering the learning rate of the heads raised this to 41.7% and 42.5% for the two small encoders and to 46.0% +-0.2 (three seeds) for Qwen3-8B. Shuffling the questions against the options returns accuracy to chance, so the heads do use the question. Flan-PaLM and Med-PaLM 2 report 57.6% and 72.3% on MedMCQA under a different protocol, so ours is a cheap, fast baseline and not a competitive system. We also say what our evidence cannot show: validation gains were about 1.7 to 2 times the test gains, and we could not separate the effect of reading several encoder layers from the other changes.
BibTeX
@misc{mantha2026system,
  author = {Pavan Kumar Mantha},
  title = {{A System 1 Decision Model for Medical Multiple-Choice Questions Built on Frozen Language Models}},
  year = {2026},
  eprint = {2611.23832},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23832},
}
Figure