Training Medical QA Models Based on Mixed Rewards from Multiple-Choice and Open-Ended Questions

Yue Qiu, Yujan Ting, Pei Dong, Terrence Chen, Weijing HuangEMNLP 2025 · OpenReview

TL;DR

This paper tackles the challenge of training medical QA models using reinforcement learning (RL) when clear reward signals are scarce, especially for open-ended questions. They propose a novel approach of mixing multiple-choice questions (MCQs) with clear binary rewards and open-ended questions with noisier rewards, demonstrating consistent improvements in medical QA performance across various model sizes.

What Problem Does This Paper Address?

Training large language models (LLMs) with reinforcement learning (RL) typically relies on well-defined reward signals to guide the learning process. However, in specialized domains like medicine, evaluating the correctness of answers to open-ended questions is inherently ambiguous. For instance, determining if an answer like "Start IV dextrose-containing fluids" is acceptable for a question about managing congestive heart failure with low glucose is complex. While human experts can provide accurate assessments, this is not scalable for the dynamic reward needs of RL. This ambiguity creates a significant challenge: how can we effectively train LLMs for medical QA when clear, unambiguous rewards are often unavailable for a substantial portion of the data, particularly for open-ended questions?

Methods

The core of this paper's contribution lies in its mixed-data training strategy. Instead of relying solely on either clear-reward multiple-choice questions (MCQs) or noisy-reward open-ended (OE) questions, the authors propose combining them within the same training batches. The hypothesis is that the MCQs, with their definitive binary rewards (correct/incorrect), act as an anchor, stabilizing the RL training process. Simultaneously, the model benefits from the broader coverage and diverse reasoning patterns learned from OE questions, even if their reward signals are less precise.

For MCQs, they use a straightforward rule-based reward: 1 for a correct answer and 0 for an incorrect one. For OE questions, they explore two types of rewards:

  • Jaccard Similarity: A metric that measures the overlap between the predicted answer and the ground truth answer after tokenization. This provides a softer, continuous reward signal (0 to 1).
  • LLM-based Reward Models: Pre-trained models that assign a score (e.g., 0 to 1) to an answer.

The training itself is performed using the DAPO (Decouple Clip and Dynamic sAmpling Policy Optimization) algorithm, chosen for its effectiveness in policy optimization and its suitability for mixed-reward scenarios. DAPO's dynamic sampling feature is particularly useful as it adaptively curates batches to ensure effective gradients, preventing issues with zero-gradient groups that can arise from mixed-reward data.

Datasets

The authors utilize several medical QA datasets for their experiments:

  • Training Data:
    • English MCQs (MedQA-USMLE)
    • Chinese MCQs (CMB-Exam)
    • Open-ended medical questions (from HuaTuo medical verifiable questions)
    • A mixed dataset combining MCQs and OE questions.
  • Benchmark Datasets for Evaluation:
    • MedQA-USMLE (a 4-choice concise version)
    • CMB-Exam (multiple-choice and multiple-answer questions)
    • MMLU-Pro (health and biology tracks)
    • HealthBench-Small (a subset of doctor-patient dialogues, noted as having a format mismatch with their training data).

Key Insights and Ideas

  • MCQs as RL Stabilizers: The paper strongly suggests that clear, binary rewards from MCQs are crucial for stabilizing the reinforcement learning process, even when combined with noisier rewards from open-ended questions. This is a key takeaway for anyone struggling with RL training in domains with ambiguous feedback.
  • The Power of Mixed Data: Simply combining different types of data with varying reward clarity leads to consistent performance improvements, highlighting that diversity in reward signals can be more beneficial than relying on a single, potentially flawed, reward type.
  • Jaccard Similarity as a Practical OE Reward: While not perfect, Jaccard similarity offers a computationally efficient and effective way to generate reward signals for OE questions, avoiding the need for complex, domain-specific reward model training.
  • Potential for Reward Hacking: The paper points out a critical issue with LLM-based reward models: they can be susceptible to "reward hacking," where models learn to exploit evaluation biases (like length bias) rather than genuinely improving answer quality. This is demonstrated by models achieving high scores by simply repeating the question.

Key Results

The mixed-data, mixed-reward strategy consistently improved medical QA performance across various model scales (3B and 7B parameters).

  • On benchmarks like MedQA-USMLE, CMB-Exam, and MMLU-Pro, models trained with the mixed approach outperformed those trained solely on MCQs or OE questions. For example, Qwen2.5-7B-Instruct showed better performance on the mixed dataset compared to using only MCQs or only OE data.
  • The paper also highlights the potential of newer models like Qwen3-4B, which, despite being smaller, achieved comparable or better results than larger models when trained with their proposed strategy.
  • In an ablation study, training exclusively with MCQs yielded the highest performance on some benchmarks, but the mixed approach still significantly outperformed training on purely open-ended questions.

Limitations and Future Work

  • Computational Constraints: The authors acknowledge that due to time and computational resource limitations, they were unable to experiment with larger or more recent model families, suggesting their findings are primarily limited to the tested model line.
  • Generalizability to Diverse Formats: The performance drop on the HealthBench-Small dataset indicates a limitation in generalizing to diverse medical communication formats (like doctor-patient dialogues) that differ from their training data's direct QA format. This also highlights the challenge of reward modeling for such formats.
  • Reward Noise Characterization: While the paper demonstrates robustness to noisy rewards, it suggests future work could investigate more advanced methods for characterizing and handling reward noise, especially for complex, long-sequence tasks.
  • Scaling and Upper Bounds: Further exploration is needed to understand how the performance gains from the mixed dataset scale with model size and to determine the upper bounds of this approach.

Summary updated

Abstract

Reinforcement learning (RL) for large language models (LLMs) typically requires clear reward signals, which are often unavailable for open-ended (OE) questions where answer evaluation is ambiguous without scalable expert labeling. We investigate whether LLMs benefit from training on mixed data with varying reward clarity. Our approach combines Multiple-choice questions (MCQs), which offer clear binary rewards, with OE questions, for which we use simpler, potentially noisy rewards such as Jaccard similarity or LLM-based evaluators. We hypothesize that MCQs can stabilize training when mixed with OE questions. Our experiments show this mixed-data approach consistently improves medical question-answering performance across model scales.
BibTeX
@inproceedings{qiu2025training,
  author = {Yue Qiu and Yujan Ting and Pei Dong and Terrence Chen and Weijing Huang},
  title = {{Training Medical QA Models Based on Mixed Rewards from Multiple-Choice and Open-Ended Questions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=KU6I3xZ8Us},
}
Figure