Reward Mixology: Crafting Hybrid Signals for Reinforcement Learning Driven In-Context Learning

Changshuo Zhang, Ang Gao, Xiao Zhang, Yong Liu, Deyang Li, Fangchao Liu, Xinyu ZhangEMNLP 2025

TL;DR

This paper introduces R-Mix, a novel reinforcement learning approach for selecting demonstrations to improve in-context learning (ICL) in large language models. By combining outcome-based accuracy rewards with process-oriented signals like influence and label entropy, R-Mix achieves state-of-the-art performance on various NLP tasks, addressing the limitations of previous methods that relied on overly simplistic or sparse rewards.

What Problem Does This Paper Address?

In-context learning (ICL) allows large language models (LLMs) to perform tasks without explicit fine-tuning, simply by providing a few examples (demonstrations) in the prompt. The effectiveness of ICL heavily depends on the quality and order of these demonstrations. While methods exist for selecting demonstrations, they often struggle with:

  1. Oversimplified Reward Signals: Existing iterative selection methods often use limited, single-dimensional metrics (process rewards) to guide the selection process, which don't fully capture the nuances of intermediate steps.
  2. Sparse Outcome Rewards: Methods that aim to directly optimize final task accuracy (outcome rewards) often rely on binary feedback (correct/incorrect) only at the very end of the selection process, making it difficult for the model to learn effectively.

This gap hinders the full potential of LLMs for ICL, as current methods fail to optimally balance immediate progress with the ultimate goal of task performance.

Methods

The core of R-Mix is a reinforcement learning (RL) framework that models demonstration selection as a Markov Decision Process (MDP). Key innovations include:

  • Hybrid Reward System: R-Mix crafts a "hybrid reward" by combining:
  • Outcome Reward (OR): Measures the actual improvement in final task accuracy after the entire demonstration sequence is selected. This is a global, but sparse, reward.
  • Process Rewards (PR): These are immediate, step-wise rewards designed to guide the selection process. They include:
  • Influence Improvement: Measures how much a selected demonstration increases the probability of the correct label for the test input.
  • Label Entropy Improvement: Encourages the model's output distribution to become more uniform, reducing uncertainty.
  • Dual-Head Policy Architecture: To effectively leverage both input and label information from demonstrations, R-Mix uses a dual-head policy. One head processes the input (using BERT embeddings), and the other processes the label (using learnable embeddings). These are then combined to generate compatibility scores, which in turn determine the selection probabilities. This architecture explicitly disentangles the contributions of input relevance and label compatibility.
  • MDP Formulation: The demonstration selection process is formalized as an MDP with states representing the current demonstration sequence and query, actions being the selection of the next demonstration, and transitions defined by adding the selected demonstration.

Datasets

The paper evaluates R-Mix on three standard textual classification datasets:

  • CR (Customer Reviews)
  • MR (Movie Reviews)
  • SST2 (Stanford Sentiment Treebank)

No new datasets were introduced.

Key Insights and Ideas

  • The "Mixology" of Rewards is Crucial: The paper strongly suggests that neither outcome rewards alone nor process rewards alone are sufficient. There's a positive but trade-off relationship between them, meaning a balanced hybrid approach is essential for effective policy optimization.
  • Input Matters More (Generally): Ablation studies show that using only input-based scores or only label-based scores leads to performance degradation. However, relying solely on input scores performs better than relying solely on label scores, which is intuitively explained by the input often containing richer information than the label itself.
  • Demonstration Count Trade-off: While more demonstrations generally improve performance, there's a point (around 8-shots in their experiments) where adding too many can dilute the learning signal and hurt performance, possibly due to noisy or redundant examples.
  • Generalizability Beyond Task-Specific Cues: The label entropy reward is designed to be task-agnostic, contributing to the model's ability to generalize across different datasets.

Key Results

  • State-of-the-Art Performance: R-Mix achieves superior performance compared to various baselines, including learning-free and learning-based methods, on CR (90.62%), MR (90.23%), and SST2 (91.02%).
  • Ablation Studies Validate Components: Removing either the outcome reward or any of the process rewards (influence or label entropy) individually leads to a significant performance drop, underscoring the necessity of the hybrid reward system. Similarly, the dual-head policy is shown to be critical.
  • Efficient Inference: R-Mix achieves inference latency comparable to simpler similarity-based methods, significantly outperforming more complex list-wise methods and being competitive with other learning-based approaches.
  • Cross-Dataset Generalization: R-Mix demonstrates strong generalization capabilities, maintaining competitive performance when trained on one dataset and tested on others.

Limitations and Future Work

The authors acknowledge several limitations:

  • Linear Fusion of Rewards: The hybrid reward system uses a linear combination of components. More sophisticated methods, like attention-based weighting, could potentially improve fusion.
  • Simple Score Aggregation: The dual-head policy directly sums input and label scores. A gating network could offer more adaptive aggregation.
  • On-Policy Training: The current on-policy training samples a subset of the dataset. Off-policy training with mini-batches could improve generalization by leveraging the full training set more effectively.

Future work could explore these areas to further enhance R-Mix's performance and applicability.

Summary updated

BibTeX
@inproceedings{zhang2025reward,
  author = {Changshuo Zhang and Ang Gao and Xiao Zhang and Yong Liu and Deyang Li and Fangchao Liu and Xinyu Zhang},
  title = {{Reward Mixology: Crafting Hybrid Signals for Reinforcement Learning Driven In-Context Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure