Ada-RS: Adaptive Rejection Sampling for Selective Thinking

Yirou Ge, Yixi Li, Alec M. Chiu, Shivani Shekhar, Zijie Pan, Avinash Thangali, Yun-Shiuan Chuang, Chaitanya Kulkarni, Uma Kona, Linsey Pang, Prakhar Mehrotra

TL;DR

Ada-RS (Adaptive Rejection Sampling) is a new framework that teaches Large Language Models (LLMs) "selective thinking"—the ability to skip expensive Chain-of-Thought (CoT) reasoning for simple tasks while reservedly using it for complex ones. By filtering training data through an adaptive, length-penalized reward system, the authors reduced average output tokens by 80% and "thinking" frequency by 95% without sacrificing tool-calling accuracy.

What Problem Does This Paper Address?

Chain-of-Thought (CoT) reasoning has significantly boosted LLM performance on complex tasks, but it comes with a high "token tax." In production environments like customer service bots or e-commerce assistants, generating long reasoning traces for every simple query (e.g., "What is my balance?") is slow and expensive.

The challenge is selective thinking: how do we train a model to internally decide when a query requires deep deliberation and when it can be answered directly? Existing methods often rely on complex multi-stage training or rigid prompt-level controls. This paper tackles the problem at the source: the training signal.

Methods

The authors introduce Adaptive Rejection Sampling (Ada-RS), a lightweight, algorithm-agnostic framework that filters the samples used during fine-tuning.

  1. Adaptive Length Penalty (ALP): The core of Ada-RS is a reward function that balances correctness against verbosity. Crucially, the penalty is adaptive. If a prompt is "easy" (most model samples get it right), the length penalty is high to discourage overthinking. If the prompt is "hard," the penalty shrinks, allowing the model the "budget" it needs to reason through the problem.
  2. Stochastic Rejection Sampling: Instead of just picking the best response, Ada-RS uses rejection sampling to retain a diverse but high-quality set of candidates. It keeps responses that are both correct and efficient while downsampling unnecessarily verbose ones.
  3. Algorithm Agnostic: Ada-RS isn't a new training objective; it's a data filter. The authors demonstrate it working with two popular methods:
  4. Ada-RS-DPO: Pairwise rejection sampling for Direct Preference Optimization.
  5. Ada-RS-DAPO: Group-wise rejection sampling for grouped policy optimization (similar to GRPO).

Datasets

The researchers evaluated their method on a synthetic tool-calling e-commerce benchmark.

  • Scale: 15,000 tool invocations across ~8,000 conversations.
  • Content: Multi-turn, multi-step tasks involving product searches, account retrieval, and transaction look-ups.
  • Base Model: Qwen3-8B with LoRA adapters.
  • Relation to Existing Work: The tools and tasks mirror the $\tau^2$-Bench retail benchmark, designed to simulate real-world e-commerce copilot scenarios.

Key Insights and Ideas

  • DPO Alone Isn't Enough: The authors found that standard preference optimization (DPO) often fails to stop "always-think" behavior. Without specific filtering, models tend to default to their longest, most verbose reasoning patterns even for trivial tasks.
  • The Power of the Filter: The "aha moment" here is that what you show the model during training matters as much as how you optimize it. By simply filtering out "lazy" reasoning on hard tasks and "verbose" reasoning on easy tasks, the model learns the boundary of when to think.
  • Stability via NLL: Rejection sampling can sometimes lead to "forgetting" or language degradation. The authors found that adding an auxiliary Negative Log-Likelihood (NLL) loss term was essential to stabilize learning and maintain overall model quality.

Key Results

Using Ada-RS-DPO and Ada-RS-DAPO, the researchers achieved a significantly better "accuracy-efficiency frontier":

  • Token Reduction: Average output tokens were reduced by 70% to 80%.
  • Thinking Rate: The frequency of the model invoking a <think> block dropped by up to 95% (from nearly 100% in base models to ~6% in optimized versions).
  • Accuracy: Despite the massive reduction in tokens, tool-calling accuracy remained stable or even improved compared to standard Supervised Fine-Tuning (SFT) and DPO baselines.
  • Training Efficiency: Because the framework rejects uninformative samples, it can reduce the number of training samples needed by over 50%, speeding up the training process.

Limitations and Future Work

  • Domain Specificity: The study focused on a single domain (e-commerce) and one model size (8B). It remains to be seen how well these specific penalty weights ($\alpha$) generalize to broader reasoning tasks or much larger models.
  • Metric Granularity: The evaluation focused on per-step tool-call accuracy. The authors acknowledge that future work should look at end-to-end goal success in multi-turn dialogues to ensure that "skipping thinking" doesn't lead to long-term planning errors.
  • Reward Design: While the adaptive penalty is effective, it still relies on a binary "correctness" signal from tool calls. Applying this to open-ended creative writing or subjective tasks would require a more nuanced reward model.

Summary updated

Abstract

Large language models (LLMs) are increasingly being deployed in cost and latency-sensitive settings. While chain-of-thought improves reasoning, it can waste tokens on simple requests. We study selective thinking for tool-using LLMs and introduce Adaptive Rejection Sampling (Ada-RS), an algorithm-agnostic sample filtering framework for learning selective and efficient reasoning. For each given context, Ada-RS scores multiple sampled completions with an adaptive length-penalized reward then applies stochastic rejection sampling to retain only high-reward candidates (or preference pairs) for downstream optimization. We demonstrate how Ada-RS plugs into both preference pair (e.g. DPO) or grouped policy optimization strategies (e.g. DAPO). Using Qwen3-8B with LoRA on a synthetic tool call-oriented e-commerce benchmark, Ada-RS improves the accuracy-efficiency frontier over standard algorithms by reducing average output tokens by up to 80% and reducing thinking rate by up to 95% while maintaining or improving tool call accuracy. These results highlight that training-signal selection is a powerful lever for efficient reasoning in latency-sensitive deployments.
BibTeX
@inproceedings{ge2026ada,
  author = {Yirou Ge and Yixi Li and Alec M. Chiu and Shivani Shekhar and Zijie Pan and Avinash Thangali and Yun-Shiuan Chuang and Chaitanya Kulkarni and Uma Kona and Linsey Pang and Prakhar Mehrotra},
  title = {{Ada-RS: Adaptive Rejection Sampling for Selective Thinking}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.88},
}

Loading related content...

Figure