TL;DR
ODSA (Orchestrated Discovery and Self-Aggregation) is a framework that puts an LLM "orchestrator" in charge of the test-time scaling process. By dynamically choosing between discovering new solutions, aggregating existing ones, or verifying proposals within a single conversation context, ODSA achieves significant accuracy gains on hard reasoning and coding benchmarks compared to fixed-schedule scaling methods.
What Problem Does This Paper Address?
Language models often perform better if you give them more "thinking time" at inference—a concept known as test-time scaling. However, current approaches are often rigid. Sequential scaling (refining one answer) can get stuck in a local optimum because it never sees alternative approaches. Parallel scaling (sampling many answers and picking one) often generates the correct answer but fails to select it from the pile.
Existing hybrid methods usually follow a fixed schedule (e.g., "always sample 5 times, then aggregate twice"), which wastes compute on easy problems and falls short on hard ones. The authors of ODSA ask: Can we let the LLM itself decide how to spend its computation budget based on the problem at hand?
Methods
ODSA introduces a clean division of labor between an Orchestrator and Stateless Workers.
The Orchestrator manages a single, ongoing conversation. It doesn't try to solve the problem; instead, it issues "action lines" to workers. Crucially, every decision the orchestrator makes is informed by the entire history of the investigation.
The orchestrator has three primary actions at its disposal:
- Discover($d$): Commissions a worker to solve the problem from scratch using a specific directive $d$ (e.g., "try using dynamic programming").
- Compare($s_1, ..., s_k$): Hands a worker several existing candidate solutions and asks it to aggregate them into a better one.
- Accept($s$): Proposes a solution as final. This triggers a Verifier worker to hunt for flaws. If the verifier refutes the answer, its counter-argument becomes a new candidate solution for the orchestrator to consider.
To keep the orchestrator's context window from overflowing on short-answer tasks, the framework uses a Summarizer to distill long reasoning chains into compact, numbered logical steps.
Datasets
The authors evaluated ODSA on two challenging benchmarks:
- Humanity’s Last Exam (HLE): A set of expert-authored questions across various fields where frontier models are still far from perfect.
- LiveCodeBench: A collection of recent coding problems from platforms like LeetCode and Codeforces, ensuring the models haven't seen the solutions during training.
Key Insights and Ideas
- Allocation, Not Solving: The orchestrator's performance improves because it focuses entirely on resource management. By not attempting the problem itself, it avoids the biases or "rabbit holes" that a solver might fall into.
- The "Conversion" Advantage: The most striking finding is that ODSA is exceptionally good at "conversion"—the probability of returning the correct answer given that the correct answer exists somewhere in the pool of candidates. ODSA's conversion rate (80.6%–81.9%) significantly outperformed majority voting and other aggregation methods.
- Context Continuity: Unlike other multi-agent or recursive methods, ODSA issues every action from a single running context. This allows the orchestrator to see exactly where different attempts diverged and which intermediate steps were disputed.
- Adaptive Budgeting: ODSA naturally adapts to problem difficulty. It can stop early if a solution is verified quickly or spend its full budget (and even "refresh" it upon a valid dispute) for complex tasks.
Key Results
ODSA showed consistent improvements across different models (Gemma 4 31B-IT, GPT-OSS-120B, and DeepSeek V4 Flash):
- Accuracy Gains: On HLE, ODSA improved accuracy by 4.7 to 18.0 points over a single-sample baseline. On LiveCodeBench, it added 0.6 to 14.0 points.
- Consistency: While some specialized methods might beat ODSA in a specific category, ODSA was the most consistent "top performer." It was never more than 3.3 points behind the best-performing cost-matched method in any test, whereas baselines like Self-Refine often lagged by 16 points or more.
- Efficiency: Averaged across all tests, ODSA gained 9.0 points over the baseline, outperforming other scaling techniques like GSA (7.4 points) and Best-of-N (2.4 points) at a comparable compute cost.
Limitations and Future Work
- Cost Predictability: Because ODSA is adaptive, it is harder to predict exactly how much a single query will cost in advance compared to fixed-sample methods like majority voting.
- Verifier Bias: The system relies on the verifier being rigorous. A "pessimistic" verifier that refutes everything could waste the budget, though the authors implemented an "acceptance budget" to cap this behavior.
- Generative Headroom: The authors note that since ODSA already "converts" most correct answers in its pool, future improvements should focus on the "generative side"—widening the discovery process to ensure the correct answer enters the pool in the first place.
Summary updated