JET: Just-Enough Exploration Tuning for Efficient Tool-Calling Agents in Repository-Level Question Answering

Yoonho Choi, Taehyeon Han, Seonghyeon LeearXiv 2026 · arXiv

TL;DR

LLM agents often suffer from "over-exploration," continuing to invoke tools and search codebases long after they have gathered sufficient information to answer a user's question. JET (Just-Enough Exploration Tuning) is a novel fine-tuning method that trains agents to recognize this sufficiency, reducing tool calls by up to 33% and significantly lowering inference costs while maintaining or even improving answer quality.

What Problem Does This Paper Address?

In repository-level question answering (QA), agents must navigate massive codebases using tools like semantic search and file viewers. A major bottleneck is over-exploration: agents frequently continue to explore even after the relevant code has been found.

As shown in Figure 1, an agent might discover the correct version identifier in a setup.py file by the third step, yet continue for 26 more unnecessary tool calls. This is problematic for three reasons:

  1. Latency: Every tool call adds time.
  2. Cost: LLMs process the entire conversation history at every step; as the context grows, the cost per step increases superlinearly.
  3. Quality: Excessive context can actually confuse the model, leading to a lower final score than an earlier, "forced" answer.

Standard Supervised Fine-Tuning (SFT) often exacerbates this because it trains agents to strictly imitate "expert" trajectories, which themselves may contain redundant exploration steps.

Figure 1: An example of over-exploration on SWE-QA-Pro (Qwen3-8B). The answer forced after the third tool call (green) scores higher than the final answer after 26 more calls (red).

Methods

JET introduces a three-stage pipeline to bake efficiency directly into the agent's policy:

  1. Golden Step Labeling: To find the optimal stopping point, the researchers force the agent to generate an answer at every step of a training trajectory. An LLM judge scores these intermediate answers. The "golden step" ($i^*$) is defined as the earliest step that achieves the maximum score.
  2. Rationale Insertion: The trajectory is truncated at the golden step. A stronger "rationale generator" model then writes a short explanation (a rationale) describing why the information gathered so far is sufficient to answer the question.
  3. Training with a Halting Weight: The agent is fine-tuned on these truncated trajectories. Crucially, JET uses a halting weight ($\lambda > 1$) to upweight the loss on the rationale tokens. This forces the model to pay more attention to the decision to stop and answer.

This approach differs from prior work by supporting free-form answers (rather than just binary "correct/incorrect" signals) and requiring no additional LLM calls at inference time.

Datasets

The authors evaluated JET using two primary repository-level QA benchmarks:

  • SWE-QA-Pro: 260 questions focused on deep repository understanding.
  • SWE-QA: A larger set of 720 questions.

Training was performed on two types of data: Self-generated trajectories (400 rollouts from the base models) and Expert trajectories (1,000 rollouts generated by Claude 3.5 Sonnet). The experiments utilized two open-weight 8B models: Qwen3-8B and Ministral-3-8B.

Key Insights and Ideas

  • The Score Plateau: Figure 2 illustrates that for both base agents and experts, answer quality reaches a plateau early in the exploration process. Continuing to explore beyond this point provides diminishing returns or even negative utility.
  • SFT is an Efficiency Trap: Standard SFT on expert trajectories successfully improves scores but often increases the number of tool calls because the student model learns to imitate the expert's exhaustive (and sometimes redundant) search patterns.
  • Rationale as a Signal: Simply truncating trajectories isn't enough. The rationale acts as a "thought process" that helps the model internalize the logic of sufficiency, making the "stop" command more robust.
  • Halting Weight is Critical: The authors found that the rationale is only effective when combined with the halting weight. Without upweighting, the "decision to stop" tokens are such a small fraction of the total trajectory that the model tends to ignore them.
JET pipeline: Golden step labeling, Rationale insertion, and Training with a halting weight.

Key Results

JET consistently outperformed both base models and standard SFT across multiple metrics:

  • Reduced Tool Calls: JET reduced tool calls by 11% to 33% on self-generated trajectories and 18% to 28% on expert trajectories compared to SFT.
  • Lower Inference Cost: By stopping earlier, JET slashed costs significantly. For Qwen3-8B on SWE-QA-Pro, the cost per 1,000 questions dropped from $19.2 to $8.5.
  • Performance Preservation: Despite the drastic reduction in exploration, JET largely preserved (and sometimes improved) the task scores. In the "Expert" setting for Ministral-3-8B, JET actually achieved a higher score than standard SFT while using 20% fewer tool calls.
  • Avoiding the Step Limit: As shown in Figure 4, JET agents reach the maximum step limit (30 steps) far less frequently than SFT agents, indicating they are successfully learning to conclude tasks autonomously.

Limitations and Future Work

  • Domain Specificity: The study focused exclusively on Python repositories. Generalization to other programming languages or non-coding agent tasks remains to be seen.
  • Judge Dependency: The "Golden Step" relies on an LLM judge. While common in current research, this introduces potential bias or variance from the judge model.
  • Training Overhead: Labeling the golden step requires generating and scoring multiple answers for every step of every training trajectory, which is a significant one-time offline cost.
  • Scaling: The researchers only tested 8B parameter models; future work could explore if these efficiency gains hold for much larger frontier models.

Summary updated

Abstract

In repository-level question answering (QA), an agent explores a code repository with tools and should answer a question once enough information is gathered. Nevertheless, we find that LLM agents often continue to invoke tools even after the gathered information is already sufficient to answer. We call this phenomenon over-exploration. Moreover, even though existing methods improve task performance through supervised fine-tuning (SFT) on expert trajectories, SFT also propagates the over-exploration in the trajectories to the agent. To mitigate over-exploration, we propose Just-Enough Exploration Tuning (JET), a fine-tuning method that trains the agent to answer once the gathered information is sufficient. JET truncates each training trajectory at the step where the gathered information becomes sufficient, inserts a rationale that explains why the agent can answer at that step, and fine-tunes the agent with a higher loss weight on the rationale. Experiments on two repository-level QA benchmarks with two open-weight 8B models demonstrate that JET reduces tool calls by 11 to 33% on self-generated trajectories while largely preserving the score. On expert trajectories, JET makes 18 to 28% fewer tool calls than SFT while retaining most of the score gain of SFT. In both cases, JET also lowers the inference cost.
BibTeX
@misc{choi2026jet,
  author = {Yoonho Choi and Taehyeon Han and Seonghyeon Lee},
  title = {{JET: Just-Enough Exploration Tuning for Efficient Tool-Calling Agents in Repository-Level Question Answering}},
  year = {2026},
  eprint = {2611.22539},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22539},
}
Figure