Late but Valuable: Why Deadlines Change the Measured Value of LLM Reasoning

Sina Barazandeh, Benjamin E. Lundell, Jose Lugo-MartinezarXiv 2026 · arXiv

This paper challenges the way we evaluate Large Language Model (LLM) reasoning, particularly for real-time applications like automated laboratories. While "thinking" modes in LLMs can significantly improve answer quality, they also increase latency. The authors demonstrate that in systems with hard deadlines, standard metrics that report quality and latency separately can overstate the actual value of reasoning by as much as 2x.

TL;DR

Standard LLM benchmarks fail to account for hard deadlines where a late answer is as useless as a wrong one. This study shows that because the most valuable reasoning often takes the longest, deadlines disproportionately discard high-value answers, making "thinking" modes far less useful in practice than separate quality/latency metrics suggest.

What Problem Does This Paper Address?

As LLMs move from chatbots to autonomous agents in science, they are increasingly operating in environments with hard deadlines. For example, a robotic chemist might need a decision on whether to stop a reaction at exactly 35 seconds; an answer arriving at 36 seconds is worthless, forcing the system to use a "safe fallback" action (like discarding the sample).

The authors identify a "systems problem": researchers typically measure average accuracy and average latency independently. However, in reasoning tasks, these two are often correlated—harder problems require more "thinking" tokens and thus more time. If the most valuable answers are also the slowest, a deadline will kill the very responses that justify the extra cost of reasoning.

Methods

The researchers used a simulation environment featuring two synthetic laboratory tasks: optical-fiber drawing and quantum-dot synthesis. These are "sample-stopping" problems where an agent decides to accept, reject, or continue tuning a sample based on noisy data.

  • Ground Truth: Because the physical models are fully specified, the authors could compute the exact Bayes-optimal decision to serve as a perfect scoring reference.
  • Timing Protocols: They tested two measurement methods: "dispatch timing" (starts at the request) and "evidence-onset timing" (includes client-side prep and pacing).
  • Models: The primary subject was Claude Haiku 5.5 (comparing "fast" vs. "thinking" modes). They also evaluated Claude Sonnet 5.5, Opus, Fable, and a local Qwen3-8B model.
  • Evaluation: They compared "Separate Summaries" (multiplying average on-time rate by average quality) against "Joint Estimates" (scoring only the action that actually executes before the deadline).
Figure 1: Two timing protocols and runtime execution. Evidence-onset timing includes request preparation (counting, pacing and admission), the provider call and polling. Dispatch timing and the simulated process window start together at the provider call, and the simulator is paused until then. Each protocol allows deadline s plus 10 s for removal. Provider-call latency includes service overhead. am is mode m’s answer, afb the fallback and Lm the delay to receipt. The runtime executes valid, on-time answers or the fallback.

Datasets

The study utilized synthetic datasets generated from the physical models of the lab tasks:

  • Decision Quality: 1,200 terminal instances.
  • Deadline Execution: 600 instances per timing clock.
  • Prospective Replay: 1,200 instances to measure prediction error.
  • Serving Load: 36 fixed instances served under concurrent request bursts (8 to 56 requests) on a single NVIDIA A6000 GPU.

Key Insights and Ideas

  • The Value-Latency Correlation: Late answers from Haiku’s thinking mode were worth five times more on average than on-time ones. The model only "thinks" deeply on difficult cases where reasoning actually provides a gain over the fallback.
  • Statistical Overprediction: Using separate averages for quality and latency predicted roughly twice the value that thinking mode actually delivered at a 2.1s deadline. This happens because the product of two averages does not equal the average of their product when value and time vary together.
  • Throughput vs. Utility: In the Qwen3-8B experiment, tripling the total token throughput by increasing concurrent requests actually halved the generation speed per request. This caused the share of thinking answers meeting a 45s deadline to crash from 79.2% to just 1.4%.
  • Fallback Sensitivity: The error in measurement depends on the fallback. With a "safe" fallback (rejecting), metrics overpredict value; with an "unsafe" fallback (accepting by default), they can actually underpredict it.

Key Results

  • Reasoning Quality: Without deadlines, thinking mode reduced Haiku’s decision regret by 79% compared to fast mode.
  • Deadline Impact: At a tight 2.1s deadline, thinking mode's advantage was nearly eliminated. In some scenarios, it delivered less than half the value of the "fast" mode.
  • Prediction Error: The overestimate of thinking’s advantage was measured at 0.0241 per decision (95% CI [0.0165, 0.0317]), confirming that separate metrics are statistically misleading.
  • Model Differences: For Claude Sonnet 5.5, the "fast" mode was already highly effective, resulting in no detectable quality gain from thinking and a negligible deadline effect.
Figure 2: Inference effort and executed outcomes. (a) Exact terminal decision regret on 1,200 synthetic stopping instances for Claude Haiku 5.5. Labels give median/99th-percentile provider-call latency and cost per thousand decisions. (b) Realized utility, low-effort thinking minus fast mode, timed from provider dispatch (line) or from evidence onset (gray squares). The dotted curve is a post hoc, descriptive sweep of fixed 60 s-prompt answers, and deadlines are equally spaced categories. (c) Posterior expected gain over rejection at a 2.1 s deadline on 1,200 new instances answered under 60 s prompts. Gains are averaged within groups of comparable instances and are either predicted from separate summaries (on-time rate times mean answer value) or delivered by on-time answers. Bars and 95% CIs are descriptive. Only the difference in overprediction, D = +0.0241 (thinking minus fast mode), is confirmatory. In (a), the low-minus-fast interval is 98.75%, and the medium-minus-low and high-minus-medium intervals are 97.5%. In (b), points are descriptive, and only the advantage at 60 s minus that at 2.1 s is tested.

Limitations and Future Work

  • Synthetic Tasks: The tasks have exact mathematical solutions. Real-world laboratory planning is more open-ended and may not have such a clear "perfect" reference.
  • Single-Step Decisions: The study focuses on one terminal decision at a time. Future research should explore how these deadline effects compound in multi-step agentic workflows.
  • Adaptive Policies: The authors suggest a need for per-request reasoning policies, where an agent decides how much to "think" based on the remaining time before a deadline.
  • Scheduling: Future work should focus on LLM schedulers that optimize for "goodput"—the value of answers delivered on time—rather than just raw token throughput.

Summary based on "Late but Valuable: Why Deadlines Change the Measured Value of LLM Reasoning" by Barazandeh et al. (2026).

Summary updated

Abstract

Large language models (LLMs) increasingly plan and run experiments in automated laboratories. These experiments keep running while the model reasons and run on schedules with hard deadlines. If an LLM-generated answer arrives late, the system executes a safe fallback action instead, such as rejecting a sample. This creates a systems problem that standard serving metrics do not capture. An answer that matches the fallback adds no value, and answers that differ from the fallback may require more reasoning time. Reporting answer quality and latency separately can therefore overstate the value of reasoning. We test this on synthetic sample decisions, where the best choice is known exactly and a runtime enforces the deadline. We compare one LLM's fast mode, which answers directly, with its thinking mode, which reasons first. In a prespecified replay of 1,200 new decisions, thinking answers that missed a deadline were worth about five times as much as on-time ones. Multiplying the on-time rate by the mean answer value predicted about twice the value that thinking delivered. This prediction overstated thinking's advantage over fast mode by 0.0241 per decision, and by 0.0191 in a repeat on 300 new decisions. Both 95% confidence intervals excluded zero. In a single-GPU Qwen3-8B experiment using simultaneous request bursts, raising the number of requests from 8 to 56 reduced the share of thinking answers that met a 45 s deadline from 79.2% to 1.4%, although total output tripled. Evaluations of LLM agents for laboratories should therefore measure each answer's value and arrival time together and score the action that actually runs.
BibTeX
@misc{barazandeh2026late,
  author = {Sina Barazandeh and Benjamin E. Lundell and Jose Lugo-Martinez},
  title = {{Late but Valuable: Why Deadlines Change the Measured Value of LLM Reasoning}},
  year = {2026},
  eprint = {2611.11910},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11910},
}
Figure