MINT: Meeting-time Indicators for Truncation in Multi-Step Off-Policy RL

Seungyub Han, Taehyun Cho, Dohyeong Kim, Kyungjae Lee, Jungwoo LeearXiv 2026 · arXiv

In the world of Reinforcement Learning (RL), "multi-step" methods are a double-edged sword. By looking further into the future, they can propagate rewards faster and solve credit assignment problems more efficiently. However, in off-policy settings—where we learn about a target policy using data from a different behavior policy—longer horizons often lead to a "variance explosion."

A new paper from researchers at Seoul National University and Korea University introduces MINT (Meeting-time INdicators for Truncation), a clever way to tame this variance by rethinking how we truncate multi-step traces.

TL;DR

MINT replaces the traditional product of real-valued trace coefficients with a simple binary stopping rule: keep the trace active until the behavior and target trajectories "diverge," then stop. This stochastic truncation adaptively balances long-horizon signals with target stability, significantly reducing sensitivity to the "maximum trace length" hyperparameter.

What Problem Does This Paper Address?

Off-policy multi-step learning usually relies on importance sampling ratios to correct for the mismatch between policies. Methods like Retrace or Peng’s $Q(\lambda)$ multiply these ratios at every step. If you have a long trace, these products can become extremely large (exploding variance) or extremely small (vanishing signal).

To prevent this, researchers typically "cap" the trace at a fixed length (e.g., $n=5$ or $n=10$). However, the optimal cap is highly task-dependent. If the cap is too short, learning is slow; if it's too long, the targets become too noisy to learn from. As shown in Figure 1, standard baselines are often very sensitive to this choice.

Figure 1: Cap-length sensitivity on MuJoCo Hopper (SAC). (a) Bias and variance at 100K and 1M steps for caps {3, 5, 10, 20} (darker shades indicate shorter caps). MINT exhibits a favorable empirical bias–variance balance across the tested caps; several baseline configurations have higher variance at longer caps. (b) Learning curves for the same caps. MINT attains high returns across the tested values, whereas baseline performance varies more with the cap.

Methods

The core idea of MINT is inspired by the concept of coupling in stochastic processes.

  1. The Theoretical Ideal: Imagine two state processes—one following the behavior policy and one following the target policy—starting from the same state. We "couple" them so they stay together as long as possible. MINT defines a binary mask that is $1$ until these two processes first "disagree" (the meeting time) and $0$ thereafter.
  2. The Practical Surrogate: Since we don't usually have a transition model to check if states have coupled, MINT uses a model-free action-level surrogate. At each step $i$, it calculates a "keep probability" based on the policy ratio: $$\hat{p}_i = 1 - \min\left(1, \frac{\pi(a_i | s_i)}{\beta(a_i | s_i)}\right)$$ The algorithm then samples a binary indicator from this probability. If it samples a $0$, the trace is truncated immediately.

  3. Idempotence: Because the masks are binary ($0$ or $1$), they are idempotent ($W^2 = W$). This mathematical property allows the authors to derive much tighter variance bounds than methods using products of real-valued coefficients.

Datasets

The authors evaluated MINT across a massive battery of benchmarks:

  • MuJoCo: Standard continuous control tasks (Hopper, Ant, HalfCheetah, etc.).
  • DeepMind Control (DMC) Suite: 11 tasks ranging from "Easy" to "Medium" difficulty.
  • Atari 100k: A data-efficient version of the classic Atari benchmark (26 games).
  • Real-World RL (RWRL): Tasks specifically designed to test robustness to noise.
  • OGBench: Offline goal-conditioned RL tasks (AntMaze).

Key Insights and Ideas

  • Adaptive Horizons without Tuning: MINT naturally adapts the backup horizon to the encountered policy mismatch. When the policies are similar, the trace stays long; when they disagree, it truncates early.
  • Better Bias-Variance Trade-off: By using a binary indicator, MINT avoids the "compounding noise" of multiplying importance sampling ratios. Figure 2 shows that in a toy "Single-Trap" environment, MINT's variance levels off much more gracefully than Retrace or Peng's $Q(\lambda)$ as the trace length increases.
Figure 2: Single-Trap chain: empirical second moment. (a) Chain MDP. (b) Second moment of the multi-step Bellman residual ˆ∆over trace lengths at γ = 0.9, λ = 1 (log–log axes). Error bars (±1 standard deviation, 30 replicates) are smaller than the markers. At t = 50, Retrace and Peng’s Q(λ) are 1.43× and 2.28× MINT, whose curve levels off over the tested lengths. Setup: Appendix D.2.
  • Model-Free Simplicity: Despite the deep theoretical roots in coupling and mixing times, the implementation is trivial: it just requires a single Bernoulli sample per step based on the policy ratio.

Key Results

  • MuJoCo & DMC: MINT consistently outperformed multi-step baselines. In the 11-task DMC comparison, MINT achieved an Interquartile Mean (IQM) score of 0.457, a 24.7% gain over one-step SAC and a 60.8% gain over $n$-step SAC.
  • Hyperparameter Robustness: In MuJoCo Hopper experiments, MINT achieved high returns across all tested caps ({3, 5, 10, 20}), whereas the performance of $n$-step and Peng's $Q(\lambda)$ fluctuated wildly.
  • Atari 100k: MINT improved the mean Human-Normalized Score (HNS) to 0.512 (compared to 0.465 for the DrQ($\epsilon$) baseline).

Limitations and Future Work

  • Surrogate Gap: The theoretical proofs rely on state-level coupling, but the practical implementation uses an action-level surrogate. While the authors provide a "data-processing inequality" to link the two, there is still a formal gap between the theory and the code.
  • Stochastic Dynamics: In environments with extremely high stochasticity (like the "teleport" mazes in OGBench), the benefits of MINT's truncation can be more nuanced, sometimes being overtaken by Retrace in specific learned-critic settings.
  • Exploration: MINT is an evaluation operator; it doesn't inherently solve the problem of "reward discovery" in sparse environments, though it helps propagate discovered rewards more reliably.

Overall, MINT offers a compelling, theoretically grounded alternative to the "fixed-cap" status quo in off-policy RL, making multi-step methods significantly more robust and easier to deploy.

Summary updated

Abstract

Many multi-step off-policy temporal-difference methods handle policy mismatch through products of per-step trace coefficients, leading to a trade-off between variance control and long-horizon signal. The maximum trace length consequently becomes a sensitive hyperparameter. We propose MINT (Meeting-time INdicators for Truncation), which replaces these products with a binary stopping indicator: one until coupled behavior and target trajectories first meet, zero thereafter. At the true value function, the indicator's idempotence yields a variance bound governed by meeting-time survival. For practical implementation, a single sampled action provides a model-free surrogate for the stopping rule, requiring no transition model or mixing-time estimate. We derive separate sufficient conditions for contraction of the state-coupled operator and its practical surrogate under fixed-policy evaluation. On MuJoCo and DeepMind Control tasks, MINT outperforms the reported multi-step baselines; the main MuJoCo comparison uses a common cap without per-task cap-length tuning. We also demonstrate the effectiveness of MINT on Atari 100k, RWRL, and OGBench experiments; MINT adapts the backup horizon to reduce target variability relative to long fixed-horizon backups across tasks and dynamics.
BibTeX
@misc{han2026mint,
  author = {Seungyub Han and Taehyun Cho and Dohyeong Kim and Kyungjae Lee and Jungwoo Lee},
  title = {{MINT: Meeting-time Indicators for Truncation in Multi-Step Off-Policy RL}},
  year = {2026},
  eprint = {2611.24413},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24413},
}
Figure