KnitTrans: Coordinated Trajectory Transfer for Effective and Efficient LLM Reinforcement Learning

Bangjun Xiao, Chiheng Lou, Xiaodong Ji, Xiangwei Deng, Yikai Zhao, Gang Huang, Hailin ZhangarXiv 2026 · arXiv

TL;DR

KnitTrans is a new system designed to bridge the efficiency gap in Large Language Model (LLM) reinforcement learning by optimizing the "handoff" between data generation (rollout) and model updates (training). By introducing early mini-batch admission and topology-aware data access, it reduces "architectural staleness" by up to 93% and speeds up the transfer-to-training pipeline by up to 5.8x.

What Problem Does This Paper Address?

In LLM Reinforcement Learning (RL), the workflow is a loop: the model generates "trajectories" (rollouts), and then a trainer uses those trajectories to update the model. Modern systems often "disaggregate" these steps, running them on separate pools of GPUs to maximize throughput.

However, this creates two major bottlenecks:

  1. Architectural Staleness: Trainers usually wait for a full "train-batch" of trajectories to finish before starting. This delay means the model being updated is several versions behind the model that generated the data. This "staleness" can significantly degrade learning quality.
  2. Data Transfer Inefficiency: Trajectories are massive, especially when they include multimodal data (images/videos) or auxiliary per-token data. Standard systems move data in "sample-level" chunks that don't match how distributed training actually consumes data (across Tensor, Context, and Data Parallel ranks), leading to redundant movement and idle GPUs.

Methods

KnitTrans coordinates the rollout-to-training path through three core designs:

  • Constraint-Preserving Mini-Batch Admission: Instead of waiting for a full batch, KnitTrans admits smaller mini-batches as soon as they are ready. To ensure that early admission doesn't violate overall batch requirements (like specific ratios of data sources), it uses feasibility guards. These are mathematical checks that ensure the remaining "slots" in a batch can still satisfy all algorithmic constraints.
  • On-Demand Trajectory Data Access: The system stores data as atomic units (independently addressable pieces). It then reorganizes these units on-the-fly to match the specific training topology (e.g., splitting sequences for Context Parallelism). This allows data preparation to overlap with computation, as shown in Figure 1.
  • Multimodal Encoding Deduplication: In algorithms like GRPO, one prompt (and its associated image/video) might generate multiple trajectories. KnitTrans identifies these shared inputs and encodes them only once per mini-batch, sharing the results across all relevant trajectories while correctly accumulating gradients for the encoder update.
Figure 1: Timeline of trajectory admission, data deliv- ery, and training computation under KnitTrans.

Datasets

The authors evaluated KnitTrans on two primary RL tasks:

  • CodeAgent: A mixture of software engineering datasets including SWE-smith, Multi-SWE-bench, Terminal-Lego, and Scale-SWE.
  • Video Grounding: Two in-house multimodal datasets (Grounding A and B) with varying fractions of video data to test different payload sizes.
  • Models: Large-scale models ranging from 235B to 671B parameters, including Qwen3-VL, MiMo-V2.5, and DeepSeek-v3.2.

Key Insights and Ideas

  • Breaking the Admission Barrier: The "aha moment" here is that you don't need a full batch to start training. By treating the train-batch as a logical boundary and the mini-batch as the execution boundary, you can start training much earlier, which directly reduces the version gap (staleness).
  • Topology-Awareness is Essential: Simply moving data faster isn't enough. Because LLM training is highly distributed, the storage system needs to understand how the model is partitioned (TP/CP/DP) to avoid sending the same data to multiple GPUs or sending data a GPU doesn't need.
  • Pipelining the Handoff: By treating trajectory transfer as a first-class citizen in the pipeline—rather than just a file copy between stages—KnitTrans can hide almost all transfer latency behind the previous step's computation.

Key Results

  • Reduced Staleness: KnitTrans achieved up to 93% lower architectural staleness compared to traditional train-batch admission. This effectively closed the "quality gap" between disaggregated and colocated execution (see Figure 2).
  • Massive Latency Reduction: On multimodal workloads, KnitTrans reduced exposed transfer latency by two orders of magnitude (217x–264x).
  • End-to-End Speedup: The system achieved a speedup of up to 5.8x in the combined transfer plus training phase over the TransferQueue baseline.
  • Constraint Satisfaction: Despite admitting data early, the feasibility guards ensured that 100% of data-source quotas were met.
Figure 2: Architectural staleness under partial rollout. (a) Colocated and disaggregated execution with batch-level streaming, where 𝑖denotes the 𝑖-th version of parameters, assuming the durations of the two stages are balanced. (b) Average staleness and avg@𝑛of VeRL (colocated) and slime (disaggregated).

Limitations and Future Work

While KnitTrans significantly optimizes the transfer path, the authors note that:

  • Diminishing Returns: Increasing the number of mini-batches per train-batch beyond a certain point (e.g., 8) yields diminishing returns in staleness reduction.
  • Implementation Specifics: The current system is built on top of VeRL and Megatron; while the principles are general, the performance gains are tied to the efficiency of the underlying communication primitives (like Ray's Plasma store and RDMA).
  • Future Directions: The authors suggest that future work could explore even tighter integration between the inference engine (rollout) and the training engine to further reduce the overhead of metadata reporting.

Summary updated

Abstract

Trajectory transfer from rollout to training affects both learning quality and system efficiency in large language model reinforcement learning (RL). Enforcing batch-wide training requirements can delay the admission of completed trajectories, resulting in higher architectural staleness: the additional parameter-version gap from disaggregated pipelining relative to colocated execution. Beyond admission, misalignment between data access and training consumption causes redundant movement and unnecessary waiting when transferring large trajectory payloads. We present KnitTrans, a system that coordinates the rollout-to-training path through three designs. First, constraint-preserving mini-batch admission allows flexibility in early mini-batch composition, while checking that enough capacity remains to satisfy the remaining quotas. Second, on-demand trajectory data access matches storage units to consumer assignments and overlaps micro-batch data preparation with computation. Third, multimodal encoding deduplication supports reuse across a mini-batch while preserving gradient contributions from all backbone micro-batches. Experiments show up to 93% lower architectural staleness than with train-batch admission, narrowing the learning-quality gap between disaggregated execution and colocated execution. On multimodal workloads, KnitTrans reduces transfer latency by two orders of magnitude and achieves a speedup of up to 5.8 times in transfer plus training over the baseline.
BibTeX
@misc{xiao2026knittrans,
  author = {Bangjun Xiao and Chiheng Lou and Xiaodong Ji and Xiangwei Deng and Yikai Zhao and Gang Huang and Hailin Zhang},
  title = {{KnitTrans: Coordinated Trajectory Transfer for Effective and Efficient LLM Reinforcement Learning}},
  year = {2026},
  eprint = {2610.32389},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.32389},
}
Figure