TL;DR
KnitTrans is a new system designed to bridge the efficiency gap in Large Language Model (LLM) reinforcement learning by optimizing the "handoff" between data generation (rollout) and model updates (training). By introducing early mini-batch admission and topology-aware data access, it reduces "architectural staleness" by up to 93% and speeds up the transfer-to-training pipeline by up to 5.8x.
What Problem Does This Paper Address?
In LLM Reinforcement Learning (RL), the workflow is a loop: the model generates "trajectories" (rollouts), and then a trainer uses those trajectories to update the model. Modern systems often "disaggregate" these steps, running them on separate pools of GPUs to maximize throughput.
However, this creates two major bottlenecks:
- Architectural Staleness: Trainers usually wait for a full "train-batch" of trajectories to finish before starting. This delay means the model being updated is several versions behind the model that generated the data. This "staleness" can significantly degrade learning quality.
- Data Transfer Inefficiency: Trajectories are massive, especially when they include multimodal data (images/videos) or auxiliary per-token data. Standard systems move data in "sample-level" chunks that don't match how distributed training actually consumes data (across Tensor, Context, and Data Parallel ranks), leading to redundant movement and idle GPUs.
Methods
KnitTrans coordinates the rollout-to-training path through three core designs:
- Constraint-Preserving Mini-Batch Admission: Instead of waiting for a full batch, KnitTrans admits smaller mini-batches as soon as they are ready. To ensure that early admission doesn't violate overall batch requirements (like specific ratios of data sources), it uses feasibility guards. These are mathematical checks that ensure the remaining "slots" in a batch can still satisfy all algorithmic constraints.
- On-Demand Trajectory Data Access: The system stores data as atomic units (independently addressable pieces). It then reorganizes these units on-the-fly to match the specific training topology (e.g., splitting sequences for Context Parallelism). This allows data preparation to overlap with computation, as shown in Figure 1.
- Multimodal Encoding Deduplication: In algorithms like GRPO, one prompt (and its associated image/video) might generate multiple trajectories. KnitTrans identifies these shared inputs and encodes them only once per mini-batch, sharing the results across all relevant trajectories while correctly accumulating gradients for the encoder update.
Datasets
The authors evaluated KnitTrans on two primary RL tasks:
- CodeAgent: A mixture of software engineering datasets including SWE-smith, Multi-SWE-bench, Terminal-Lego, and Scale-SWE.
- Video Grounding: Two in-house multimodal datasets (Grounding A and B) with varying fractions of video data to test different payload sizes.
- Models: Large-scale models ranging from 235B to 671B parameters, including Qwen3-VL, MiMo-V2.5, and DeepSeek-v3.2.
Key Insights and Ideas
- Breaking the Admission Barrier: The "aha moment" here is that you don't need a full batch to start training. By treating the train-batch as a logical boundary and the mini-batch as the execution boundary, you can start training much earlier, which directly reduces the version gap (staleness).
- Topology-Awareness is Essential: Simply moving data faster isn't enough. Because LLM training is highly distributed, the storage system needs to understand how the model is partitioned (TP/CP/DP) to avoid sending the same data to multiple GPUs or sending data a GPU doesn't need.
- Pipelining the Handoff: By treating trajectory transfer as a first-class citizen in the pipeline—rather than just a file copy between stages—KnitTrans can hide almost all transfer latency behind the previous step's computation.
Key Results
- Reduced Staleness: KnitTrans achieved up to 93% lower architectural staleness compared to traditional train-batch admission. This effectively closed the "quality gap" between disaggregated and colocated execution (see Figure 2).
- Massive Latency Reduction: On multimodal workloads, KnitTrans reduced exposed transfer latency by two orders of magnitude (217x–264x).
- End-to-End Speedup: The system achieved a speedup of up to 5.8x in the combined transfer plus training phase over the TransferQueue baseline.
- Constraint Satisfaction: Despite admitting data early, the feasibility guards ensured that 100% of data-source quotas were met.
Limitations and Future Work
While KnitTrans significantly optimizes the transfer path, the authors note that:
- Diminishing Returns: Increasing the number of mini-batches per train-batch beyond a certain point (e.g., 8) yields diminishing returns in staleness reduction.
- Implementation Specifics: The current system is built on top of VeRL and Megatron; while the principles are general, the performance gains are tied to the efficiency of the underlying communication primitives (like Ray's Plasma store and RDMA).
- Future Directions: The authors suggest that future work could explore even tighter integration between the inference engine (rollout) and the training engine to further reduce the overhead of metadata reporting.
Summary updated