TL;DR
PriorityFEC is a novel transport scheme designed for multi-agent LLM systems at the edge, where context must be handed off between nodes over unreliable links. By prioritizing high-utility KV-cache blocks and using a reinforcement-learning-based adaptive streaming FEC, it ensures that the most critical model state arrives before strict deadlines, significantly boosting task accuracy compared to standard protocols.
What Problem Does This Paper Address?
In collaborative multi-agent LLM systems, tasks are often passed from one agent to another. To avoid the massive computational overhead of "prefilling" (re-processing) the entire conversation history at every step, agents can share their Key-Value (KV) cache—the materialized intermediate states of the model.
However, at the network edge, this handoff faces a major hurdle: the Time-to-First-Token (TTFT) deadline. KV caches can be massive, and edge links (like 5G or IoT connections) are often bandwidth-constrained and prone to bursty packet loss. Traditional reliable protocols like TCP or QUIC rely on retransmissions, which often take too long, causing the receiving agent to miss its deadline. Conversely, unreliable protocols like UDP result in missing data that can break the LLM’s reasoning.
The authors argue that we shouldn't treat every byte of the KV cache as equally important. If we can't send it all in time, we should at least send the parts the model needs most.
Methods
PriorityFEC tackles this through a two-pronged approach:
- Utility-Aware Source Ordering: The authors first profile LLMs (Qwen and Llama) to see which parts of the KV cache are most sensitive to data loss. They found that early layers and the very beginning/end of the token sequence are disproportionately important for model fidelity. PriorityFEC maps these "importance scores" to transport blocks and places high-utility blocks at the front of the transmission queue.
- Adaptive Streaming Forward Erasure Correction (FEC): Instead of waiting for retransmissions, FEC sends "repair packets" alongside data. PriorityFEC uses an ACK-sliding Streaming FEC mechanism. Crucially, it employs a Double DQN (Deep Q-Network) scheduler to decide when to send a repair packet versus a new data packet. This RL agent considers the remaining time, the remaining repair budget, and the utility of the blocks currently at risk of being lost.
Datasets
The system was evaluated using two representative open-source LLMs:
- Qwen2.5-7B-Instruct
- Llama-3.1-8B-Instruct
Performance was tested on three complex question-answering workloads from the LongBench-E benchmark:
- 2WikiMultiHopQA
- HotpotQA
- TriviaQA
Network conditions were simulated using both a synthetic Gilbert-Elliott burst-loss model and real-world 5G SA (Standalone) uplink traces.
Key Insights and Ideas
- The "Aha" Moment on Utility: The paper proves that equal amounts of recovered KV data can result in wildly different model accuracy. For example, at 40% data retention, picking the "right" blocks (profile-ranked) yields 55.2% agreement with the original model, compared to only 44% for random selection.
- Opportunity Cost of Repair: In a deadline-constrained environment, every repair packet you send is a transmission slot you didn't use to send a new piece of the KV cache. The RL scheduler learns this trade-off, only "spending" its repair budget when the lost data is high-utility and the deadline is looming.
- Layer-wise Heterogeneity: LLMs aren't monolithic. The sensitivity analysis (Figure 2a) shows that early layers are often much more critical than middle layers, a finding that directly informs the priority queue.
Key Results
PriorityFEC demonstrates its greatest strength under "tight" deadlines (where there isn't enough time to send the full cache even without losses):
- Utility Retention: Under a tight deadline ($\rho = 0.8$), PriorityFEC preserved ~99% of the normalized KV utility, while the best non-priority baseline only managed 75-89%.
- Task Accuracy (F1 Score): On the TriviaQA dataset with Llama-3.1-8B, PriorityFEC achieved a token-level F1 score of 75.2%. In contrast, standard adaptive FEC (FLEC-RT) plummeted to just 16.8% because it didn't prioritize the most important blocks.
- Computational Feasibility: The RL scheduler is extremely lightweight. The decision-making process takes only ~77 microseconds, which is well within the ~112-microsecond window required to keep up with a 100-Mbps link.
Limitations and Future Work
- Scalability: The current profiling method requires constructing full attention tensors, which might become a bottleneck for extremely long contexts (e.g., 32k or 128k tokens).
- Generalization: While the system was tested on 5G traces, the authors note that more extensive testing across different RTTs, bandwidths, and non-stationary network conditions is needed to ensure broad generalization.
- Attention Masking: While the paper mentions using an explicit partial-KV attention mask, the primary results rely on "zero-filling" missing data. Further exploration into how different recovery strategies interact with the transport layer could be fruitful.
Summary updated