Coordinated Networking for On-Device Agent-Augmented Real-Time Communication

Goodsol Lee, Juheon Yi, Jinglu Wang, Haowen Xu, Saewoong Bahk, Yan LuarXiv 2026 · arXiv

TL;DR

On-device AI agents in real-time communication (RTC) apps introduce heavy network traffic when sharing large context files (like LLM KV caches), severely degrading live video quality. This paper presents HAFS (Human-Agent Flow Scheduling), an app-layer framework that co-optimizes human video and agent context flows. HAFS achieves a 1.5× improvement in video quality while slashing agent response times by 31%.

What Problem Does This Paper Address?

Video conferencing platforms are evolving to include on-device AI agents that help users collaborate in real time—for instance, by automatically cross-checking documents or drafting text during a live call. To protect user privacy and avoid massive cloud GPU costs, running these small language models locally on user devices is highly desirable.

However, edge devices struggle with the computationally heavy "prefill" stage of LLM inference. To keep response times low, it is much faster for a sender's agent to pre-compute and stream the raw context as a Key-Value (KV) cache to the receiver. The catch? Even after compression, these KV caches are bulky (often tens of megabytes), leading to severe network contention with the delay-sensitive live video stream.

Figure 1: On-device agent-augmented RTC scenario and its network contention problem.

Current commercial platforms handle this poorly: Zoom allows file transfers to aggressively crowd out video traffic, causing severe video degradation, while Microsoft Teams heavily throttles data transfers, resulting in underutilized bandwidth and sluggish agent performance.

Methods

The authors design HAFS (Human-Agent Flow Scheduling), a coordinated networking framework built directly into the application layer atop WebRTC and llama.cpp. HAFS features two primary components:

  1. Human-Agent Flow Coordinator: This module dynamically manages the aggregate sending rate. It introduces a Dual Feedback-based Frame Queuing Estimator that pairs standard video transport feedback (RTP) with dense data channel acknowledgments (SCTP) to measure network queuing at a fine, frame-level granularity. An MI/AI-MD (Multiplicative/Additive Increase, Multiplicative Decrease) rate controller then safely pushes agent throughput to the limit without violating the video's frame-delay deadlines.
  2. Multi-Agent Flow Coordinator: When a query requires pulling multiple contexts simultaneously, this module schedules their transmission order. By framing the challenge as a classic two-machine flow shop problem and applying Johnson's rule, it pipelines network transfers and local GPU decoding. To schedule effectively before decoding occurs, it predicts the expected response length using a lightweight, single-step LLM decoding trick.

Datasets

The framework was evaluated using two standard NLP benchmarks adapted for on-device agent tasks:

  • MuSiQue: A multi-document question-answering dataset used to evaluate short-context tasks (<4k tokens).
  • QMSum: A meeting summarization dataset used to evaluate long-context workloads (8k–16k tokens).

Key Insights and Ideas

  • The Prefill Bottleneck is Real: On edge hardware like phones or edge embedded modules, the LLM prefill phase consumes 71% to 92% of the total response time. Shifting the burden to network transmission via KV cache streaming eliminates this massive compute bottleneck.
  • Leveraging the SFU Advantage: Modern video conferencing relies on Selective Forwarding Units (SFUs), which effectively split long end-to-end paths into short sender-SFU and SFU-receiver links. HAFS capitalizes on these low-RTT links to achieve precise, reactive app-layer rate control that normally requires complex in-network router scheduling.
  • Relative Ordering Trumps Absolute Accuracy: Pipelining network and compute tasks optimally doesn't require knowing the exact token length of an LLM's response. Because Johnson's rule relies purely on the relative order of workloads, the system remains highly robust to prediction errors.
Figure 3: LLM inference la- tency breakdown.

Key Results

  • Video Quality Restored: HAFS delivers 1.5× higher video quality (VMAF) compared to uncoordinated baselines, matching the visual performance of a video-only stream.
  • Faster Responses: HAFS reduces agent response time by 31% compared to standard WebRTC/SCTP setups, and runs up to 3.0× faster than processing raw text entirely on the receiving device.
  • Bounded Queues: Under Wi-Fi 6 testing, HAFS achieved a 4.6× lower data channel p95 RTT (94 ms vs. 432 ms for uncoordinated traffic), keeping video frame stalls near zero (0.1%).
  • Hardware Outperformance: Streaming a remote KV cache via HAFS finished 2.2× to 6.2× faster than performing local prefill even on a dedicated on-device NPU ASIC.

Limitations and Future Work

  • Fixed Context Granularity: HAFS currently assumes that input contexts are prefilled into KV caches at a fixed, task-specific chunk size. Overly coarse chunks waste bandwidth on irrelevant data, while overly fine chunks degrade LLM attention accuracy. The authors plan to explore dynamic KV cache granularity optimization.
  • SFU Memory Scalability: As the number of concurrent users scales up, managing individual transport buffers for every agent flow could overwhelm the memory capacity of the central SFU. Integrating hierarchical storage offloading is a noted direction for future engineering.
  • Cross-Host Contention Limits: HAFS operates at the end-host app layer; it cannot enforce fairness or cross-flow priorities if independent traffic from entirely different hosts induces network queuing.

Summary updated

Abstract

AI agents are enabling a new paradigm of agent-augmented real-time communication (RTC), where humans focus on high-level collaboration, while agents autonomously retrieve, analyze, and generate information in real time to support their interactions. These apps enable new experiences across various domains: for example, when corporate employees co-author a legal document, their agents can discuss and draft on their behalf, sparing them the burden of manually reviewing each other's work. As existing cloud-based agents suffer from privacy risks and unscalable server costs, on-device agent-augmented RTC offers a promising alternative. However, this on-device paradigm introduces a new networking challenge: contention between concurrent traffic flows generated by humans (for live video streaming) and agents (for sending context files for analysis). We design HFS, a framework to ensure both high live video quality and low agent response latency in agent-augmented RTC apps. We achieve the goal through an app-guided multi-flow transport approach, where a unified app-layer orchestrator jointly controls the sending rates of live video and agent context flows based on their heterogeneous app requirements. Our prototype built atop WebRTC and llama.cpp demonstrates that HAFS outperforms baselines, achieving 1.5x higher video quality while reducing agent response time by 31%.
BibTeX
@misc{lee2026coordinated,
  author = {Goodsol Lee and Juheon Yi and Jinglu Wang and Haowen Xu and Saewoong Bahk and Yan Lu},
  title = {{Coordinated Networking for On-Device Agent-Augmented Real-Time Communication}},
  year = {2026},
  eprint = {2607.22854},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.22854},
}
Figure