TL;DR
On-device AI agents in real-time communication (RTC) apps introduce heavy network traffic when sharing large context files (like LLM KV caches), severely degrading live video quality. This paper presents HAFS (Human-Agent Flow Scheduling), an app-layer framework that co-optimizes human video and agent context flows. HAFS achieves a 1.5× improvement in video quality while slashing agent response times by 31%.
What Problem Does This Paper Address?
Video conferencing platforms are evolving to include on-device AI agents that help users collaborate in real time—for instance, by automatically cross-checking documents or drafting text during a live call. To protect user privacy and avoid massive cloud GPU costs, running these small language models locally on user devices is highly desirable.
However, edge devices struggle with the computationally heavy "prefill" stage of LLM inference. To keep response times low, it is much faster for a sender's agent to pre-compute and stream the raw context as a Key-Value (KV) cache to the receiver. The catch? Even after compression, these KV caches are bulky (often tens of megabytes), leading to severe network contention with the delay-sensitive live video stream.
Current commercial platforms handle this poorly: Zoom allows file transfers to aggressively crowd out video traffic, causing severe video degradation, while Microsoft Teams heavily throttles data transfers, resulting in underutilized bandwidth and sluggish agent performance.
Methods
The authors design HAFS (Human-Agent Flow Scheduling), a coordinated networking framework built directly into the application layer atop WebRTC and llama.cpp. HAFS features two primary components:
- Human-Agent Flow Coordinator: This module dynamically manages the aggregate sending rate. It introduces a Dual Feedback-based Frame Queuing Estimator that pairs standard video transport feedback (RTP) with dense data channel acknowledgments (SCTP) to measure network queuing at a fine, frame-level granularity. An MI/AI-MD (Multiplicative/Additive Increase, Multiplicative Decrease) rate controller then safely pushes agent throughput to the limit without violating the video's frame-delay deadlines.
- Multi-Agent Flow Coordinator: When a query requires pulling multiple contexts simultaneously, this module schedules their transmission order. By framing the challenge as a classic two-machine flow shop problem and applying Johnson's rule, it pipelines network transfers and local GPU decoding. To schedule effectively before decoding occurs, it predicts the expected response length using a lightweight, single-step LLM decoding trick.
Datasets
The framework was evaluated using two standard NLP benchmarks adapted for on-device agent tasks:
- MuSiQue: A multi-document question-answering dataset used to evaluate short-context tasks (<4k tokens).
- QMSum: A meeting summarization dataset used to evaluate long-context workloads (8k–16k tokens).
Key Insights and Ideas
- The Prefill Bottleneck is Real: On edge hardware like phones or edge embedded modules, the LLM prefill phase consumes 71% to 92% of the total response time. Shifting the burden to network transmission via KV cache streaming eliminates this massive compute bottleneck.
- Leveraging the SFU Advantage: Modern video conferencing relies on Selective Forwarding Units (SFUs), which effectively split long end-to-end paths into short sender-SFU and SFU-receiver links. HAFS capitalizes on these low-RTT links to achieve precise, reactive app-layer rate control that normally requires complex in-network router scheduling.
- Relative Ordering Trumps Absolute Accuracy: Pipelining network and compute tasks optimally doesn't require knowing the exact token length of an LLM's response. Because Johnson's rule relies purely on the relative order of workloads, the system remains highly robust to prediction errors.
Key Results
- Video Quality Restored: HAFS delivers 1.5× higher video quality (VMAF) compared to uncoordinated baselines, matching the visual performance of a video-only stream.
- Faster Responses: HAFS reduces agent response time by 31% compared to standard WebRTC/SCTP setups, and runs up to 3.0× faster than processing raw text entirely on the receiving device.
- Bounded Queues: Under Wi-Fi 6 testing, HAFS achieved a 4.6× lower data channel p95 RTT (94 ms vs. 432 ms for uncoordinated traffic), keeping video frame stalls near zero (0.1%).
- Hardware Outperformance: Streaming a remote KV cache via HAFS finished 2.2× to 6.2× faster than performing local prefill even on a dedicated on-device NPU ASIC.
Limitations and Future Work
- Fixed Context Granularity: HAFS currently assumes that input contexts are prefilled into KV caches at a fixed, task-specific chunk size. Overly coarse chunks waste bandwidth on irrelevant data, while overly fine chunks degrade LLM attention accuracy. The authors plan to explore dynamic KV cache granularity optimization.
- SFU Memory Scalability: As the number of concurrent users scales up, managing individual transport buffers for every agent flow could overwhelm the memory capacity of the central SFU. Integrating hierarchical storage offloading is a noted direction for future engineering.
- Cross-Host Contention Limits: HAFS operates at the end-host app layer; it cannot enforce fairness or cross-flow priorities if independent traffic from entirely different hosts induces network queuing.
Summary updated