Chatterbox: Robust Transport for LLM Token Streaming under Unstable Network

Hanchen Li, Yuhan Liu, Yihua Cheng, Siddhant Ray, Kuntai Du, Junchen Jiang

Introduction

Large Language Model (LLM) applications, such as AI chatbots, rely on token streaming to provide a responsive user experience. Instead of waiting for a complete response, the server transmits tokens as they are generated. However, this real-time interaction is highly sensitive to network conditions. In unstable environments—such as mobile networks or congested public Wi-Fi—packet loss leads to significant "stalls," where the user interface freezes despite subsequent data arriving at the device.

LLM Token Generation and Streaming Workflow

The Problem: Retransmission Blocking

The primary cause of these stalls is Head-of-Line (HoL) blocking inherent in standard Transmission Control Protocol (TCP). TCP requires data to be delivered in strict sequential order. If a packet containing an early token (e.g., "Token #1") is lost, the receiver cannot pass later tokens (e.g., "Token #2" and "#3") to the application until the missing packet is retransmitted and received. Measurement studies show that even a modest 10% packet loss can escalate a sub-second delay into a multi-second stall, severely degrading the user experience.

The Eloquent Transmission Scheme

To address this, the authors introduce Eloquent, a transport scheme tailored for the unique characteristics of LLM traffic. Unlike video streaming, where data packets are large and bandwidth-heavy, LLM tokens are extremely small (often just a few bytes). However, standard network packets (MTU) typically support up to 1,500 bytes. Eloquent exploits this "free real estate" by bundling previously unacknowledged tokens into every new packet.

Key Mechanisms

  1. Smart Redundancy: The sender maintains a buffer of "unacked" tokens—those sent but not yet confirmed by the client. When a new token is generated, it is placed in a packet along with as many previous unacked tokens as the MTU allows.
  2. Independent Rendering: Because each packet is "self-contained" with its immediate history, the client can render the entire sequence immediately upon receiving any packet, bypassing the need to wait for specific TCP retransmissions.
  3. Protocol Agnostic: While the logic is demonstrated as a new scheme, it is designed to be implemented on top of modern UDP-based protocols like QUIC or RTP.
Comparison of Original TCP, 100% Duplication, and Eloquent Transmission

Evaluation and Results

The researchers evaluated Eloquent using a combination of synthetic simulations and real-world LTE network traces collected from mobile hotspots.

  • Stall Reduction: In simulations with 15% packet loss, Eloquent reduced the stall ratio by 71% compared to standard TCP.
  • Efficiency: Eloquent outperformed simple packet duplication (sending every packet twice) by 31.6%. By only including tokens that haven't been acknowledged, it maintains a similar sending rate to standard protocols while providing superior resilience.
  • Real-World Impact: Using traces from an iPhone 11 LTE hotspot, the system reduced late tokens by 40.4%, ensuring a much smoother reading experience even under intermittent connectivity.

Limitations and Future Work

While highly effective, Eloquent faces physical constraints. If a network outage persists, the unacked buffer may eventually exceed the MTU, requiring the system to prioritize the most recent tokens. Furthermore, implementing Eloquent requires application-layer changes to handle custom acknowledgments. Future research aims to develop sophisticated Quality of Experience (QoE) models for LLMs to better balance the tradeoff between redundancy and latency, potentially making the system "context-aware" to optimize resource usage based on user activity.

Summary updated

BibTeX
@inproceedings{li2024chatterbox,
  author = {Hanchen Li and Yuhan Liu and Yihua Cheng and Siddhant Ray and Kuntai Du and Junchen Jiang},
  title = {{Chatterbox: Robust Transport for LLM Token Streaming under Unstable Network}},
  booktitle = {NAIC},
  year = {2024},
  url = {https://openreview.net/forum?id=VqkdlpmhSJ},
}
Figure