A New View on What Limits TCP/IP Throughput in Local Area Networks

Marc Abrams, Qi ChenVTechWorks 1991

Abstract

This paper presents experimental results on what limits local area network throughput at the application program level for two popular transport protocols, TCP and UDP, using two application program interfaces, Berkeley sockets and System V transport layer interface. The sensitivity of application-level performance to the choice of host computer speed and background load on the host are also studied. Two sets of measurements are discussed. The first contains macroscopic measurement of throughput on 68020, 68030, 68040, 80386, SPARC, and MIPS R2000 and R3000 based computers over a single Ethernet subnet. The second presents a detailed timing analysis using a hardware monitor of a TCP/IP implementation for PC architecture computers. Previous studies implicate memory copying, checksumming, and the operating system interface as the major overheads in TCP/IP, rather than the time required to execute the protocol itself. This study indicates that these factors are secondary when the sender and receiver are closely matched in speed; rather the primary bottleneck is the TCP flow control mechanism. TCP flow control becomes closer to optimal as the degree of speed mismatch increases. We draw conclusions on why window mechanisms should be augmented by rate based flow control in the new generation of high data rate networks.

Introduction

In the early 1990s, as workstations transitioned to RISC architectures and 10Mbps Ethernet became the standard, a persistent performance gap emerged: application-level throughput often fell significantly short of theoretical wire speeds. The prevailing consensus at the time blamed "heavy" protocol overheads, specifically the CPU cycles consumed by memory copying and checksum calculations.

This study challenges that assumption. By analyzing performance across a wide array of hardware and using high-precision hardware monitoring, the researchers discovered that the primary bottleneck wasn't the CPU being overworked—it was the CPU sitting idle. The culprit was the TCP window-based flow control mechanism, which struggled to maintain a steady data stream in local area network (LAN) environments.

Experimental Setup

The researchers employed a two-tiered approach to isolate performance bottlenecks. First, a macroscopic study measured throughput across a diverse "hardware dataset" of CISC and RISC machines, varying host loads, write sizes, and APIs (Berkeley Sockets vs. System V TLI).

Table 1: Characteristics of the diverse hardware and software platforms used in the macroscopic throughput study.

Second, a microscopic study utilized a specialized hardware monitor—a microsecond-resolution clock card—to instrument an MS-DOS/IP implementation. This allowed for precise logging of CPU states, distinguishing between active processing (driver, checksum, protocol logic) and idle time.

The Flow Control Bottleneck

The microscopic analysis yielded a surprising result: during a file transfer on a slow host, the sender remained idle for 45.3% of the time. Rather than struggling with computational overhead, the CPU was blocked, waiting for the receiver to acknowledge data and advance the transmission window.

The Paradox of Matched Speeds

A key insight from the study is that TCP performance is often closer to optimal when there is a significant speed mismatch between the sender and receiver. If a sender is much slower than the receiver, it simply transmits at its maximum capacity. Conversely, if the sender is much faster, it quickly fills the receiver's buffer and then waits. The most severe degradation occurs when speeds are closely matched; in these cases, the window mechanism causes a "stuttering" effect that prevents the protocol from reaching steady-state throughput.

Buffer Sensitivity and API Impact

Throughput proved highly sensitive to the TCP receive buffer size. As shown in Table 1, buffer sizes varied significantly across platforms. Small buffers (e.g., 2KB) forced frequent sender pauses, leading to massive idle times and halving the achievable throughput. Additionally, the choice of API mattered: Berkeley Sockets generally outperformed System V TLI for TCP at small write sizes, while TLI showed advantages for large UDP transfers.

Key Results

  • Throughput Variance: High-end RISC workstations like the DECstation 5000 could nearly saturate the 10Mbps Ethernet (reaching ~8.4 Mbps). In contrast, 80386-based machines peaked at roughly 1.2 Mbps, utilizing only 13% of the available bandwidth.
  • Overhead Breakdown: While data copying and checksumming each accounted for roughly one-third of active processing time, these factors were secondary to the nearly 50% of total time spent idle due to flow control constraints.
  • UDP vs. TCP: UDP, which lacks flow control, achieved higher raw throughput on matched hosts but at the cost of extreme packet loss (up to 57%) when a fast sender overwhelmed a slower receiver.

Conclusion and Future Directions

The findings suggest that for the next generation of high-speed networks, traditional window-based flow control may be insufficient. The authors advocate for augmenting these mechanisms with rate-based flow control, where hosts negotiate a steady transmission rate ($\lambda$) tailored to the receiver's capacity ($\mu$). This approach aims to eliminate the "stop-and-go" behavior of windowing, allowing for more efficient utilization of high-bandwidth links. While modern networks have evolved significantly since 10Mbps Ethernet, the fundamental tension between protocol-induced idle time and hardware capacity remains a cornerstone of network performance research.

Summary updated

Abstract

This paper presents experimental results on what limits local area network throughput at the application program level for two popular transport protocols, TCP and UDP, using two application program interfaces, Berkeley sockets and System V transport layer interface. The sensitivity of application-level performance to the choice of host computer speed and background load on the host are also studied. Two sets of measurements are discussed. The first contains macroscopic measurement of throughput on 68020, 68030, 68040, 80386, SPARC, and MIPS R2000 and R3000 based computers over a single Ethernet subnet. The second presents a detailed timing analysis using a hardware monitor of a TCP/IP implementation for PC architecture computers. Previous studies implicate memory copying, checksumming, and the operating system interface as the major overheads in TCP/IP, rather than the time required to execute the protocol itself. This study indicates that these factors are secondary when the sender and receiver are closely matched in speed; rather the primary bottleneck is the TCP flow control mechanism. TCP flow control becomes closer to optimal as the degree of speed mismatch increases. We draw conclusions on why window mechanisms should be augmented by rate based flow control in the new generation of high data rate networks.
BibTeX
@inproceedings{abrams1991new,
  author = {Marc Abrams and Qi Chen},
  title = {{A New View on What Limits TCP/IP Throughput in Local Area Networks}},
  booktitle = {VTechWorks},
  year = {1991},
}
Figure