In the pursuit of higher network throughput, the industry has moved toward "multipath" architectures—using Wi-Fi and 5G simultaneously on a phone, or spraying packets across dozens of paths in a data center. However, this creates a fundamental conflict: while the network layer wants to use every available path to maximize capacity, the transport layer (TCP) was designed with the assumption of a single, in-order pipe.
When packets from the same stream take different paths, they often arrive out of order. To a traditional transport protocol, this reordering looks indistinguishable from packet loss caused by congestion. This "multipath paradox" means that adding more paths can actually slow a connection down.
The Cost of Out of Order Delivery
The primary reason packet reordering degrades performance is a misunderstanding at the protocol level. As Gabriel Istrate, Anders Hansson, and colleagues note in An investigation of packet reordering in TCP traces, TCP interprets "gap" packets as a sign of network distress. This triggers unnecessary retransmissions and causes the sender to slash its transmission window, leading to massive throughput collapses.
Beyond protocol confusion, there is a physical bottleneck: the receiver buffer. In concurrent multipath transfer (CMT), if a packet on a slow path is delayed, all subsequent packets arriving from faster paths must wait in a buffer until the "hole" is filled. This phenomenon, known as head-of-line blocking, is explored by Xiaomin Jin and Yuanan Liu in A Throughput Improved Path Selection Method. They demonstrate that the application layer effectively stalls, negating the benefits of the faster paths.
The Asymmetry Trap
Intuition suggests that combining a 100Mbps link and a 10Mbps link should yield 110Mbps. However, research into 5G and heterogeneous networks shows that the speed of the links matters less than their latency.
Jiayang Song and Ping Dong found that delay asymmetry is a far more potent performance killer than bandwidth asymmetry. In A Performance Analysis Model of TCP over Multiple Heterogeneous Paths, they show that large differences in Round-Trip Times (RTT) can reduce performance by nearly 3x, as the receiver is constantly waiting for the "slow" path to deliver its packets so the stream can be reassembled.
Strategies for Orderly Multipathing
Researchers have developed several distinct strategies to reconcile multipath efficiency with the need for order.
1. Flowlet and Flowcut Switching
Instead of spraying individual packets, systems can route "flowlets"—bursts of packets separated by a small time gap. Ang Chen and Kuo-Feng Hsu describe this in Contra. If the gap is larger than the latency difference between paths, the new burst can be safely moved to a different path without arriving before the previous burst.
Building on this, Tommaso Bonato and Daniele De Sensi proposed Flowcut switching in Flowcut Switching: High-Performance Adaptive Routing with In-Order Delivery Guarantees. This algorithm dynamically identifies these "cuts" in a flow to maximize network utilization while providing hardware-level guarantees that packets stay in order.
2. Predictive Scheduling
For mobile devices using multiple radios (like LTE and Wi-Fi), the MARS algorithm by Guolin Sun and colleagues MARS: Multiple Access Radio Scheduling uses real-time RTT measurements to "offset" packet transmissions. By sending packets earlier on slower paths, they aim to have them arrive at the receiver at the same time as packets sent later on faster paths, minimizing the time data spends in the reassembly buffer.
3. Traffic-Aware Coding
In data centers, where "random spraying" is common, Jiawei Huang 0001 and Shiqi Wang 0012 introduced HTPC HTPC: heterogeneous traffic-aware partition coding. This method uses redundancy to fix reordering. By sending extra "parity" packets, the receiver can mathematically reconstruct a missing (delayed) packet without waiting for it to arrive over a congested path.
Moving Beyond Strict In-Order Delivery
The most radical shift involves changing the transport protocols themselves. In Datacenter Ethernet and RDMA: Issues at Hyperscale, Torsten Hoefler and Duncan Roweth argue that the "strict in-order" requirement is a relic of the past.
They advocate for selective acknowledgment (SACK) and out-of-order processing. Instead of a "Go-back-N" strategy—where losing one packet forces the retransmission of everything that followed—modern interfaces should process packets as they arrive, regardless of order. This shift would allow networks to balance traffic across every available route without the constant fear of a single delayed packet stalling the entire system.