In traditional networking, packets are treated like a sequence of numbered pages in a book. If page 5 arrives before page 4, the system stops and waits—a phenomenon known as Head-of-Line (HoL) blocking. To prevent this, data centers typically use "flow-based" routing, where every packet in a single conversation follows the exact same path. While this keeps things in order, it leads to massive inefficiency: one path might be congested while three others sit idle.
A growing body of research suggests that for high-performance AI and data center workloads, we should stop trying to keep packets in order. Instead, we can embrace out-of-order delivery by making packets "self-describing" and placing them directly into their final destination in memory.
The Puzzle Piece Intuition
Imagine you are sending a 1,000-piece jigsaw puzzle to a friend. In a traditional network (like TCP), you must send piece #1, then #2, then #3. If piece #4 gets lost, your friend stops working until it's resent.
In the emerging "Direct Memory Placement" paradigm, every puzzle piece has its exact X and Y coordinates printed on the back. Your friend has the puzzle board laid out. When a piece arrives—regardless of whether it's piece #500 or piece #2—they simply look at the coordinates and snap it into the correct spot on the board. They don't need to wait for the "previous" piece to continue their work.
Leveraging RDMA for Multi-Path Routing
The foundation of this idea lies in Remote Direct Memory Access (RDMA). As Wendi Feng, Yang Zhang, and Zhi-Li Zhang demonstrate in A Novel Software-based Multi-path RDMA Solutionfor Data Center Networks, RDMA's ability to write to specific memory offsets naturally solves the out-of-order problem.
Because the hardware knows exactly where a segment belongs in the application buffer, the message is correctly assembled regardless of which network path it took or what order it arrived in. This allows systems like Virtuoso to "spray" packets across every available path in the network, maximizing throughput without the software overhead of reassembly.
Per-Packet Spraying and Load Balancing
When we stop worrying about order, we gain the ability to perform "per-packet load balancing." Instead of pinning a whole "flow" to one path, we can send every individual packet down the least-congested route available at that microsecond.
Tommaso Bonato and colleagues introduce this in REPS: Recycled Entropy Packet Spraying for Adaptive Load Balancing and Failure Mitigation. By decoupling the transport from in-order delivery, their system can react to link failures or congestion instantly. If a path becomes slow, the next packet simply goes elsewhere. This "packet spraying" achieves up to a 6x speedup in bursty data center traffic because it eliminates the idle capacity inherent in traditional hashing.
Why AI Training is the Perfect Use Case
Distributed Deep Learning (DNN) is particularly well-suited for this approach. In Domain-specific Communication Optimization for Distributed DNN Training, Hao Wang and Jingrong Chen point out that gradient updates in stochastic gradient descent are often additive and commutative.
In these workloads, the network doesn't just benefit from out-of-order delivery—it can sometimes ignore order entirely. If the goal is to sum up a bunch of numbers (gradients), it doesn't matter which number arrives first. By treating packets as independent updates rather than a rigid stream, researchers can maximize bandwidth utilization and remove the "anchor" of communication overhead that slows down GPU clusters.
Stripping Down the Hardware
This shift also allows for much simpler, faster hardware. Traditional network interfaces require large "Reorder Buffers" (RoB) to hold out-of-order packets until the missing ones arrive. These buffers consume expensive silicon area and add latency.
In FlooNoC: A 645 Gbps/link 0.15 pJ/B/hop Open-Source NoC with Wide Physical Links and End-to-End AXI4 Parallel Multi-Stream Support, Tim Fischer and Luca Benini show that by offloading the responsibility of sequencing to the DMA engine or the application itself, they can create a "RoB-less" architecture. This significantly reduces the energy and area footprint of the Network-on-Chip (NoC), allowing for staggering speeds of over 600 Gbps per link.
Similarly, Ertza Warraich and the team behind OptiNIC: A Resilient and Tail-Optimal RDMA NIC for Distributed ML Workloads use self-describing packets to enable "best-effort" delivery. By allowing the system to process data as soon as any packet arrives, they reduce tail latency by 3.5×, ensuring that one slow packet doesn't hold up the entire training process.