Offloading Distributed Coordination to Programmable Network Hardware

This direction is split into other directions: protocol processing overhead. Its last version stays readable and citable here.

Version 1 of 1137 papers167 conceptsfrozen 2026-10-04

For decades, the fundamental constraint in distributed systems was the "slow network." Architects designed databases and consensus protocols around a single goal: minimize the number of messages sent across the wire. However, as network speeds have climbed from 10 Gbps to 400 Gbps and beyond, a strange inversion has occurred. The network fabric is now so fast that the primary bottleneck has shifted from the wire to the host CPU.

In modern data centers, the time it takes for a packet to cross the network is often shorter than the time the receiving CPU spends just "noticing" the packet. This shift has birthed a new research direction: moving complex distributed system primitives—like locking, consensus, and telemetry—out of the application and directly into the network hardware.

The Software Communication Tax

Traditional networking relies on a heavy software stack. When a packet arrives, the CPU must handle interrupts, perform context switches, and traverse the TCP/IP stack. Ugur Çetintemel, Carsten Binnig, and their colleagues argue in The End of Slow Networks: It's Time for a Redesign that these legacy layers prevent systems from utilizing the full potential of modern interconnects.

The overhead is not just about latency; it is about CPU exhaustion. In high-speed environments, the host CPU can spend over 90% of its cycles managing the network stack, leaving almost no room for actual application logic. This "communication tax" is particularly visible in microservices, where Nikita Lazarev and Johnson Umeike have shown that Remote Procedure Call (RPC) serialization and parsing can consume a massive portion of a server's compute power.

Bypassing the CPU with RDMA

The first major step toward solving this was Remote Direct Memory Access (RDMA). RDMA allows one computer to read or write directly to the memory of another without involving the remote CPU.

Erfan Zamanian demonstrated the power of this approach in Using RDMA for Lock Management. Instead of a central server managing a lock table and burning CPU cycles to process every request, clients use RDMA "atomic" operations to flip bits in remote memory themselves. By treating remote memory like local RAM—an architecture called Network-Attached Memory (NAM)—researchers like Zamanian have shown that distributed transactions can scale linearly, reaching millions of operations per second.

Moving Logic into the Wire

While RDMA bypasses the CPU, it is still a relatively "dumb" transport. The next frontier is making the network itself "smart" by using programmable switches (via the P4 language) and SmartNICs (using FPGAs or dedicated cores).

Consensus on the Data Plane

Consensus protocols like Paxos are notoriously CPU-intensive because every message requires ordering and persistence logic. Huynh Tu Dang and Pietro Bressana pioneered the idea of "Consensus As A Network Service" (CAANS). In their work on Network Hardware-Accelerated Consensus, the ordering of messages happens "on the wire" inside the switch. Because the switch sees every packet in order anyway, it can perform the role of a Paxos coordinator at line rate, reducing latency from milliseconds to microseconds.

Zero-CPU Telemetry

Monitoring a 100 Gbps network generates a firehose of data that can easily overwhelm a software-based collector. Minlan Yu and Jonatan Langlet addressed this with Direct Telemetry Access. Their system uses programmable switches to translate lightweight network reports into RDMA writes. The telemetry data is written directly into the collector's memory, allowing the system to ingest over a billion reports per second while the collector's CPU remains virtually idle.

The Future of Hardware-Integrated Systems

The research direction is clear: we are moving away from "network-aware" software toward "hardware-integrated" systems.

Projects like Dagger and Arcalis are pushing this further by placing specialized microengines directly adjacent to the CPU's cache to handle RPC tasks. Meanwhile, Tushar Swamy and Alexander Rucker are exploring Taurus, an "intelligent data plane" that can run machine learning models inside the switch to detect anomalies at line rate.

The intuition behind this entire field is that computation is now cheaper than coordination. By offloading the "bookkeeping" of distributed systems—the locking, the ordering, and the protocol parsing—to the network hardware, we can finally build systems where the only limit is the speed of light, not the speed of the interrupt handler.

Papers in this version

Showing 60 of 137 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_offloading_distributed_coordination_to_p_48f843de,
  title = {Offloading Distributed Coordination to Programmable Network Hardware},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/offloading-distributed-coordination-to-programmable-network-hardware/rdr_48f843de8afe4dbd92b468ea2cdfc593},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_offloading_distributed_coordination_to_p_48f843de_v1,
  title = {Offloading Distributed Coordination to Programmable Network Hardware},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/offloading-distributed-coordination-to-programmable-network-hardware/rdr_48f843de8afe4dbd92b468ea2cdfc593/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}