Superfluid: Local AI Server for Distributed LLM Inference

Prabod Rathnayaka, Isura Manchanayake, Srikandabala Kogul, Steve Siu, Fabian Waschkowski, Lukas WesemannarXiv 2026 · arXiv

TL;DR

Superfluid is an open-source LLM server designed to handle the "messy" reality of local AI: multiple background agents competing with a human user on a single device, and a "fleet" of idle, heterogeneous machines (Macs, PCs, workstations) scattered across a local network. By separating the serving logic from the inference runtime and implementing a QoS-aware scheduler, it achieves near-instant interactive responsiveness even under heavy load and turns a collection of random hardware into a single, fault-tolerant API endpoint.

What Problem Does This Paper Address?

Local LLM inference is booming due to privacy, zero per-token costs, and offline availability. However, the authors identify two major shifts that current tools like Ollama or llama-server aren't built for:

  1. The Agentic Shift: We no longer just chat with one model. We have coding agents, email summarizers, and document readers running in the background. On a single device, these long-running "agentic" tasks often saturate the GPU, causing a human user to wait minutes for a simple chat response.
  2. Compute Fragmentation: Most developers or small offices own several capable machines (e.g., a MacBook, a Mac Studio, and an NVIDIA-powered workstation). Currently, these sit idle because setting up a distributed cluster is complex, and existing local servers are usually bound to one runtime on one machine.

Methods

Superfluid introduces a decoupled architecture where a daemon (the "fleet head") manages the API, scheduling, and session logging, while isolated worker processes handle the heavy lifting of model inference.

Figure 1: Superfluid’s processes. The daemon owns the APIs, the scheduler, and the session log. Each local model runs in its own worker process; in fleet mode, nodes on other machines serve copies of the same model, each on the runtime that suits its hardware.

Key technical innovations include:

  • Multi-Runtime Support: It drives llama.cpp, MLX, or BaseRT through a unified interface.
  • QoS Scheduling: It implements four priority classes (interactive, completion, agent, background). Interactive requests can preempt lower-priority agent work, ensuring the human user gets a fast response.
  • Durable Sessions: Every token is written to a durable log (using fsync) before delivery. If a worker process crashes due to an out-of-memory (OOM) error or a GPU fault, the daemon simply restarts it and resumes the session from the log.
  • Fleet Mode: Multiple machines can join a single Superfluid head via mDNS. The head load-balances requests based on the measured speed of each node (prefill/decode rates) rather than just static hardware specs.
  • Shared-Prefix Coalescing: If 32 agents all start with the same 4k-token system prompt, Superfluid computes it once and shares the result, rather than recomputing it 32 times.

Datasets

As a systems paper, the authors do not introduce new training datasets. Instead, they evaluate the system using:

  • Models: Qwen3.8-27B and Qwen3.6-35B-A3B (quantized to 4-bit).
  • Simulated "Office Workload": A custom benchmark where 64 simulated users perform multi-turn tasks (emails, coding, document analysis) with varying pause times to mimic real-world human-agent interaction.

Key Insights and Ideas

  • Robustness through Isolation: By running the inference engine in a separate process from the API server, a "segmentation fault" in a beta runtime doesn't take down the entire service.
  • Preemption vs. FIFO: Standard LLM servers treat all requests equally. Superfluid’s "aha moment" is treating LLM inference like an OS scheduler—pausing background "batch" jobs to give the "UI" (the human chat) immediate CPU/GPU cycles.
  • Heterogeneity is a Strength: In Fleet Mode, you don't need a cluster of identical H100s. You can mix an Apple M1 Max with an NVIDIA GB10. The scheduler automatically routes more work to the faster machine without manual configuration.

Key Results

The performance gains are most dramatic under heavy multi-agent load.

  • Interactive Latency: On an Apple M5 Pro, a chat request sent while 8 agents were busy received its first token in 1.5 seconds with Superfluid. Under the same conditions, Ollama and llama-server took between 291 and 612 seconds.
  • Fleet Efficiency: In a test with six heterogeneous machines, Superfluid handled over 5,000 requests without a single failure, even when nodes were intentionally killed and rejoined mid-stream.
Table 1: Local serving on Qwen3.8-27B at 4 bits, Apple M5 Pro (64 GB). Each Superfluid runtime is compared with the servers for the same model format. Arrows give the better direction; the best value within each format is in bold. A dash marks a scenario the server cannot run (it has no separate worker process).
  • Massive Scaling: When deployed on twelve Mac Studio M3 Ultras, the system sustained a staggering 10,656 tokens/second across nearly 40,000 requests from 384 concurrent agents.
Figure 2: Run A: six heterogeneous machines as one endpoint, Qwen3-4B on llama.cpp, 64 simulated users for one hour. Stacked areas give the output tokens per second each machine served (30 s rolling rate); the line gives the median time to first token per minute (right axis, log scale). Dotted lines mark machines joining (+) and the node killed at 42 minutes (−). All 5,041 requests were answered.

Limitations and Future Work

  • Model Replication: Fleet mode currently requires every node to hold a full copy of the model. It does not support model sharding (splitting one giant model across multiple small GPUs), though the authors note this is complementary to their approach.
  • Context Recomputation: If a session moves from one node to another (e.g., due to a failure), the new node must recompute the prompt because KV caches are not currently exchanged between nodes.
  • Feature Gaps: Advanced features like image/audio support and speculative decoding are currently only implemented in the BaseRT runtime.
  • Security: The system currently lacks TLS for fleet communication, making it suitable only for trusted local networks or VPNs.

Summary updated

Abstract

Running large language models (LLMs) on local devices keeps data private, removes per-token cost, cuts network latency, and keeps working without a connection. Two shifts now strain how such models are served. On one device, several agents share a locally served model with an interactive user. Across devices, a developer or small team often owns several capable machines from different vendors, which sit idle most of the day because every local server serves one runtime on one machine. We present Superfluid, an open-source LLM server built for local inference at both scales. On one device, Superfluid drives llama.cpp, MLX, or BaseRT in isolated worker processes through one scheduler that moves interactive requests ahead of agent work, computes a prompt shared by many requests once, and writes every token to a durable log before it reaches the client. Across devices, one Superfluid server turns a fleet of machines into a single API endpoint. Machines with different hardware serve copies of the same model, receive work according to their measured speed, and can join or fail under load without losing a request. For Qwen3.8-27B on an Apple M5 Pro, a chat sent while eight agents are working receives its first token in 1.5 s, against 291-612 s on llama-server, Ollama, and mlx_lm.server. On six machines from three chip vendors serving 64 simulated users for one hour, Superfluid answers all 5,041 requests while machines join one at a time and one is killed and rejoins; throughput grows from 112 to 534 tok/s and the median first-token time falls from 99 to 12 s. On twelve Mac Studio M3 Ultras it sustains 10,656 tok/s across 38,563 requests from 384 agents without a failure. Superfluid is available under the Apache-2.0 licence at https://github.com/basecompute/superfluid.
BibTeX
@misc{rathnayaka2026superfluid,
  author = {Prabod Rathnayaka and Isura Manchanayake and Srikandabala Kogul and Steve Siu and Fabian Waschkowski and Lukas Wesemann},
  title = {{Superfluid: Local AI Server for Distributed LLM Inference}},
  year = {2026},
  eprint = {2611.17753},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17753},
}
Figure