TL;DR
Superfluid is an open-source LLM server designed to handle the "messy" reality of local AI: multiple background agents competing with a human user on a single device, and a "fleet" of idle, heterogeneous machines (Macs, PCs, workstations) scattered across a local network. By separating the serving logic from the inference runtime and implementing a QoS-aware scheduler, it achieves near-instant interactive responsiveness even under heavy load and turns a collection of random hardware into a single, fault-tolerant API endpoint.
What Problem Does This Paper Address?
Local LLM inference is booming due to privacy, zero per-token costs, and offline availability. However, the authors identify two major shifts that current tools like Ollama or llama-server aren't built for:
- The Agentic Shift: We no longer just chat with one model. We have coding agents, email summarizers, and document readers running in the background. On a single device, these long-running "agentic" tasks often saturate the GPU, causing a human user to wait minutes for a simple chat response.
- Compute Fragmentation: Most developers or small offices own several capable machines (e.g., a MacBook, a Mac Studio, and an NVIDIA-powered workstation). Currently, these sit idle because setting up a distributed cluster is complex, and existing local servers are usually bound to one runtime on one machine.
Methods
Superfluid introduces a decoupled architecture where a daemon (the "fleet head") manages the API, scheduling, and session logging, while isolated worker processes handle the heavy lifting of model inference.
Key technical innovations include:
- Multi-Runtime Support: It drives
llama.cpp,MLX, orBaseRTthrough a unified interface. - QoS Scheduling: It implements four priority classes (interactive, completion, agent, background). Interactive requests can preempt lower-priority agent work, ensuring the human user gets a fast response.
- Durable Sessions: Every token is written to a durable log (using
fsync) before delivery. If a worker process crashes due to an out-of-memory (OOM) error or a GPU fault, the daemon simply restarts it and resumes the session from the log. - Fleet Mode: Multiple machines can join a single Superfluid head via mDNS. The head load-balances requests based on the measured speed of each node (prefill/decode rates) rather than just static hardware specs.
- Shared-Prefix Coalescing: If 32 agents all start with the same 4k-token system prompt, Superfluid computes it once and shares the result, rather than recomputing it 32 times.
Datasets
As a systems paper, the authors do not introduce new training datasets. Instead, they evaluate the system using:
- Models: Qwen3.8-27B and Qwen3.6-35B-A3B (quantized to 4-bit).
- Simulated "Office Workload": A custom benchmark where 64 simulated users perform multi-turn tasks (emails, coding, document analysis) with varying pause times to mimic real-world human-agent interaction.
Key Insights and Ideas
- Robustness through Isolation: By running the inference engine in a separate process from the API server, a "segmentation fault" in a beta runtime doesn't take down the entire service.
- Preemption vs. FIFO: Standard LLM servers treat all requests equally. Superfluid’s "aha moment" is treating LLM inference like an OS scheduler—pausing background "batch" jobs to give the "UI" (the human chat) immediate CPU/GPU cycles.
- Heterogeneity is a Strength: In Fleet Mode, you don't need a cluster of identical H100s. You can mix an Apple M1 Max with an NVIDIA GB10. The scheduler automatically routes more work to the faster machine without manual configuration.
Key Results
The performance gains are most dramatic under heavy multi-agent load.
- Interactive Latency: On an Apple M5 Pro, a chat request sent while 8 agents were busy received its first token in 1.5 seconds with Superfluid. Under the same conditions,
Ollamaandllama-servertook between 291 and 612 seconds. - Fleet Efficiency: In a test with six heterogeneous machines, Superfluid handled over 5,000 requests without a single failure, even when nodes were intentionally killed and rejoined mid-stream.
- Massive Scaling: When deployed on twelve Mac Studio M3 Ultras, the system sustained a staggering 10,656 tokens/second across nearly 40,000 requests from 384 concurrent agents.
Limitations and Future Work
- Model Replication: Fleet mode currently requires every node to hold a full copy of the model. It does not support model sharding (splitting one giant model across multiple small GPUs), though the authors note this is complementary to their approach.
- Context Recomputation: If a session moves from one node to another (e.g., due to a failure), the new node must recompute the prompt because KV caches are not currently exchanged between nodes.
- Feature Gaps: Advanced features like image/audio support and speculative decoding are currently only implemented in the
BaseRTruntime. - Security: The system currently lacks TLS for fleet communication, making it suitable only for trusted local networks or VPNs.
Summary updated