TL;DR
Fisc is a novel cloud-native-oriented file system designed to address the limitations of traditional systems in modern containerized environments. By offloading heavy client functions to DPUs and introducing a storage-aware distributed gateway, Fisc significantly reduces client resource consumption, improves performance, and enhances availability for cloud-native applications.
What Problem Does This Paper Address?
The paper tackles the challenge of providing efficient and scalable file system services for cloud-native applications, which are increasingly deployed using containers. Existing distributed file systems are ill-suited for this environment due to two main issues:
- Heavyweight Clients: Traditional file system clients are resource-intensive, requiring significant CPU and memory. In containerized environments where many containers share a single server, this leads to low resource utilization and limits the number of containers a server can host.
- Network Gateway Bottlenecks: Cloud-native applications often rely on a network gateway for file system access. This gateway introduces latency, can be a single point of failure, and struggles with efficient load balancing and data locality optimization, hindering the performance and availability required by modern applications.
Methods
Fisc introduces a multi-pronged approach to overcome these challenges:
- Lightweight Fisc Client: Fisc significantly reduces the resource footprint of its client by offloading non-essential functions (storage protocols, security, network stacks) to DPUs (Data Processing Units) on computation servers (Fisc agents) and to proxies on storage nodes (Fisc proxies). This two-layer aggregation allows for efficient resource multiplexing.
- Storage-Aware Distributed Gateway (SaDGW): Instead of a centralized gateway, Fisc employs a distributed SaDGW. This gateway establishes direct, high-performance "highways" between Fisc agents and Fisc proxies. Crucially, it operates with file granularity, enabling storage-aware routing, failure handling, and locality optimizations.
- Software-Hardware Co-design with DPUs: Fisc leverages DPUs to implement key functionalities. A virtio-Fisc device is developed for secure and efficient passthrough from virtual containers to the physical storage cluster. A fast path is implemented in the DPU's FPGA to cache route tables, accelerating file request processing.
- Abstraction and Compatibility: Fisc provides an abstracted virtual RPC (vRPC) service, simplifying integration for cloud-native applications. It also employs a Protocol Buffers (PB)-based mechanism for maintaining compatibility across different client versions.
Datasets
Not discussed in the paper. The evaluation uses microbenchmarks and a production environment.
Key Insights and Ideas
- Resource Aggregation is Key: The core insight is that many file system functions are common across applications and can be aggregated and offloaded to specialized hardware (DPUs) and distributed gateways, rather than being replicated in every client.
- File Granularity Matters: Moving beyond network connection-based routing to file-granularity routing in the gateway is crucial for effective load balancing and storage-aware operations.
- DPUs are Enablers: DPUs are instrumental in enabling Fisc's lightweight clients and efficient gateway by providing the necessary processing power and hardware acceleration for offloaded functions.
- End-to-End QoS: Fisc's design allows for end-to-end Quality of Service (QoS) enforcement, which is vital for supporting diverse cloud-native applications with varying latency and throughput requirements.
Key Results
- Client Efficiency: Fisc clients consume 69% less CPU and 20% less memory compared to traditional file system clients. In a production environment, this translated to a 16% reduction in CPU and 57% reduction in memory for a Swift service.
- Performance Improvement: Fisc reduces average write and read latency by 63% and 61% respectively, compared to a network-gateway-based load balancing solution. In some scenarios, Fisc reduces average and P999 tail latency by up to 76% and 92%.
- Availability: Fisc demonstrates robust failure handling, with proxy failures having a limited impact on tail latency, quickly returning to normal levels. P999 tail latency in production remains stable at around 30 ms.
- Production Deployment: Fisc has been deployed in a production data center for over three years, serving applications running on over 3 million cores.
Limitations and Future Work
- DPU Resource Scarcity: While DPUs are powerful, their resources are still scarce and must be shared with other virtualization services. Fisc employs various optimization techniques, but continued efforts are needed as throughput increases.
- Virtio-Fisc Device Evolution: The paper mentions two generations of virtio-Fisc devices, with the latter being a customized design more akin to a NIC device, suggesting ongoing development and refinement.
- Ecosystem Expansion: Fisc aims to extend its ecosystem by abstracting virtio-Fisc devices to more general virtio-RPC devices, making them suitable for a broader range of cloud-native services. They also plan to contribute their virtio-Fisc driver to the open-source community.
Summary updated