Fisc: A Large-scale Cloud-native-oriented File System

Qiang Li, Lulu Chen, Xiaoliang Wang, Shuo Huang, Qiao Xiang, Yuanyuan Dong, Wenhui Yao, Minfei Huang, Puyuan Yang, Shanyang Liu, Zhaosheng Zhu, Huayong Wang, Haonan Qiu, Derui Liu, Shaozong Liu, Yujie Zhou, Yaohui Wu, Zhiwu Wu, Shang Gao, Chao Han, Zicheng Luo, Yuchao Shao, Gexiao Tian, Zhongjie Wu, Zheng Cao, Jinbo Wu, Jiwu Shu, Jie WuFAST 2023 · OpenReview

TL;DR

Fisc is a novel cloud-native-oriented file system designed to address the limitations of traditional systems in modern containerized environments. By offloading heavy client functions to DPUs and introducing a storage-aware distributed gateway, Fisc significantly reduces client resource consumption, improves performance, and enhances availability for cloud-native applications.

What Problem Does This Paper Address?

The paper tackles the challenge of providing efficient and scalable file system services for cloud-native applications, which are increasingly deployed using containers. Existing distributed file systems are ill-suited for this environment due to two main issues:

  1. Heavyweight Clients: Traditional file system clients are resource-intensive, requiring significant CPU and memory. In containerized environments where many containers share a single server, this leads to low resource utilization and limits the number of containers a server can host.
  2. Network Gateway Bottlenecks: Cloud-native applications often rely on a network gateway for file system access. This gateway introduces latency, can be a single point of failure, and struggles with efficient load balancing and data locality optimization, hindering the performance and availability required by modern applications.

Methods

Fisc introduces a multi-pronged approach to overcome these challenges:

  • Lightweight Fisc Client: Fisc significantly reduces the resource footprint of its client by offloading non-essential functions (storage protocols, security, network stacks) to DPUs (Data Processing Units) on computation servers (Fisc agents) and to proxies on storage nodes (Fisc proxies). This two-layer aggregation allows for efficient resource multiplexing.
  • Storage-Aware Distributed Gateway (SaDGW): Instead of a centralized gateway, Fisc employs a distributed SaDGW. This gateway establishes direct, high-performance "highways" between Fisc agents and Fisc proxies. Crucially, it operates with file granularity, enabling storage-aware routing, failure handling, and locality optimizations.
  • Software-Hardware Co-design with DPUs: Fisc leverages DPUs to implement key functionalities. A virtio-Fisc device is developed for secure and efficient passthrough from virtual containers to the physical storage cluster. A fast path is implemented in the DPU's FPGA to cache route tables, accelerating file request processing.
  • Abstraction and Compatibility: Fisc provides an abstracted virtual RPC (vRPC) service, simplifying integration for cloud-native applications. It also employs a Protocol Buffers (PB)-based mechanism for maintaining compatibility across different client versions.

Datasets

Not discussed in the paper. The evaluation uses microbenchmarks and a production environment.

Key Insights and Ideas

  • Resource Aggregation is Key: The core insight is that many file system functions are common across applications and can be aggregated and offloaded to specialized hardware (DPUs) and distributed gateways, rather than being replicated in every client.
  • File Granularity Matters: Moving beyond network connection-based routing to file-granularity routing in the gateway is crucial for effective load balancing and storage-aware operations.
  • DPUs are Enablers: DPUs are instrumental in enabling Fisc's lightweight clients and efficient gateway by providing the necessary processing power and hardware acceleration for offloaded functions.
  • End-to-End QoS: Fisc's design allows for end-to-end Quality of Service (QoS) enforcement, which is vital for supporting diverse cloud-native applications with varying latency and throughput requirements.

Key Results

  • Client Efficiency: Fisc clients consume 69% less CPU and 20% less memory compared to traditional file system clients. In a production environment, this translated to a 16% reduction in CPU and 57% reduction in memory for a Swift service.
  • Performance Improvement: Fisc reduces average write and read latency by 63% and 61% respectively, compared to a network-gateway-based load balancing solution. In some scenarios, Fisc reduces average and P999 tail latency by up to 76% and 92%.
  • Availability: Fisc demonstrates robust failure handling, with proxy failures having a limited impact on tail latency, quickly returning to normal levels. P999 tail latency in production remains stable at around 30 ms.
  • Production Deployment: Fisc has been deployed in a production data center for over three years, serving applications running on over 3 million cores.

Limitations and Future Work

  • DPU Resource Scarcity: While DPUs are powerful, their resources are still scarce and must be shared with other virtualization services. Fisc employs various optimization techniques, but continued efforts are needed as throughput increases.
  • Virtio-Fisc Device Evolution: The paper mentions two generations of virtio-Fisc devices, with the latter being a customized design more akin to a NIC device, suggesting ongoing development and refinement.
  • Ecosystem Expansion: Fisc aims to extend its ecosystem by abstracting virtio-Fisc devices to more general virtio-RPC devices, making them suitable for a broader range of cloud-native services. They also plan to contribute their virtio-Fisc driver to the open-source community.

Summary updated

BibTeX
@inproceedings{li2023fisc,
  author = {Qiang Li and Lulu Chen and Xiaoliang Wang and Shuo Huang and Qiao Xiang and Yuanyuan Dong and Wenhui Yao and Minfei Huang and Puyuan Yang and Shanyang Liu and Zhaosheng Zhu and Huayong Wang and Haonan Qiu and Derui Liu and Shaozong Liu and Yujie Zhou and Yaohui Wu and Zhiwu Wu and Shang Gao and Chao Han and Zicheng Luo and Yuchao Shao and Gexiao Tian and Zhongjie Wu and Zheng Cao and Jinbo Wu and Jiwu Shu and Jie Wu and Jiesheng Wu},
  title = {{Fisc: A Large-scale Cloud-native-oriented File System}},
  booktitle = {FAST},
  year = {2023},
  url = {https://openreview.net/forum?id=PXAUox7Ria},
}

Loading related content...

Figure