SpanFS: A Scalable File System on Fast Storage Devices

Junbin Kang, Benlong Zhang, Tianyu Wo, Weiren Yu, Lian Du, Shuai Ma, Jinpeng Huai

TL;DR

Modern file systems struggle to keep up with the performance of fast storage devices like SSDs, especially on multi-core processors, due to bottlenecks in their centralized design. SpanFS tackles this by breaking down the file system into independent "domains," allowing for parallel processing and significantly improving scalability and performance.

What Problem Does This Paper Address?

As storage devices like Solid State Drives (SSDs) have become much faster and offer high parallelism, traditional file systems, designed for slower Hard Disk Drives (HDDs), are becoming a bottleneck. These older file systems, particularly their journaling mechanisms, rely on centralized structures and processes that can't effectively utilize the power of modern multi-core processors. This leads to poor scalability, where adding more cores doesn't translate to proportional performance gains. The paper aims to address this scalability issue by redesigning the file system architecture to better exploit the capabilities of fast storage and many-core systems.

Methods

The core innovation of SpanFS is its domain-based architecture. Instead of a single, monolithic file system service, SpanFS divides the file system functionality into multiple independent "domains." Each domain acts as a micro file system service, handling its own data allocation, journaling, and other operations.

Key techniques include:

  • Domain Partitioning: The storage device is partitioned into blocks, and these blocks are allocated to different domains. This prevents domains from contending for the same on-disk structures.
  • Dedicated Buffer Cache: Each domain gets its own dedicated buffer cache address space, preventing contention on the global buffer cache layer.
  • Global Namespace Management: SpanFS builds a global hierarchical namespace by designating a root domain and distributing files and directories across other domains. This involves a novel directory entry system (normal, shadow, and remote dentries) to manage distributed objects.
  • Crash Consistency: To maintain consistency across domains, SpanFS employs techniques like stale object deletion, garbage collection, and a two-phase synchronization mechanism for operations like fsync and rename.
  • Parallel Journaling: Each domain has its own journaling instance (based on JBD2), allowing for parallel journaling, committing, and checkpointing without the serialization issues found in centralized journaling.

Datasets

The paper does not explicitly mention specific datasets. Instead, it relies on standard benchmarks and its own micro-benchmarks for evaluation.

  • Benchmarks Used:
    • Micro-benchmarks: catd (create, append, truncate, delete), IOzone, Sysbench.
  • Application-level benchmarks: Filebench (Fileserver and Varmail workloads), Dbench.
  • New Benchmarks/Techniques: The paper introduces a custom catd benchmark to specifically test metadata-intensive operations and uses IOzone and Sysbench to evaluate data-intensive performance.

Key Insights and Ideas

  • Centralized Journaling is the Bottleneck: The paper convincingly demonstrates that the centralized nature of journaling in traditional file systems is a primary cause of poor scalability on many-core systems.
  • Parallelism is Key: By breaking down the file system into independent domains, SpanFS unlocks significant parallelism, allowing multiple I/O operations to proceed concurrently without interference.
  • Namespace Management is Tricky: Distributing files and directories across domains requires careful handling of inter-domain dependencies, which SpanFS addresses with its dentry system and consistency mechanisms.
  • Dedicated Caches Matter: Isolating buffer cache address spaces per domain is crucial to avoid contention that would otherwise arise from multiple cores accessing the same device.
  • Rename is Expensive: The distributed nature of SpanFS introduces significant overhead for operations like rename due to the need to update metadata across multiple domains and maintain consistency.

Key Results

SpanFS demonstrates substantial performance improvements over Ext4, especially as the number of cores increases:

  • Micro-benchmarks: SpanFS outperforms Ext4 by up to 1226% in micro-benchmarks.
  • Application-level benchmarks: SpanFS improves performance by up to 73% relative to Ext4.
  • Scalability: Figures show SpanFS scaling much better than Ext4 on a 32-core machine, with performance gains increasing with more cores. For example, in the append benchmark, SpanFS with 16 domains is 7.53X faster than Ext4 at 32 cores.
  • Lock Contention: SpanFS significantly reduces lock contention, with the total wait time for hot locks reduced by approximately 18X compared to Ext4.
  • Rename Overhead: While generally faster, SpanFS incurs significant overhead for rename operations, being 24X to 39X slower than Ext4 at one and 32 cores, respectively.

Limitations and Future Work

  • Static Domain Allocation: The current prototype uses static allocation of block groups to domains. The authors suggest that dynamic reallocation of block groups on demand would be a valuable future enhancement.
  • Rename Performance: The significant overhead for rename operations is a notable limitation.
  • Dynamic Domain Creation: The current prototype builds domains sequentially; exploring parallel domain creation and dynamic domain resizing strategies is identified as future work.
  • Online Adjusting of Domain Size: Implementing the online adjustment of domain sizes and exploring different reallocation policies are also future research directions.

Summary updated

BibTeX
@inproceedings{kang2015spanfs,
  author = {Junbin Kang and Benlong Zhang and Tianyu Wo and Weiren Yu and Lian Du and Shuai Ma and Jinpeng Huai},
  title = {{SpanFS: A Scalable File System on Fast Storage Devices}},
  booktitle = {USENIX ATC},
  year = {2015},
}
Figure