Boosting I/O performance of internet servers with user-level custom file systems

Jun Wang, Rui Min, Zhuying Wu, Yiming HuSigmetrics Perform. Evaluation rev 2001 · DOI

TL;DR

Internet servers, especially proxy caches, are bottlenecked by slow disk I/O. This paper introduces WPSFS, a user-level custom file system that bypasses traditional OS file systems to dramatically improve I/O performance for these applications. By managing data directly on raw disks and employing intelligent in-memory metadata and page-based storage, WPSFS achieves significant speedups.

What Problem Does This Paper Address?

Large-scale internet servers, such as web proxy caches, often face severe performance limitations due to disk I/O. Traditional operating system file systems (like Unix FFS or Windows NT) are designed for general-purpose use and incur significant overhead, especially with the small file sizes common in web caching. While some optimizations exist (e.g., SQUIDML using mmap), they offer limited improvements. Specialized operating systems are too complex and costly to maintain. The challenge is to achieve high I/O performance for these servers without the complexity and cost of kernel-level modifications or specialized OSs.

Methods

The core idea is to build a user-level, custom-designed file system that runs as an application component, managing data directly on a raw disk or partition. This bypasses the overhead of the host OS file system.

Key techniques include:

  • In-Memory Metadata Table: Instead of disk-based directory lookups, WPSFS stores MD5 checksums of URLs and file locations in a large in-memory hash table. This allows for extremely fast, disk-free metadata lookups and updates.
  • Page-structured File System (PFS): WPSFS divides disk space into large, fixed-size pages (e.g., 64KB).
  • Writes: Data and metadata are buffered in RAM and written to disk in full page-sized writes, similar to Log-structured File Systems (LFS), which is efficient for sequential writes.
  • Reads: When a file is read, the entire page containing that file is read into memory. This acts as implicit prefetching, leveraging the locality of web accesses. This also eliminates the garbage collection problem inherent in LFS, as entire pages are invalidated in RAM, not on disk.
  • Locality-based Grouping: An algorithm groups files with similar locality (e.g., from the same host or path) into the same disk pages. This enhances the effectiveness of the page-based read and prefetching strategy.
  • Buffer Cache: Manages active files in RAM, working with the locality-based grouping to prepare pages for disk writes.

Datasets

The paper uses NLANR cache traces for evaluation. These are large, real-world web proxy traces that capture typical internet server workloads. The authors simulated requests that are cacheable according to HTTP 1.1 standards and used the first 150,000 requests for system warm-up before measuring performance.

Key Insights and Ideas

  • User-level is Sufficient: The paper demonstrates that significant I/O performance gains can be achieved by moving file system logic to user-space, avoiding the need for complex kernel modifications or specialized OSs.
  • Metadata is the Bottleneck: By keeping all metadata in memory, WPSFS eliminates a major source of I/O overhead that plagues traditional file systems.
  • Page-based I/O as Prefetching: Reading entire pages, even if only one file is immediately needed, acts as effective prefetching. This is particularly beneficial for web workloads where files from the same site are often accessed together, turning reads into opportunities to load future data.
  • Eliminating Garbage Collection: The page-only read approach in PFS elegantly sidesteps the garbage collection overhead that plagues LFS, a significant advantage for file system design.
  • Small File Efficiency: The page-based approach, combined with locality grouping, efficiently handles the large number of small files common in web caching, reducing internal fragmentation.

Key Results

WPSFS shows dramatic improvements in I/O performance:

  • Average Disk Latency per URL Request: WPSFS achieves 9–20 times better average I/O latency than SQUID/FFS and 4–10 times better than SQUIDML.
  • Disk I/O Numbers: WPSFS reduces disk I/Os by 89–93% compared to SQUID/FFS and by 55% on average compared to SQUIDML.
  • Read/Write Speedup: Disk write operations see a 10–43 times improvement, while disk reads achieve a 5–10 times speedup.
  • Memory Hit Rates: WPSFS achieves significantly higher memory hit rates (up to 309% and 178% better than baselines) due to its effective prefetching.

Limitations and Future Work

  • Limitations:
    • The paper focuses on proxy servers, and while it suggests applicability to web servers, extensive evaluation for other server types is not detailed.
  • The current implementation has a limit of 4 million cache entries in the metadata hash table (though this is still substantial and consumes less RAM than SQUID).
  • Files larger than 256KB are not cached.
  • Future Work: The authors suggest that this user-level, custom file system approach can be used to accelerate I/O performance for other types of web servers. Further details of the design can be found in their technical report [9].

Summary updated

BibTeX
@article{wang2001boosting,
  author = {Jun Wang and Rui Min and Zhuying Wu and Yiming Hu},
  title = {{Boosting I/O performance of internet servers with user-level custom file systems}},
  journal = {Sigmetrics Perform. Evaluation rev},
  year = {2001},
  doi = {10.1145/572317.572322},
}
Figure