Ceph Parallel File System Evaluation Report

Feiyi Wang, Sarp Oral, Douglas Fuller, Blake Caldwell, James A Simmons, Bradley W. Settlemyer, Scott Atchley, Jason Hill, Mark Nelson2013 · DOI

Abstract

We used Data Direct Networks’ (DDN) SFA10K as the storage backend during this evaluation. It consists of 200 SAS drives and 280 SATA drives, organized into various RAID levels by two active-active RAID controllers. The exported RAID groups by these controllers are driven by four hosts. Each host has two InfiniBand (IB) QDR connections to the storage backend. We used a single dualport Mellanox connectX IB card per host. By our calculation, this setup can saturate SFA10K’s maximum theoretical throughput (~12 GB/s). Our Ceph testbed employs a collection of testing nodes. These nodes and their roles are summarized in Table 1. In the following discussion, we use “servers”, “osd servers”, “server hosts” interchangeably. We will emphasize with “client” prefix when we want to distinguish it from above. All hosts (client and servers) were configured with Redhat 6.3 and kernel version 3.5.1 initially, and later upgraded to 3.9 (rhl-ceph image), Glibc 2.12 with syncfs support, locally patched. We used the Ceph 0.48 and 0.55 release in the initial tests, upgraded to 0.64 and then to 0.67RC for a final round of tests.

Introduction

As High-Performance Computing (HPC) moves toward the Exascale era, traditional storage solutions face daunting scalability and fault-tolerance challenges. Ceph emerged as a promising alternative due to its unique CRUSH algorithm—which allows clients to calculate data placement rather than relying on a central lookup table—and its unified architecture that handles both object and file storage.

However, Ceph was originally designed for commodity hardware. This report investigates whether Ceph can effectively utilize "leadership-class" hardware, specifically the high-bandwidth DDN SFA10K storage backend. The goal is to bridge the gap between Ceph’s theoretical architectural advantages and the practical performance requirements of a top-tier supercomputing facility.

Methodology

The evaluation utilized a dedicated testbed at the National Center for Computational Sciences (NCCS), featuring a DDN SFA10K storage array connected via InfiniBand (IB) QDR. The architecture of this testbed, showing the connection between Object Storage Servers (OSS) and the RAID controllers, is illustrated in Figure 1.

DDN SFA10K hardware and host connection diagram

The researchers employed a "bottom-up" testing strategy:

  1. Baseline Block I/O: Establishing raw performance limits using xdd.
  2. Backend File System Selection: Comparing XFS, Btrfs, and Ext4. XFS was chosen for its maturity and superior performance scaling.
  3. RADOS Benchmarking: Testing the underlying Reliable Autonomic Distributed Object Store (RADOS) for linear scaling.
  4. CephFS Evaluation: Measuring file-level throughput with IOR and metadata operations with mdtest.
  5. Iterative Tuning: Adjusting system parameters and upgrading kernels to resolve performance bottlenecks.

Performance Analysis and Tuning

Backend File System Scaling

A critical step was selecting the local file system for the Object Storage Daemons (OSDs). XFS demonstrated strong read performance scaling as the number of Logical Unit Numbers (LUNs) increased, as shown in Figure 2.

XFS read performance scaling with queue depth and LUN count

Key Insights and Optimization

  • The Journaling Bottleneck: Ceph's "double-write" journaling ensures consistency but effectively halves observed bandwidth on high-end RAID controllers that already provide hardware protection.
  • Kernel Version Impact: Upgrading from Linux kernel 3.5 to 3.9 resolved heavy lock contention in the memory manager, significantly improving read performance.
  • TCP and Checksum Tuning: Disabling standard TCP autotuning (tcp_moderate_rcvbuf) and client-side checksumming (nocrc) was necessary to maintain steady high-concurrency throughput and reduce CPU overhead.

Results

The study pushed Ceph to its limits on the DDN hardware:

  • Hardware Baseline: The raw storage backend achieved ~11 GB/s.
  • RADOS Scaling: The object store showed near-linear scaling, reaching hardware saturation with 9 OSDs per server.
  • CephFS Throughput: After tuning and upgrading to Ceph 0.64, write speeds reached ~5.2 GB/s and read speeds ranged from 5.6 to 7 GB/s across 8 client nodes.
  • Metadata Performance: While directory creation was efficient, both file and directory operations saw a 50% performance drop as client counts increased, indicating contention on the single Metadata Server (MDS).

Conclusion and Future Work

While Ceph shows promise for HPC, several hurdles remain. The "double-write" journaling mechanism must be optimized to avoid sacrificing 50% of hardware potential. Furthermore, future evaluations must address MDS scalability through clustered configurations to handle the metadata demands of large-scale concurrent clients. Continued maturation of the Ceph codebase is essential for its adoption in production supercomputing environments.

Summary updated

Abstract

We used Data Direct Networks’ (DDN) SFA10K as the storage backend during this evaluation. It consists of 200 SAS drives and 280 SATA drives, organized into various RAID levels by two active-active RAID controllers. The exported RAID groups by these controllers are driven by four hosts. Each host has two InfiniBand (IB) QDR connections to the storage backend. We used a single dualport Mellanox connectX IB card per host. By our calculation, this setup can saturate SFA10K’s maximum theoretical throughput (~12 GB/s). Our Ceph testbed employs a collection of testing nodes. These nodes and their roles are summarized in Table 1. In the following discussion, we use “servers”, “osd servers”, “server hosts” interchangeably. We will emphasize with “client” prefix when we want to distinguish it from above. All hosts (client and servers) were configured with Redhat 6.3 and kernel version 3.5.1 initially, and later upgraded to 3.9 (rhl-ceph image), Glibc 2.12 with syncfs support, locally patched. We used the Ceph 0.48 and 0.55 release in the initial tests, upgraded to 0.64 and then to 0.67RC for a final round of tests.
BibTeX
@misc{wang2013ceph,
  author = {Feiyi Wang and Sarp Oral and Douglas Fuller and Blake Caldwell and James A Simmons and Bradley W. Settlemyer and Scott Atchley and Jason Hill and Mark Nelson},
  title = {{Ceph Parallel File System Evaluation Report}},
  year = {2013},
  doi = {10.2172/1104795},
}
Figure