The Study of Mixed Storage Scheme of Private Cloud Platform Based on Ceph

Zuoyang Qu, Chenhui Xie, Chengliang LiuProceedings of the 2018 International Conference on Network, Communication, Computer Engineering (NCCE 2018) 2018 · DOI

Abstract

Ceph is a unique unified system that supports object storage, block storage, and file storage with the characteristic of being highly available, manageable, and free. To build a distributed storage system based on Ceph in a private cloud environment with a high level of data privacy, it can support object storage, block storage, and file storage simultaneously. While Ceph FS and MDS in current community versions are compatible with the OpenStack ecosystem, they often present challenges in actual production use. This article discusses a hybrid storage solution designed for real-world environments. Based on the original Ceph framework, this scheme adds extensions to make native Ceph more suitable for enterprise production, offering an attractive solution for private cloud back-end storage considering both cost and business requirements.

Introduction: Addressing Storage Fragmentation

In a typical OpenStack deployment, storage is often siloed. Different components like Nova (compute), Glance (image services), and Cinder (block storage) frequently rely on separate backend systems. This fragmentation leads to "internal power consumption": when creating a virtual machine, data must be transmitted between these disparate modules, often taking several minutes and creating significant network overhead.

While the community version of Ceph offers a "unified" solution supporting object, block, and file storage, it often falls short in production environments. Issues like the "double-write" penalty in Ceph’s FileStore and the complexity of managing large-scale clusters make it difficult for enterprises to deploy out-of-the-box.

Proposed Solution: Unified Ceph Architecture

The proposed hybrid storage solution integrates Ceph as the single backend for Nova, Glance, and Cinder. This allows for "zero-copy" operations; for example, booting a VM from an image becomes a pointer operation within the same storage pool rather than a massive data transfer.

The layered architecture of Ceph, showing how LIBRADOS provides a foundation for object, block, and file storage interfaces.

Key technical optimizations include:

  • Metadata/Data Separation: To bypass the performance hit of writing logs and data simultaneously (the double-write penalty), metadata writes are separated from actual data blocks.
  • Intelligent Tiering (Hot/Cold): A "hotspot read-ahead" strategy stores frequently accessed data on high-speed NVMe SSDs, while "cold" data is migrated to mechanical SATA disks to optimize hardware life and energy use.
  • Network Aggregation: Communications are optimized by aggregating TCP links to reduce latency.
  • Hardware Acceleration: The architecture utilizes NVMe SSDs (PCIe 3.0) and RAID cards with at least 1GB of cache to handle high IOPS demands.

Implementation and Node Distribution

The physical architecture is divided into specialized nodes to ensure stability and performance. This includes deployment nodes, control nodes managing the cloud platform (Dashboard, Nova, Glance, etc.), computing nodes for guest OS instances, and distributed storage nodes.

The cloud platform architecture illustrating the distribution of OpenStack services across control, computing, and storage nodes.

Experimental Setup and Results

The study utilized a real-world experimental hardware cluster running CentOS 7.3 on the host and Ubuntu 14.04 for guests, powered by the OpenStack "M" version. The storage hardware consisted of a mix of 3.2 TB NVMe SSDs, 600GB 10K SAS drives, and 6TB 7.2K SATA drives. Benchmarking was performed using the fio (Flexible I/O tester) utility.

Performance Gains

The optimizations yielded significant quantitative improvements:

  • Efficiency Boost: The optimized Ceph architecture increased overall efficiency by 130% compared to the standard community version.
  • IOPS Gains: In SSD pool tests, optimized read IOPS jumped from approximately 20,000 to nearly 29,000.
  • Latency: The use of NVMe allowed for 4K read delays of less than 90μs, meeting enterprise-grade requirements.

Conclusion

The biggest advantage of this scheme is the elimination of unnecessary data movement. By concentrating all cloud platform data into a single Ceph resource pool, operations like migrating or expanding virtual machines avoid network bottlenecks. However, while highly effective, this optimized Ceph scheme requires a professional technical service team to maintain. For smaller enterprises without such resources, simpler solutions like NFS might remain more practical despite the performance trade-offs.

Summary updated

Abstract

Ceph is a unique unified system that supports object storage, block storage, and file storage with the characteristic of high available, manageable, and free. To build a distributed storage system based on Ceph in a private cloud environment with a high level of data privacy, it can support object storage, block storage, and file storage simultaneously. Ceph FS and MDS in the current community version are friendlier to ecological environment of OpenStack. However, community versions of Ceph have many problems with their actual use. This article discusses a hybrid storage solution that can be used in a real environment. Based on Ceph's original framework, it adds some extensions to make the native Ceph more suitable for the enterprise's production environment. There is a very attractive solution by using Ceph as a back-end storage for a private cloud platform whether consider about cost and business considerations.
BibTeX
@inproceedings{qu2018study,
  author = {Zuoyang Qu and Chenhui Xie and Chengliang Liu},
  title = {{The Study of Mixed Storage Scheme of Private Cloud Platform Based on Ceph}},
  booktitle = {Proceedings of the 2018 International Conference on Network, Communication, Computer Engineering (NCCE 2018)},
  year = {2018},
  doi = {10.2991/ncce-18.2018.40},
}
Figure