Towards Efficient Graph Processing in Geo-Distributed Data Centers

Feng Yao, Qian Tao, Shengyuan Lin, Yanfeng Zhang, Wenyuan Yu, Shufeng Gong, Qiange Wang, Ge Yu, Jingren Zhou

Abstract

Iterative graph processing is widely used as a significant paradigm for large-scale data analysis. In many global businesses of multinational enterprises, graph-structure data is usually geographically distributed in different regions to support low-latency services. Geo-distributed graph processing suffers from the Wide Area Networks (WANs) with scarce and heterogeneous bandwidth, thus essentially differs from traditional distributed graph processing. In this paper, we propose RAGraph, a Region-Aware framework for geo-distributed graph processing . At the core of RAGraph, we design a region-aware graph processing framework that allows advancing inefficient global updates locally and enables sensible coordination-free message interactions and flexible replaceable communication module. In terms of graph data preprocessing, RAGraph introduces a contribution-driven edge migration algorithm to effectively utilize network resources. RAGraph also contains an adaptive hierarchical message interaction engine to switch interaction modes adaptively based on network heterogeneity and fluctuation, and a discrepancy-aware message filtering strategy to filter important messages. Experimental results show that RAGraph can achieve an average speedup of 9.7× (up to 98×) and an average WAN cost reduction of 78.5 $\%$ (up to 97.3 $\%$ ) compared with state-of-the-art systems.
BibTeX
@article{yao2024towards,
  author = {Feng Yao and Qian Tao and Shengyuan Lin and Yanfeng Zhang and Wenyuan Yu and Shufeng Gong and Qiange Wang and Ge Yu and Jingren Zhou},
  title = {{Towards Efficient Graph Processing in Geo-Distributed Data Centers}},
  journal = {IEEE Trans. Parallel Distributed Syst.},
  year = {2024},
  doi = {10.1109/tpds.2024.3453872},
}

Loading related content...

Figure