HALCONE : A Hardware-Level Timestamp-based Cache Coherence Scheme for Multi-GPU systems

Saiful A. Mojumder, Yifan Sun, Leila Delshadtehrani, Yenai Ma, Trinayan Baruah, José Luis Abellán, John Kim, David Kaeli, Ajay Joshi

Abstract

While multi-GPU (MGPU) systems are extremely popular for compute-intensive workloads, several inefficiencies in the memory hierarchy and data movement result in a waste of GPU resources and difficulties in programming MGPU systems. First, due to the lack of hardware-level coherence, the MGPU programming model requires the programmer to replicate and repeatedly transfer data between the GPUs' memory. This leads to inefficient use of precious GPU memory. Second, to maintain coherency across an MGPU system, transferring data using low-bandwidth and high-latency off-chip links leads to degradation in system performance. Third, since the programmer needs to manually maintain data coherence, the programming of an MGPU system to maximize its throughput is extremely challenging. To address the above issues, we propose a novel lightweight timestamp-based coherence protocol, HALCONE, for MGPU systems and modify the memory hierarchy of the GPUs to support physically shared memory. HALCONE replaces the Compute Unit (CU) level logical time counters with cache level logical time counters to reduce coherence traffic. Furthermore, HALCONE introduces a novel timestamp storage unit (TSU) with no additional performance overhead in the main memory to perform coherence actions. Our proposed HALCONE protocol maintains the data coherence in the memory hierarchy of the MGPU with minimal performance overhead (less than 1%). Using a set of standard MGPU benchmarks, we observe that a 4-GPU MGPU system with shared memory and HALCONE performs, on average, 4.6× and 3× better than a 4-GPU MGPU system with existing RDMA and with the recently proposed HMG coherence protocol, respectively. We demonstrate the scalability of HALCONE using different GPU counts (2, 4, 8, and 16) and different CU counts (32, 48, and 64 CUs per GPU) for 11 standard benchmarks.

Introduction

Modern AI and simulation workloads have outpaced the memory capacity of single GPUs, necessitating the use of multi-GPU (MGPU) clusters. However, these systems face a significant "memory wall." Currently, most GPUs lack hardware-level cache coherence across separate chips. This deficiency forces developers to manually manage data movement, leads to redundant data replication that wastes VRAM, and subjects the system to the "NUMA effect," where remote memory access via PCIe or NVLink is significantly slower than local access. While traditional CPU coherence protocols like MESI exist, they do not scale to the thousands of parallel threads in a GPU, which would overwhelm the interconnect with coherence traffic.

The HALCONE Architecture

To solve these challenges, we propose HALCONE, a protocol designed for systems with physically shared main memory (MGPU-SM). HALCONE enables a unified memory view while minimizing the communication overhead typically associated with hardware coherence.

Figure 1: High-level system architecture of the MGPU-SM with HALCONE, featuring the Timestamp Storage Unit (TSU) integrated into the HBM stack.

As shown in the architecture above, HALCONE introduces a Timestamp Storage Unit (TSU) within the logic layer of the High Bandwidth Memory (HBM) stack. The TSU is a small, 8-way associative cache that tracks "memory timestamps" (memts) for data blocks. By placing this unit in the HBM logic layer, coherence bookkeeping can occur in parallel with DRAM access, effectively hiding the latency of coherence checks.

Lightweight Timestamp-Based Coherence

HALCONE's core innovation is its move from ownership-based coherence to a timestamp-based "lease" system. Instead of constant polling, HALCONE uses logical timestamps to determine data validity.

Key features include:

  • Cache-Level Counters: Unlike previous work that placed logical clocks in Compute Units (CUs), HALCONE moves them to the L1 and L2 caches. This reduces the synchronization traffic required to keep clocks aligned across the system.
  • Write-Through Policy: HALCONE utilizes a write-through policy for L1 and L2 caches. While often avoided in CPUs, this policy suits the high-bandwidth, streaming nature of GPU workloads, avoiding the complex bottlenecks of write-back ownership transfers.
Figure 2: Transaction flows for read and write operations in HALCONE, illustrating how timestamps (Cts vs Brts) determine hits and coherence misses.

The protocol logic (illustrated above) ensures that when a cache fetches data, it receives a "lease" period. If the current logical time (Cts) is less than the block's expiration timestamp (Brts), the data is a hit. If a write occurs, the TSU updates the memory timestamp, effectively invalidating older leases across the system without needing explicit broadcast messages.

Performance Evaluation

We evaluated HALCONE using the MGPUSim high-fidelity simulator across 11 standard benchmarks (from Hetero-Mark, PolyBench, SHOC, and DNNMark) and a new synthetic suite, Xtreme, designed to stress-test data contention.

  • Speedup: On a 4-GPU system, HALCONE achieved a 4.6× speedup over standard RDMA and a 3× improvement over the state-of-the-art HMG protocol.
  • Efficiency: The performance overhead for maintaining coherence was remarkably low, at less than 1% for standard workloads.
  • Scalability: HALCONE demonstrated near-linear scalability up to 16 GPUs, proving that a shared-memory abstraction is viable for large-scale GPU systems if the coherence mechanism is sufficiently lightweight.
  • Traffic Reduction: Moving counters to the cache level reduced request traffic by up to 41.7% compared to prior timestamp-based methods.

Conclusion

HALCONE demonstrates that hardware-level cache coherence for multi-GPU systems is not only possible but highly efficient. By leveraging logical time and integrating specialized hardware like the TSU into the memory stack, HALCONE removes the manual data management burden from programmers while significantly improving system throughput. Future work will explore the protocol's performance under different consistency models and even higher-bandwidth interconnect architectures.

Summary updated

Abstract

While multi-GPU (MGPU) systems are extremely popular for compute-intensive workloads, several inefficiencies in the memory hierarchy and data movement result in a waste of GPU resources and difficulties in programming MGPU systems. First, due to the lack of hardware-level coherence, the MGPU programming model requires the programmer to replicate and repeatedly transfer data between the GPUs' memory. This leads to inefficient use of precious GPU memory. Second, to maintain coherency across an MGPU system, transferring data using low-bandwidth and high-latency off-chip links leads to degradation in system performance. Third, since the programmer needs to manually maintain data coherence, the programming of an MGPU system to maximize its throughput is extremely challenging. To address the above issues, we propose a novel lightweight timestamp-based coherence protocol, HALCONE, for MGPU systems and modify the memory hierarchy of the GPUs to support physically shared memory. HALCONE replaces the Compute Unit (CU) level logical time counters with cache level logical time counters to reduce coherence traffic. Furthermore, HALCONE introduces a novel timestamp storage unit (TSU) with no additional performance overhead in the main memory to perform coherence actions. Our proposed HALCONE protocol maintains the data coherence in the memory hierarchy of the MGPU with minimal performance overhead (less than 1\%). Using a set of standard MGPU benchmarks, we observe that a 4-GPU MGPU system with shared memory and HALCONE performs, on average, 4.6$\times$ and 3$\times$ better than a 4-GPU MGPU system with existing RDMA and with the recently proposed HMG coherence protocol, respectively. We demonstrate the scalability of HALCONE using different GPU counts (2, 4, 8, and 16) and different CU counts (32, 48, and 64 CUs per GPU) for 11 standard benchmarks.
BibTeX
@misc{mojumder2020halcone,
  author = {Saiful A. Mojumder and Yifan Sun and Leila Delshadtehrani and Yenai Ma and Trinayan Baruah and José Luis Abellán and John Kim and David Kaeli and Ajay Joshi},
  title = {{HALCONE : A Hardware-Level Timestamp-based Cache Coherence Scheme for Multi-GPU systems}},
  year = {2020},
  eprint = {2007.04292},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.04292},
}
Figure