Hot-and-Cold: Using Criticality in the Design of Energy-Efficient Caches

Rajeev Balasubramonian, Viji Srinivasan, Sandhya Dwarkadas, Alper BuyuktosunogluPacs 2003 · DOI

TL;DR

This paper proposes a "hot-and-cold" cache design that partitions L1 data cache banks into high-performance ("hot") and energy-efficient ("cold") banks. By accurately predicting which memory accesses are on the critical path of program execution, the system can steer critical accesses to the hot bank and non-critical accesses to the cold bank, achieving significant energy savings with minimal performance degradation.

What Problem Does This Paper Address?

As processor technology advances, power and energy consumption are becoming critical design constraints, especially for on-chip caches which consume a significant portion of a chip's energy. While techniques exist to reduce cache energy, they often come at the cost of increased access latency. This paper addresses the challenge of reducing both dynamic and leakage energy in L1 data caches without significantly impacting performance. The core idea is to exploit the observation that not all instructions and their associated data accesses are equally critical to program execution time. Instructions on the "critical path" directly impact performance if delayed, while "non-critical" instructions have some latency tolerance.

Methods

The paper introduces a "hot-and-cold" banked cache organization. The L1 data cache is split into two banks:

  • Hot Bank: Designed for high performance (low latency).
  • Cold Bank: Designed for energy efficiency (lower performance, higher latency).

The key to this approach is criticality prediction:

  1. Instruction Criticality: The paper uses an "Oldest-N" heuristic to predict instruction criticality. Instructions among the oldest N in the issue queue are considered critical. This is hardware-friendly as it doesn't require complex tables.
  2. Data Block Classification: Data blocks are classified as critical or non-critical based on the historical access patterns of critical vs. non-critical instructions. An n-bit up/down counter tracks the ratio of critical accesses to a cache block.
  3. Data Placement: When a cache block is fetched from L2, it's placed into either the hot or cold bank based on its criticality classification. A "placement predictor" (a small structure) stores this criticality bit for L2 cache lines.
  4. Load/Store Steering: A "bank predictor," indexed by the instruction's program counter (PC), directs memory accesses to the appropriate hot or cold bank. This predictor is dynamically updated based on past access behavior.

The cold bank can leverage various architectural and circuit-level power-saving techniques (e.g., higher Vt, transistor sizing, gated-ground SRAM cells) that would be detrimental to performance if applied to a conventional cache.

Datasets

The evaluation uses 10 programs from the SPEC2k-Int benchmark suite with reference input datasets. The simulation parameters are detailed in Table 1, and base statistics (IPC, miss rates) for the benchmarks are in Table 2.

Key Insights and Ideas

  • Criticality is Consistent: Both instructions and data exhibit a high degree of criticality consistency across program executions. This consistency is crucial for the effectiveness of the proposed classification and partitioning scheme.
  • Latency Tolerance of Non-Critical Paths: Non-critical instructions and their data accesses can tolerate significantly increased latency without a major performance hit. The paper quantifies this as only a 0.8% performance degradation for each additional cycle in the cold cache access time.
  • Bank Contention is the Main Bottleneck: While the hot-and-cold design effectively separates critical and non-critical accesses, it can lead to increased contention for cache ports if accesses to the hot and cold banks are imbalanced. This is a more significant performance limiter than the cold cache's increased latency.
  • Dynamic Tuning of Criticality Threshold (N): Dynamically adjusting the "Oldest-N" threshold at runtime to balance accesses between the hot and cold banks is essential to mitigate port contention and improve performance.
  • Energy Savings are Proportional to Cold Bank Usage: The overall energy savings are directly related to the fraction of accesses directed to the energy-efficient cold bank.

Key Results

  • Energy Savings: The hot-and-cold cache design achieves an average L1 data cache energy reduction of 37% when the cold bank consumes 0.2 times the energy of the hot bank. Savings range from 40% to 80% depending on the cold bank's efficiency.
  • Performance Impact:
    • With perfect steering and a 2-cycle cold cache latency, performance degradation is 1.8% due to bank contention.
  • Increasing cold cache latency to 4 cycles results in only a 1.6% IPC loss, whereas a 4-cycle latency on a word-interleaved cache results in a 5.7% IPC penalty.
  • When cold cache latency is increased to 6 cycles, the hot-and-cold cache outperforms a 6-cycle word-interleaved base case by 5.7%.
  • Bank predictor mispredictions add an additional 1% IPC degradation.
  • Data Block Classification: Approximately 46% of data cache blocks are accessed almost exclusively by critical loads (90-100% critical accesses).
  • Load Criticality Consistency: Over 88% of dynamic loads show consistent criticality behavior across invocations.
  • Bank Predictor Accuracy: A 4Kx1 bit bank predictor achieves over 90% accuracy.

Limitations and Future Work

  • Bank Contention: The primary limitation is increased bank contention due to imbalanced access patterns to the hot and cold banks. This can lead to significant performance degradation (2.7% compared to a word-interleaved cache).
  • Bank Steering Mispredictions: Incorrect steering of accesses to the wrong bank incurs additional latency and energy penalties.
  • Asymmetric Sizes/Organizations: The paper suggests exploring asymmetric sizes and organizations for the hot and cold banks as future work.
  • Improving Bank Prediction: Further improvements in bank prediction accuracy are needed to mitigate performance losses.
  • Defining Energy Consumption Ratio: The authors are working with circuit designers to better define the energy consumption ratio between hot and cold cache banks.
  • Microarchitectural Inefficiency: A portion of memory operations (26%) are accessed by both critical and non-critical instructions, leading to unavoidable inefficiencies in either performance or energy.

Summary updated

BibTeX
@inproceedings{balasubramonian2003hot,
  author = {Rajeev Balasubramonian and Viji Srinivasan and Sandhya Dwarkadas and Alper Buyuktosunoglu},
  title = {{Hot-and-Cold: Using Criticality in the Design of Energy-Efficient Caches}},
  booktitle = {Pacs},
  year = {2003},
  doi = {10.1007/978-3-540-28641-7\_13},
}
Figure