Aligning Thread Scheduling with Memory Hardware Topology

13 concepts9 papers3 proposals

Modern parallel architectures, such as GPUs and multi-core chip multiprocessors (CMPs), offer massive computational throughput. However, this throughput is frequently bottlenecked by the memory subsystem. When hundreds of threads execute concurrently, they compete for shared hardware resources: L1/L2 caches, data buses, memory controllers, and physical DRAM banks.

If the hardware schedules these threads blindly—such as via a standard round-robin distribution—threads that access completely different data structures can overwrite each other's cache lines, while threads targeting the same DRAM bank can cause severe controller contention and row-buffer thrashing.

To solve this, a prominent research direction focuses on coordinating thread scheduling and memory resource allocation based on hardware topology. Instead of treating thread scheduling and memory management as independent problems, these techniques orchestrate them together. By grouping threads with shared data access patterns and mapping them to dedicated hardware structures, systems can maximize cache reuse, exploit DRAM bank parallelism, and eliminate cross-core interference.


Exploiting Spatial Cache Locality via Thread Mapping

The first layer of hardware-aware coordination happens at the core and cache level. When thread blocks (or Cooperative Thread Arrays, CTAs) are assigned to processing cores without considering their data requirements, subsequent blocks often miss in the local cache, forcing expensive round-trips to off-chip memory.

As Lifeng Liu et al. demonstrated in A Compiler-assisted locality aware CTA Mapping Scheme, mapping thread blocks with overlapping data requirements to the same processing core significantly boosts L1 cache hits. Instead of a standard round-robin distribution, assigning blocks with shared data footprints to the same core ensures that data brought into the L1 cache by an earlier block remains resident and immediately available for subsequent blocks.

Blind Round-Robin Mapping:
Thread Block 1 (Data A) -> Core 0 (Cache A)
Thread Block 2 (Data B) -> Core 1 (Cache B)
Thread Block 3 (Data A) -> Core 1 (Cache B evicted -> Cache A)  [Cache Miss!]

Locality-Aware Mapping:
Thread Block 1 (Data A) -> Core 0 (Cache A)
Thread Block 3 (Data A) -> Core 0 (Cache A retained)           [Cache Hit!]
Thread Block 2 (Data B) -> Core 1 (Cache B)

This principle of localizing memory updates is highly effective for irregular workloads. For instance, in Sparse MTTKRP Acceleration for Tensor Decomposition on GPU, Sasindu Wijeratne et al. showed that restricting memory updates to the local thread block level maximizes L1 cache throughput and streaming multiprocessor (SM) utilization. By keeping data updates strictly local, the system eliminates the need for expensive global memory synchronization and atomic operations.

This coordination is not limited to GPUs. In multi-core CPUs, Seung Woo Son et al. developed a scheme in A compiler-directed data prefetching scheme for chip multiprocessors that dynamically groups threads based on their data sharing patterns within specific program phases. By assigning dedicated helper threads to prefetch data for these coordinated groups, they mitigate harmful prefetches and reduce parallel execution latency. Even at the software emulation layer, Yufei Chen 0005 et al. showed in Scalable deterministic replay in a parallel full-system emulator that clustering multiple uncontended memory objects into a single bulk operation drastically reduces lock acquisition overhead and synchronization bottlenecks.


Maximizing DRAM Bank Parallelism and Row Locality

Moving deeper into the memory hierarchy, performance depends heavily on how requests hit the physical structures of DRAM. DRAM is organized into multiple banks, each containing a row buffer. Accessing a different row within the same bank (a row conflict) requires closing the active row and opening a new one, which incurs a massive latency penalty. Conversely, accessing different banks concurrently (bank parallelism) or hitting an already open row (row-buffer locality) hides memory latency.

To exploit this, researchers introduced the concept of Thread Batching. In TEMP: thread batch enabled memory partitioning for GPU, Mengjie Mao et al. proposed quantifying the data overlap between execution units using a stride-based metric. Thread blocks that share memory pages are grouped into a "thread batch" and dispatched to the same processing unit. This ensures that memory requests for shared data are localized, minimizing cross-unit synchronization and inter-SM interference.

DRAM Bank Partitioning via Thread Batching:
+-------------------+     +-------------------+
|  Thread Batch A   |     |  Thread Batch B   |  (Grouped by shared pages)
+-------------------+     +-------------------+
          |                             |
          v                             v
+-------------------+     +-------------------+
|   Core / SM 0     |     |   Core / SM 1     |  (Dedicated Processing)
+-------------------+     +-------------------+
          |                             |
          v                             v
+-------------------+     +-------------------+
| Physical Bank 0   |     | Physical Bank 1   |  (Isolated via Page Coloring)
+-------------------+     +-------------------+

In Thread Batching for High-performance Energy-efficient GPU Memory Design, Bing Li et al. expanded this concept by pairing thread batching with hardware page coloring. Page coloring maps specific memory pages to dedicated physical DRAM banks. Because the thread batches are formed based on these shared pages, their memory requests are naturally isolated into distinct hardware resources.

Furthermore, the authors introduced Thread Batch-Aware Scheduling, which prioritizes executing batches that target the same open memory row within a bank. By clustering row-identical requests, the system maximizes row-buffer hits, reduces memory controller contention, and minimizes the energy overhead associated with frequent row activations.

This strategy of isolating memory channels to prevent conflicts is equally vital for predictable real-time systems. For example, Ayoosh Bansal et al. demonstrated in Evaluating memory subsystem of configurable heterogeneous MPSoC that bank-aware memory allocation can be used to map the data of different CPU cores to entirely separate physical DRAM banks. This software-driven isolation eliminates row-buffer conflicts at the memory controller, reclaiming deterministic performance and predictable throughput for critical tasks.


Temporal Coordination and Channel Isolation

Beyond spatial grouping, scheduling can manipulate the timing and routing of memory requests to maximize hardware efficiency.

In Orchestrated scheduling and prefetching for GPGPUs, Adwait Jog et al. showed that memory bank utilization can be improved purely through temporal warp scheduling. By deliberately grouping and scheduling non-consecutive warps, the policy separates consecutive memory requests in time. This temporal separation allows requests to spread evenly across different memory banks rather than flooding a single bank, increasing bank-level parallelism independently of prefetching mechanisms.

When the underlying hardware channels are physically modified to prevent contention, scheduling and row locality remain the primary mechanisms to maintain performance. In Data Bus Slicing for Contention-Free Multicore Real-Time Memory Systems, Javier Jalle et al. explored slicing a wide, shared memory data bus into narrower, private data buses per core to eliminate inter-task channel contention. While narrowing the bus increases the number of sequential transfers required to move a single cache line, the authors demonstrated that this penalty is heavily mitigated by exploiting DRAM row-buffer locality. Under an open-page policy, these sequential transfers result in consecutive row hits, transforming a potential bandwidth bottleneck into a minor, easily pipelined delay.


Why It Works: Bridging Software and Hardware Layouts

Locality-aware thread mapping and scheduling work because they eliminate the mismatch between how software expresses parallelism and how hardware services memory requests.

Standard schedulers treat threads as abstract units of computation and memory as a uniform flat space. Topology-aware scheduling acknowledges the physical reality:

  1. Caches are small and local: Grouping threads by shared data ensures data is reused before it is evicted.
  2. DRAM banks are independent but internally serial: Partitioning threads into batches that map to distinct banks allows memory requests to be serviced concurrently.
  3. DRAM rows act as temporary high-speed buffers: Scheduling row-identical requests sequentially minimizes the latency and energy cost of opening and closing rows.

By orchestrating the execution order of threads to match the spatial and temporal layout of the underlying memory hardware, these techniques transform chaotic, competing memory traffic into structured, highly parallel streams.

Key Foundational Work

Orchestrated scheduling and prefetching for GPGPUs (2013) — contributed 1 concept elements to this direction.
A compiler-directed data prefetching scheme for chip multiprocessors (2009) — contributed 1 concept elements to this direction.
Scalable deterministic replay in a parallel full-system emulator (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_locality_aware_thread_block_mapping_7b481fba,
  title = {locality-aware thread block mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/locality-aware-thread-block-mapping/rdr_7b481fba6cd04029b0e5dc7392470005},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_locality_aware_thread_block_mapping_7b481fba_v1,
  title = {locality-aware thread block mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/locality-aware-thread-block-mapping/rdr_7b481fba6cd04029b0e5dc7392470005/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake