For decades, computer systems operated under a comfortable division of labor. Hardware designers focused on making processors faster and storage denser, while software engineers built abstractions—like operating systems, databases, and file formats—predicated on a stable, predictable hardware model. This traditional model assumed a single-core CPU, a sharp divide between volatile RAM and slow, non-volatile disk storage, and a "shared-nothing" distributed architecture designed to minimize network communication.
However, physical silicon limits and power constraints disrupted this trajectory, forcing a shift toward massive parallelism and hardware heterogeneity (Benini). Today's infrastructure features many-core CPUs, programmable GPUs, FPGAs, Non-Volatile Memory (NVM), and ultra-fast Remote Direct Memory Access (RDMA) networks.
The core challenge is that while hardware has fundamentally transformed, our software abstractions remain haunted by the ghosts of legacy assumptions. This architectural impedance mismatch creates a widening gap between theoretical peak hardware performance and actual software efficiency (Xiao and Yuan).
The Anatomy of the Mismatch
To understand why modern software struggles on modern hardware, we must look at where traditional abstractions break down across storage, compute, and networking.
Storage Assumptions vs. Fast Persistent Media
Traditional database engines spend significant computational overhead on buffer management and page-oriented structures. This design was highly intentional: it shielded the application from the massive latency of spinning hard disks by keeping hot data in RAM (Jagadish et al.).
When these legacy architectures are deployed on modern hardware like NVM or high-speed NVMe SSDs, the assumptions flip. The storage media is no longer the primary bottleneck; instead, the software's own coordination and buffer management layers become the chokepoint (Pan et al.).
Similarly, popular columnar storage formats like Parquet were optimized for the high-latency storage and heavy block compression constraints of the Hadoop era. When paired with modern high-bandwidth NVMe SSDs and parallel GPUs, these formats introduce severe CPU decompression overhead that prevents the system from saturating the available hardware bandwidth (Zeng et al.). Even in cloud environments, developers frequently apply local-hardware heuristics—such as aggressively prioritizing sequential writes over random writes—which can inadvertently degrade performance on cloud-native elastic SSDs (Yang and Wang).
Compute Assumptions vs. Massively Parallel Execution
In the era of single-core processors, software execution was fundamentally sequential. Relational database query engines, for example, traditionally relied on virtual machine-based execution models or row-by-row iterators (the Volcano model). These abstractions fail to exploit modern CPU capabilities like instruction pipelining, cache hierarchies, and Single Instruction Multiple Data (SIMD) vectorization (Prammer et al., Hu).
When sequential software models are naively ported to throughput-oriented processors like GPUs, the mismatch deepens. GPUs are engineered for uniform, highly parallel operations on large data arrays. They are structurally ill-suited for workloads like Online Transaction Processing (OLTP), which inherently demand diverse, small-scale, and random data updates (He). Without a fundamental redesign of how data flows through the processor, distributed software fails to translate the raw parallel processing power of modern accelerated clusters into actual application speedups (Selvitopi et al.).
Networking Assumptions vs. Fabric-Scale Interconnects
In traditional distributed systems, the network was considered the slowest component. Systems were designed around a "shared-nothing" paradigm, optimizing individual operators locally to avoid shuffling data across nodes.
Modern data centers, however, utilize high-speed RDMA networks and disaggregated hardware designs that decouple compute, memory, and storage into specialized pools (Xu). Sticking to a legacy shared-nothing execution model in this environment prevents software from leveraging direct memory access and flexible, decoupled scaling (Salama et al.). Furthermore, while modern data center hardware is explicitly engineered for multi-tenancy and hardware-level resource sharing, software stacks are still largely optimized for isolated execution, resulting in unpredictable performance and underutilized infrastructure (Ma).
Shifting to Hardware-Aware Design
Resolving this mismatch requires moving away from a reactionary software model—where applications are retrofitted onto whatever hardware happens to exist—toward a philosophy of hardware-software co-design (Roscoe et al.). This involves rewriting core abstractions, automating data structure design, and manually optimizing software for emerging instruction set architectures.
Legacy Model: [ Software Abstraction ] -> [ Rigid OS/Kernel ] -> [ Reactionary Hardware Fit ]
| (Impedance Mismatch)
Co-Designed Model: [ Hardware-Aware App ] <====== Co-Design ======> [ Heterogeneous Hardware ]
Inverting Core Abstractions
Rather than forcing modern parallel hardware to mimic a legacy serial computer, co-designed systems rebuild foundational software layers from scratch. For example, Kepner et al. proposed TabulaROSA, an architecture that reimagines the operating system kernel not as a traditional monolithic scheduler, but as a high-performance database management system defined through associative array algebra. This allows the OS to natively map process management tasks directly onto massively parallel, heterogeneous compute engines like GPUs and FPGAs without traditional kernel bottlenecks.
Automated Data Structure Synthesis
Because hardware architectures are evolving rapidly with new memory hierarchies and specialized accelerators, manually redesigning data structures by hand has become a development bottleneck. To bridge this gap, Idreos et al. introduced a paradigm shift: breaking data structures down into fundamental algorithmic "atoms" and using automated engines to synthesize custom, hardware-optimized data structures tailored to the specific physical characteristics of the underlying hardware.
Architecture-Specific Optimization for Emerging ISAs
When open-source and emerging hardware paradigms like RISC-V enter the high-performance computing and data center landscapes, they face an immediate software ecosystem gap. Standard foundational libraries and general-purpose compilers frequently fail to exploit specialized hardware features, such as scalable vector extensions or custom modular instructions (Wu et al.).
As demonstrated by Venieri et al. in the Monte Cimone v2 project, unlocking the theoretical peak performance of emerging hardware requires manual, architecture-specific optimization of mathematical and system libraries. Software must be explicitly tailored to the vector extensions and memory layouts of the physical silicon to prevent the ecosystem from lagging behind the hardware's capabilities.
Ultimately, closing the hardware-software gap means accepting that hardware is no longer a transparent execution layer. Whether building databases, operating systems, or machine learning pipelines, modern software must be designed with an explicit awareness of the underlying physical architecture—embracing parallelism, memory hierarchies, and hardware-software co-design as first-class principles.
Key Foundational Work
Dalí: A High Performance Main Memory Storage Manager (1994) — contributed 1 concept elements to this direction.