TL;DR
This paper offers a retrospective and forward-looking perspective on massively parallel computing (MPP) from the viewpoint of Sandia National Laboratories. It highlights the evolution of MPP hardware, system software, and algorithms, emphasizing the practical challenges and successes encountered in applying these systems to large-scale scientific and engineering simulations. The authors conclude that while progress has been slower than anticipated, MPP is realizing its potential, with future growth driven by commodity components and improved software.
What Problem Does This Paper Address?
The paper addresses the evolving landscape of massively parallel computing (MPP), a field that promised revolutionary advancements in scientific and engineering simulations but has faced a slower-than-expected realization of its potential. The authors, drawing from their extensive experience at Sandia National Laboratories, aim to provide a grounded perspective on the state of MPP, examining the interplay between hardware, system software, and algorithms. They explore why MPP hasn't fully met initial expectations and identify the key factors that will shape its future, particularly for high-end scientific computing.
Methods
The paper doesn't present a single novel methodology or algorithm. Instead, it offers a comprehensive review and analysis of existing trends and technologies in MPP. This includes:
- Hardware Classification: Discussing the evolution and comparison of SIMD vs. MIMD architectures and shared vs. distributed memory systems, highlighting the eventual dominance of distributed memory MIMD systems.
- System Software Analysis: Examining the challenges and advancements in operating systems, compilers, and debuggers for parallel environments, with a focus on the "partitioned system design" approach to manage complexity.
- Algorithmic Considerations: Emphasizing the importance of "data locality" and "algorithmic scalability" over raw parallel scalability, and the trend of parallelizing existing serial algorithms rather than developing entirely new ones.
- Industry Trends: Analyzing the shift from specialized MPP systems to commodity-off-the-shelf (COTS) and Shared Memory Parallel (SMP) systems, and the impact of industry adoption on MPP development.
Datasets
Not discussed in the paper. The paper focuses on the general trends and technologies in parallel computing rather than specific experimental evaluations on particular datasets.
Key Insights and Ideas
- The "Holy Grail" of Shared Memory on Distributed Systems is Elusive: While efforts to create a unified memory image on distributed systems are ongoing, explicit message passing remains the most effective way to achieve massive parallelism.
- Hardware is Only One Piece of the Puzzle: The success of MPP hinges equally on system software and algorithms. Historically, system software has lagged behind hardware advancements.
- Algorithmic Scalability Trumps Parallel Scalability: The most effective parallel algorithms are often efficient serial algorithms that can be effectively parallelized. Simply making an algorithm run on many processors is less important than ensuring the algorithm itself is efficient.
- Data Locality is Paramount: Minimizing inter-processor communication by keeping data close to the processors that need it is the most critical factor for achieving high performance and scalability on parallel machines.
- The Rise of Commodity Components: The trend towards using off-the-shelf hardware and open-source software (COTS) has made parallel computing more accessible and cost-effective, even if it requires more integration effort.
- Partitioned System Design is Key for Software: Breaking down system software into smaller, manageable partitions (e.g., service, I/O, compute) simplifies development and maintenance for complex parallel systems.
- Debugging Parallel Codes is Extremely Difficult: The challenges of representing data from thousands of threads of control and avoiding interference with program timing make debugging a significant hurdle.
Key Results
- MPP Hardware Evolution: The paper traces the evolution from early vector processors to massively parallel machines like the nCUBE, CM2, CM5, Intel Paragon, and Cray T3E/T3D, highlighting their architectures and performance milestones.
- Performance Gains: The paper notes that current MPP systems achieve peak computational rates exceeding two teraflops, significantly outperforming single-processor computers.
- Industry Shift: The paper documents the decline of specialized MPP systems in the industrial marketplace and the corresponding rise of SMP and COTS systems, which are driving broader adoption of parallel computing.
- Application Maturity: Many scientific domains, such as dense linear algebra and finite difference codes, have mature parallel algorithms. More complex areas like particle simulations and unstructured grid methods are also maturing.
- Scalability Achievements: Sandia researchers demonstrated significant speedups on machines like the nCUBE 10, achieving scaled speedups over 1000 for scientific applications. The Intel Tflops machine achieved a sustained trillion floating-point operations per second.
Limitations and Future Work
- System Software Lag: The paper acknowledges that system software has consistently lagged behind hardware and algorithm development, often being adaptations of workstation software. Future work needs to focus on improving OS, compilers, and debuggers for parallel environments.
- Debugging Complexity: The authors highlight the extreme difficulty of debugging massively parallel codes and suggest that fundamental research is needed to develop tools that can assist computers in the debugging process, rather than just scaling up serial debugging techniques.
- Industry Adoption Challenges: While COTS and SMP systems are making parallel computing more accessible, integrating them into a cohesive, single-system experience remains a challenge.
- Future of High-End Architectures: The authors express concern about the industry trend towards clusters of SMPs with weak interconnects, fearing it could be a setback for high-end computing, which has historically benefited from well-balanced architectures. They are more optimistic about "build-your-own" parallel machines from commodity components.
- Continued Research: The paper calls for continued commitment from the high-performance community to provide access to hardware for system software research and development, anticipating significant payoffs. Future work will also involve addressing the challenges of heterogeneous systems and new interconnection network hardware.
Summary updated