TL;DR
This paper tackles the performance bottleneck in high-order finite element simulations for linear elasticity. By optimizing the "matrix-free" operator evaluation in the MFEM library using a combination of techniques like sum factorization and kernel fusion, the researchers significantly accelerate computations, shifting the optimal performance sweet spot from low polynomial degrees (p≈2) to higher ones (p≥6). This advancement is crucial for enabling more accurate and efficient simulations in fields like structural mechanics.
What Problem Does This Paper Address?
High-order finite element methods (FEM) offer a way to achieve high accuracy with fewer computational elements compared to lower-order methods. This is particularly attractive for simulating complex physical phenomena like linear elasticity, which is fundamental to understanding material behavior in engineering. However, a major hurdle in realizing the full potential of high-order FEM is the computational cost associated with solving the resulting large linear systems. Iterative solvers, commonly used for these systems, repeatedly perform matrix-vector products (or their equivalent "discrete operator" evaluations).
The MFEM library, a popular tool for FEM, uses a "matrix-free" approach called Partial Assembly (PA) for these operations. While this avoids explicitly forming massive matrices, the native PA implementation for linear elasticity in MFEM had a significant performance bottleneck: an $O((p+1)^6)$ contraction in the element operator. This meant that performance peaked at low polynomial degrees (around $p \approx 2$), limiting the benefits of higher-order elements. This paper aims to close this gap by optimizing MFEM's PA path to achieve high performance at higher polynomial degrees.
Methods
The core of this work is the optimization of MFEM's native linear-elasticity Partial Assembly (PA) operator. The authors introduce a four-stage optimization strategy:
- Macro-kernel Fusion: This technique merges multiple computational steps (gradient interpolation, stress evaluation, and operator action) into a single, element-local kernel. This eliminates costly data movement between main memory and the CPU, improving data locality.
- Voigt Notation: Linear elasticity involves symmetric stress and strain tensors. Voigt notation is a way to represent these symmetric tensors using a reduced number of components (6 instead of 9 for 3D), which saves storage and reduces redundant computations.
- Sum Factorization: This is a key optimization for tensor-product elements. It decomposes complex, high-dimensional tensor contractions into a sequence of simpler, one-dimensional contractions. This significantly reduces the computational complexity from $O((p+1)^6)$ to $O((p+1)^4)$ for the element operator.
- Slice-Wise Loop Organization: This optimization focuses on managing the data flow within the element-local kernel. By organizing computations in slices and carefully managing the lifetime of intermediate data buffers, it ensures that the working set fits within the CPU's L1/L2 caches, further improving performance.
These optimizations are integrated into MFEM's existing PA path, creating a new operator called "PAop." The performance of PAop is then evaluated within a Geometric Multigrid (GMG) preconditioned Conjugate Gradient (PCG) solver.
Datasets
Not discussed in the paper. The paper uses a standard two-material cantilever beam benchmark problem from the MFEM library (data/beam-hex.mesh) for evaluation.
Key Insights and Ideas
- Shifting the "Sweet Spot": The most significant insight is that by applying these specific optimizations, the performance sweet spot for the operator-throughput can be dramatically shifted from low polynomial degrees ($p \approx 2$) to much higher ones ($p \ge 6$). This directly unlocks the potential of high-order FEM for improved accuracy without prohibitive computational cost.
- The Power of Data Locality and Reduced Arithmetic: The paper clearly demonstrates how a combination of reducing redundant computations (Voigt notation, sum factorization) and aggressively managing data movement (macro-kernel fusion, slice-wise loops) leads to substantial speedups. It's not just about raw floating-point operations, but how efficiently those operations are performed with respect to memory access.
- Hardware-Specific Optimizations Pay Off: The success of these optimizations is shown to be reproducible across different architectures (x86 and ARM), indicating that the underlying principles are broadly applicable. However, the specific performance gains are tied to the microarchitectural features of the target CPUs.
- Matrix-Free is Key for High-Order: The paper reinforces the idea that for high-order FEM, matrix-free approaches are essential to manage memory footprint and computational complexity, especially when compared to full assembly methods.
Key Results
- Significant Speedups: The optimized PAop operator achieved impressive speedups:
- Kernel Speedup: 7-83x on AMD EPYC 7713.
- End-to-End Solver Speedup: 3.6-16.8x on AMD EPYC 7713.
- Sweet Spot Shift: The operator-throughput sweet spot moved from $p \approx 2$ to $p \ge 6$.
- Cross-Platform Reproducibility: The same performance trend was observed on Huawei Kunpeng 920 (ARMv8.2).
- Memory Efficiency: PAop maintained a relatively flat memory footprint across polynomial degrees, unlike Full Assembly which grew significantly.
- Ablation Study: The cumulative ablation study clearly showed that sum factorization provided the largest single step in kernel speedup, followed by macro-kernel fusion, with Voigt notation and slice-wise loops providing further improvements.
Limitations and Future Work
- Limitations:
- The current work focuses on affine tensor-product hexahedral meshes. Problems with geometric singularities, material discontinuities, thin features, or non-affine meshes might still favor low-order adaptive discretizations.
- The performance gains are tied to specific hardware architectures, though the principles are general.
- Future Work:
- Generalization to other PA operators: Extend the PAop dataflow to other vector-valued PA operators with symmetric constitutive matrices.
- Portability to Accelerators: Adapt the optimized tensor-product operator design to libCEED-style backends for performance portability across CPUs and accelerators.
- Richer Material Models and Geometries: Extend the approach to more complex material models and non-affine geometries, where the balance between stored data, recomputation, and backend portability becomes more critical.
Summary updated