TL;DR
LUMOS is an end-to-end framework that unifies input feature selection and model pruning for Scientific Machine Learning (SciML) into a single, automated training step. By leveraging differentiable $L_0$-regularized learning, it slashes model parameters by an average of 71.45% and accelerates inference by 6.4x without sacrificing predictive accuracy.
What Problem Does This Paper Address?
Scientific Machine Learning (SciML) has revolutionized fields like climate science and drug discovery by embedding neural networks into traditional high-performance computing (HPC) workflows. However, building effective SciML models remains a dark art. Domain scientists face two major bottlenecks:
- Feature Efficiency: Identifying which high-dimensional physical features actually drive the system's behavior.
- Model Efficiency: Determining the optimal size of the neural network architecture to avoid massive computational redundancy.
Traditionally, solving these problems requires exhaustive, brute-force grid searches or ad-hoc heuristics. This challenge is present whether training a model from scratch or fine-tuning a massive foundation model for a downstream task. Existing compression and feature selection methods treat these problems separately, often resulting in high computational overhead or models that require specialized hardware to achieve actual speedups.
Methods
LUMOS introduces a unified framework that performs structured pruning and feature selection simultaneously during standard training or fine-tuning.
The core technical innovations include:
- Semi-Stochastic Gating: LUMOS attaches learnable gating parameters to both input features and hidden layer neurons. These gates dictate whether a feature or neuron remains active.
- Hard-Concrete Relaxation: Because choosing to "keep or drop" a neuron is a discrete (non-differentiable) operation, the authors use a reparameterization trick called hard-concrete relaxation. This smooths the discrete $L_0$ penalty, allowing the gates to be optimized using standard gradient descent (like Adam or SGD) alongside the model's weights.
- Structural Consistency Mapping: Pruning rows or columns from weight matrices can break a network if subsequent layers expect the original dimensions. LUMOS includes a suite of alignment tools (like a Layer Consistency Mapper) that automatically tracks and reconciles architectural changes across diverse layer types—including Fully Connected, Convolutional, Graph (GCN/GIN), and Attention layers—ensuring the final pruned model is immediately executable on standard hardware.
Datasets
The authors evaluated LUMOS across 13 diverse SciML workloads spanning multiple scientific domains and input modalities (images, vectors, matrices, and graphs):
- Fluid Dynamics: CFD, Fluid (Surrogate flux and force computations)
- Cosmology: CosmoFlow (Universe parameter regression)
- Material Sciences: EM-DN (Denoising), DMS (Image classification), STEM-DL (Diffraction patterns)
- Molecular/Chemical Sciences: PureMD (Molecular dynamics), Molhiv (Molecular properties)
- Atmospheric Science: SLSTR (Image classification)
- Other Domains: Synthetic (Math formulation), Optical (Equipment damage detection), PPA (Biological taxonomy), and BrainTumor (Medical tumor detection)
Key Insights and Ideas
- The Power of Co-Optimization: Feature selection and parameter pruning are deeply intertwined. By optimizing them jointly under a single loss function, LUMOS allows the model to dynamically reallocate its capacity to the most informative features during training.
- Hardware-Agnostic Speedups: Unlike unstructured pruning (which leaves sparse "zero" weights that standard hardware cannot accelerate without special libraries), LUMOS enforces structured pruning. It removes entire channels and features, resulting in smaller, dense matrices that deliver real-world speedups on any standard CPU or GPU.
- Regularization as an Anti-Overfitting Tool: In highly over-parameterized scientific models, removing redundant weights actually acts as a strong regularizer, occasionally helping LUMOS achieve higher predictive accuracy than the unpruned baseline.
Key Results
LUMOS was benchmarked against state-of-the-art pruning frameworks (such as OTO, L1-regularization, NEURAL, and DepGraph) on NVIDIA A100 GPUs and Intel Xeon CPUs:
- Drastic Size & FLOP Reductions: LUMOS achieved an average 71.45% reduction in active parameters and cut floating-point operations (FLOPs) by 69.6% on average.
- Substantial Speedups: Inference accelerated by an average of 6.4x on GPUs (reaching up to 51.6x for the CFD workload) and 7.6x on CPUs.
- Resource Efficiency: Peak memory usage was reduced by up to 96.7% for compact models, and overall energy consumption dropped by an average of 50.7%.
- Scalability: Distributed Data Parallel (DDP) training tests on up to 8 GPUs confirmed that LUMOS adds negligible overhead (~3% additional gate parameters), scaling just as efficiently as vanilla training pipelines.
Limitations and Future Work
While LUMOS provides an excellent hardware-agnostic framework for sparsification, the authors note that hardware-specific optimizations, such as mixed-precision quantization (e.g., INT8/FP16 execution on Tensor Cores), were left out of scope. Integrating LUMOS with low-bit quantization libraries represents a promising future direction to squeeze even more performance out of edge devices and HPC clusters.
Summary updated