Lumos: Democratizing SciML Workflows with L0-Regularized Learning for Unified Feature and Parameter Adaptation

Shouwei Gao, Xu Zheng, Dongsheng Luo, Sheng Di, Wenqian Dong

TL;DR

LUMOS is an end-to-end framework that unifies input feature selection and model pruning for Scientific Machine Learning (SciML) into a single, automated training step. By leveraging differentiable $L_0$-regularized learning, it slashes model parameters by an average of 71.45% and accelerates inference by 6.4x without sacrificing predictive accuracy.

What Problem Does This Paper Address?

Scientific Machine Learning (SciML) has revolutionized fields like climate science and drug discovery by embedding neural networks into traditional high-performance computing (HPC) workflows. However, building effective SciML models remains a dark art. Domain scientists face two major bottlenecks:

  1. Feature Efficiency: Identifying which high-dimensional physical features actually drive the system's behavior.
  2. Model Efficiency: Determining the optimal size of the neural network architecture to avoid massive computational redundancy.

Traditionally, solving these problems requires exhaustive, brute-force grid searches or ad-hoc heuristics. This challenge is present whether training a model from scratch or fine-tuning a massive foundation model for a downstream task. Existing compression and feature selection methods treat these problems separately, often resulting in high computational overhead or models that require specialized hardware to achieve actual speedups.

Two primary scenarios in SciML pipelines facing feature and parameter bottlenecks.

Methods

LUMOS introduces a unified framework that performs structured pruning and feature selection simultaneously during standard training or fine-tuning.

Overall workflow of the LUMOS framework.

The core technical innovations include:

  • Semi-Stochastic Gating: LUMOS attaches learnable gating parameters to both input features and hidden layer neurons. These gates dictate whether a feature or neuron remains active.
  • Hard-Concrete Relaxation: Because choosing to "keep or drop" a neuron is a discrete (non-differentiable) operation, the authors use a reparameterization trick called hard-concrete relaxation. This smooths the discrete $L_0$ penalty, allowing the gates to be optimized using standard gradient descent (like Adam or SGD) alongside the model's weights.
  • Structural Consistency Mapping: Pruning rows or columns from weight matrices can break a network if subsequent layers expect the original dimensions. LUMOS includes a suite of alignment tools (like a Layer Consistency Mapper) that automatically tracks and reconciles architectural changes across diverse layer types—including Fully Connected, Convolutional, Graph (GCN/GIN), and Attention layers—ensuring the final pruned model is immediately executable on standard hardware.
Example of how LUMOS handles structured transitions and tensor alignment between pruned Convolutional and Fully Connected layers.

Datasets

The authors evaluated LUMOS across 13 diverse SciML workloads spanning multiple scientific domains and input modalities (images, vectors, matrices, and graphs):

  • Fluid Dynamics: CFD, Fluid (Surrogate flux and force computations)
  • Cosmology: CosmoFlow (Universe parameter regression)
  • Material Sciences: EM-DN (Denoising), DMS (Image classification), STEM-DL (Diffraction patterns)
  • Molecular/Chemical Sciences: PureMD (Molecular dynamics), Molhiv (Molecular properties)
  • Atmospheric Science: SLSTR (Image classification)
  • Other Domains: Synthetic (Math formulation), Optical (Equipment damage detection), PPA (Biological taxonomy), and BrainTumor (Medical tumor detection)

Key Insights and Ideas

  • The Power of Co-Optimization: Feature selection and parameter pruning are deeply intertwined. By optimizing them jointly under a single loss function, LUMOS allows the model to dynamically reallocate its capacity to the most informative features during training.
  • Hardware-Agnostic Speedups: Unlike unstructured pruning (which leaves sparse "zero" weights that standard hardware cannot accelerate without special libraries), LUMOS enforces structured pruning. It removes entire channels and features, resulting in smaller, dense matrices that deliver real-world speedups on any standard CPU or GPU.
  • Regularization as an Anti-Overfitting Tool: In highly over-parameterized scientific models, removing redundant weights actually acts as a strong regularizer, occasionally helping LUMOS achieve higher predictive accuracy than the unpruned baseline.

Key Results

LUMOS was benchmarked against state-of-the-art pruning frameworks (such as OTO, L1-regularization, NEURAL, and DepGraph) on NVIDIA A100 GPUs and Intel Xeon CPUs:

  • Drastic Size & FLOP Reductions: LUMOS achieved an average 71.45% reduction in active parameters and cut floating-point operations (FLOPs) by 69.6% on average.
  • Substantial Speedups: Inference accelerated by an average of 6.4x on GPUs (reaching up to 51.6x for the CFD workload) and 7.6x on CPUs.
  • Resource Efficiency: Peak memory usage was reduced by up to 96.7% for compact models, and overall energy consumption dropped by an average of 50.7%.
  • Scalability: Distributed Data Parallel (DDP) training tests on up to 8 GPUs confirmed that LUMOS adds negligible overhead (~3% additional gate parameters), scaling just as efficiently as vanilla training pipelines.

Limitations and Future Work

While LUMOS provides an excellent hardware-agnostic framework for sparsification, the authors note that hardware-specific optimizations, such as mixed-precision quantization (e.g., INT8/FP16 execution on Tensor Cores), were left out of scope. Integrating LUMOS with low-bit quantization libraries represents a promising future direction to squeeze even more performance out of edge devices and HPC clusters.

Summary updated

Abstract

The rapid growth of scientific machine learning (SciML) has accelerated discovery across diverse domains, yet designing effective SciML models remains a challenging task. In practice, building such models often requires substantial prior knowledge and manual expertise, particularly in determining which input features to use and how large the model should be. We introduce LUMOS, an end-to-end framework based on L0-regularized learning that unifies feature selection and model pruning to democratize SciML model design. By employing semi-stochastic gating and reparameterization techniques, LUMOS dynamically selects informative features and prunes redundant parameters during training, reducing the reliance on manual tuning while maintaining predictive accuracy. We evaluate LUMOS across 13 diverse SciML workloads, including cosmology and molecular sciences, and demonstrate its effectiveness and generalizability. Experiments on 13 SciML models show that LUMOS achieves 71.45% parameter reduction and a 6.4x inference speedup on average. Furthermore, Distributed Data Parallel (DDP) training on up to eight GPUs confirms the scalability of
BibTeX
@inproceedings{gao2026lumos,
  author = {Shouwei Gao and Xu Zheng and Dongsheng Luo and Sheng Di and Wenqian Dong},
  title = {{Lumos: Democratizing SciML Workflows with L0-Regularized Learning for Unified Feature and Parameter Adaptation}},
  booktitle = {IPDPS},
  year = {2026},
  doi = {10.1109/ipdps65963.2026.00061},
}

Loading related content...

Figure