In-Training Compression for Efficient Neural Network Development

This direction is merged into another direction: one-shot pruning. Its last version stays readable and citable here.

Version 1 of 120 papers22 conceptsfrozen 2026-10-04

The traditional recipe for creating a lean, efficient neural network follows a rigid three-step pipeline: train a massive "over-parameterized" model to full convergence, prune away the least important weights, and then fine-tune the remaining structure to recover lost accuracy. While effective, this process is computationally wasteful. It requires researchers to pay the full "tax" of training a giant model just to throw most of it away.

A growing body of research suggests a more integrated approach: In-Training Compression. By identifying and removing redundancy while the model is still learning, we can treat sparsity not as a post-processing step, but as a fundamental part of the training dynamics.

Breaking the Three-Stage Pipeline

The core intuition behind in-training compression is that a model does not need its full capacity for the entire duration of training. Early in the optimization process, the network begins to establish which pathways are critical and which are redundant.

Guoliang Kang and Yang He demonstrated with Soft Filter Pruning that models can be compressed from scratch without a pre-trained baseline. Instead of waiting for convergence, the pruning process is integrated directly into the standard training cycle. This eliminates the need for a resource-intensive "teacher" model, making the development of efficient architectures accessible even to those with limited hardware.

Efficiency as a Training Objective

When we prune during training, the primary benefit shifts from inference speed to training efficiency. If a model becomes 50% smaller halfway through training, the remaining 50% of the training epochs require significantly fewer floating-point operations (FLOPs).

For instance, Deepak Ghimire and Jaesik Park introduced OCSPruner, which performs structured pruning in a single training cycle. By identifying the optimal sub-network architecture early on, the total training duration can actually be shorter than training the original unpruned baseline. Similarly, Rahul Duggal and Jimeng Sun showed that their CUP method could save 14 hours of training time for a ResNet-50 by clustering similar filters and removing them early.

Identifying Redundancy Early

The challenge of in-training compression is knowing what to prune before the weights have fully matured. Researchers have developed several sophisticated "look-ahead" mechanisms to solve this:

  • Predictive Modeling: Mohit Rajpal and Bryan Kian Hsiang Low proposed Bayesian Early Pruning, using Gaussian Processes to predict which neurons will eventually become "efficacy-less" based on early training signals.
  • Control Theory: In the realm of State Space Models (SSMs), Makram Chahine and Daniela Rus developed COMPRESSM. This framework uses balanced truncation to identify redundant hidden state dimensions during the training warm-up, exploiting the fact that the "importance spectrum" of a model stabilizes very quickly.
  • Regularization: Jose M. Alvarez and Mathieu Salzmann took a different path with Compression-aware Training, adding low-rank regularizers (like the nuclear norm) to the loss function. This forces the network to learn representations that are inherently compressible from the start.

Dynamic Resource Allocation

In-training compression isn't just about deleting weights; it's about reallocating resources. In distributed training environments, shrinking the model on the fly allows for "elastic" scaling.

Chaoyang He and Salman Avestimehr explored this with PipeTransformer. As layers converge, they are "frozen" and removed from the active training pipeline. The freed-up GPU memory and compute are then used to increase data parallelism (processing more samples at once), resulting in a nearly 3x speedup in training time.

Pruning as a Regularizer

Perhaps the most surprising finding is that pruning during training can actually improve the final model's quality. By removing noise and redundant parameters, the process acts as a form of regularization.

In the context of 3D Gaussian Splatting, Alex Hanson and Tom Goldstein found that Speedy-Splat improved training efficiency by reducing the number of primitives the GPU had to manage. Similarly, Le Thanh Nguyen-Meidine and Eric Granger showed that Progressive Gradient Pruning helps models discard domain-specific features in real-time, improving generalization for tasks like domain adaptation.

By treating compression as a first-class citizen during the training process, we move away from the "waste-then-recover" paradigm toward a more biological model of learning—one where the brain prunes its connections as it matures, becoming more efficient and specialized through the very act of learning.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_in_training_compression_for_efficient_ne_60a4a738,
  title = {In-Training Compression for Efficient Neural Network Development},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/in-training-compression-for-efficient-neural-network-development/rdr_60a4a7380699457694601821eb4b58eb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_in_training_compression_for_efficient_ne_60a4a738_v1,
  title = {In-Training Compression for Efficient Neural Network Development},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/in-training-compression-for-efficient-neural-network-development/rdr_60a4a7380699457694601821eb4b58eb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}