Regularization Through Structural Parameter Sharing

26 concepts20 papers

When designing deep neural networks, parameter reduction is often viewed strictly through the lens of engineering efficiency—a way to fit massive models onto commodity hardware. However, a broader look at machine learning architectures reveals a deeper conceptual truth: parameter sharing is a highly effective form of structural regularization. By forcing different components of a model to reuse the same underlying weights, we impose a strict mathematical constraint that prevents the model from overfitting to noise, particularly in sparse data regimes or long-tailed distributions.

Anchoring the Long Tail

High-cardinality categorical features—such as massive vocabularies in natural language processing or millions of items in recommendation engines—pose a severe risk of overfitting. When a model allocates an independent embedding vector to every single entity, rare tokens or infrequent items end up with highly volatile, overfitted representations because the model has too few observations to learn their true behavior.

Structural parameter sharing mitigates this by forcing rare entities to borrow statistical strength from more frequent ones. For instance, the authors of Clustering Embedding Tables, Without First Learning Them and Clustering the Sketch: Dynamic Compression for Embedding Tables demonstrate that interleaving clustering directly into the training loop prevents overfitting on infrequent identifiers. By anchoring the representations of rare items to a shared set of cluster centroids, the model achieves better generalization than uncompressed, full-rank tables.

Instead of clustering complete vectors, models can also share underlying compositional components. The authors of Compressing Word Embeddings via Deep Compositional Code Learning represent words as combinations of a shared set of basis vectors, forcing the network to discover universal semantic components across the entire vocabulary. Similarly, the authors of Slim Embedding Layers for Recurrent Neural Language Models randomly share sub-vectors across different words to prevent overfitting on rare tokens, while the authors of LightRNN: Memory and Computation-Efficient Recurrent Neural Networks arrange words in a table to share row and column vectors, introducing a structural prior that groups similar items together. Even simple feature hashing, as shown by the authors of Collaborative Filtering on a Budget, leverages intentional hash collisions to force entities to share a collective memory budget, providing robust regularization for long-tail items.

Latent Grouping and Discrete Bottlenecks

How do we determine which parameters should be shared? Rather than hardcoding sharing patterns, models can discover latent groups dynamically during training.

In probabilistic modeling, this is often achieved via Bayesian nonparametrics. Jie Liu et al. introduce a framework using a Dirichlet Process prior to automatically group similar parameters in large-scale undirected graphical models, preventing overparameterization by enforcing shared edge or node values. Similarly, Zhihua Zhang 0004 et al. assign unique regression matrices to individual data points but regularize them through a matrix-variate Dirichlet process, forcing similar samples to cluster and share parameters.

In deep neural networks, discrete bottlenecks serve a similar purpose. In a study on knowledge graph compression, Mrinmaya Sachan demonstrates that forcing entities to share a limited set of discrete components acts as a regularizer that improves generalization for sparse data. To optimize these discrete assignments end-to-end, Mingda Chen et al. utilize the Gumbel-Softmax trick to learn task-specific word clusters, demonstrating that forcing semantically similar terms to share parameters makes text classifiers highly robust to noise in the long tail.

Enforcing Architectural Symmetry

Beyond input features, parameter sharing can be baked into the network architecture itself to enforce symmetry across layers, domains, or tasks. When two distinct parts of a network share weights, they are forced to develop a universal, robust representation rather than over-specializing in task-specific noise.

For example, in open-domain question answering, bi-encoder models often fail when encountering out-of-distribution data. To combat this, the authors of Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answering force the question and passage encoders to share the majority of their layers, creating a regularized, shared representation space that significantly improves robustness to distribution shifts. In dual learning tasks (such as translation from English-to-French and French-to-English), the authors of Model-Level Dual Learning demonstrate that sharing parameters between the primal and dual models acts as implicit regularization, preventing the model from overfitting to task-specific noise.

This principle applies vertically across a model's hierarchy as well. The authors of A Study of Parameter Efficient Fine-tuning by Learning to Efficiently Fine-Tune use a shared hypernetwork to generate adapter weights across multiple layers during fine-tuning, preventing individual layers from overfitting. Even at the individual neuron level, Patrick Schramowski et al. show that sharing the parameters of learnable activation functions across layers stabilizes deep reinforcement learning agents by preventing them from overfitting to specific input distributions. In few-shot domain adaptation, Saeid Motiian et al. share weights between source and target feature extractors, anchoring the target domain's representations to the well-established source domain parameters to prevent overfitting on sparse target data.

Smoothing Sparse Distributions

Finally, parameter sharing acts as a powerful smoothing mechanism when estimating complex probability distributions from limited data.

In collaborative filtering, user-item interaction matrices are notoriously sparse. Yin Zheng et al. introduce shared weights across different rating values for the same item, which provides a smoothing effect that prevents the model from becoming biased toward rare rating instances. Similarly, the authors of SPMC: Socially-Aware Personalized Markov Chains for Sparse Sequential Recommendation mitigate overfitting by sharing latent representations across different roles, such as using the same vector for an item whether it acts as historical context or a target prediction.

This smoothing effect extends to continuous prediction tasks. Xingyou Song et al. treat numeric regression as a digit-by-digit sequence generation task; by sharing parameters across the sequence generation steps, the model exhibits implicit regularization that maintains smooth distribution estimates even in low-data regimes. On a broader scale, Ashish Kapoor et al. show that learning a single set of global parameters across all object categories provides greater robustness against overfitting compared to optimizing parameters for each class individually, stabilizing the model when training data is sparse.

Key Foundational Work

Bayesian Estimation of Latently-grouped Parameters in Undirected Graphical Models (2013) — contributed 2 concept elements to this direction.
Matrix-Variate Dirichlet Process Mixture Models (2010) — contributed 1 concept elements to this direction.
Gaussian Processes for Object Categorization (2010) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 20 papers
  2. Version 1 · 2026-10-04 · 24 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_parameter_sharing_regularization_6e5905f0,
  title = {parameter-sharing regularization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-sharing-regularization/rdr_6e5905f0f0734fddb25fbff328d3ac5f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_parameter_sharing_regularization_6e5905f0_v2,
  title = {parameter-sharing regularization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-sharing-regularization/rdr_6e5905f0f0734fddb25fbff328d3ac5f/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake