CP-MLPs: A Tensor-Rank Theory of Tied and Untied MLP Blocks

Md Rifat Arefin, Farzaneh Heidari, Irina Rish, Guillaume RabusseauarXiv 2026 · arXiv

TL;DR

This paper introduces CP-MLPs, a tensor-based framework that reveals how "untying" the detector and value roles in Transformer feed-forward blocks (like SwiGLU) provides a fundamental width advantage. By analyzing these blocks as sums of rank-1 tensor interactions, the authors prove that untied architectures are strictly more efficient at representing complex, asymmetric interactions, while tied architectures (like ReLU²) are specialized for symmetric "pure-power" targets.

What Problem Does This Paper Address?

Modern Transformer models rely heavily on feed-forward network (FFN) blocks, which account for roughly two-thirds of their parameters. Researchers have proposed numerous variants—such as ReLU², ReGLU, and SwiGLU—but the specific design choices that drive their success are often conflated.

The authors isolate a key design axis: parameter tying. In a "tied" block, the model uses the same direction to detect a feature and to read out its value. In an "untied" block (common in Gated Linear Units or GLUs), these roles are decoupled. The paper asks: When does untying these roles actually improve expressivity, and can we quantify the "width" (number of channels) required to represent specific mathematical interactions?

Methods

The authors propose the CP-MLP framework, which views MLP channels through the lens of Canonical Polyadic (CP) tensor decomposition.

As shown in the leg diagram below, each channel in a CP-MLP performs three roles:

  1. Detector ($u_a$): Selects the input feature to be gated.
  2. Value ($v_a$): Supplies the linear signal that the gate modulates.
  3. Output ($w_a$): Writes the result to the output space.
Leg diagram of a single CP-MLP channel.

By treating the gate's activation as a polynomial expansion (Taylor series), the authors can analyze the "effective atoms" each architecture uses at different degrees. They use tools from algebraic geometry, specifically Waring rank (the minimum number of powers needed to represent a polynomial), to prove exact separations in the width required by tied vs. untied models.

Datasets

The study primarily uses synthetic benchmarks designed to isolate specific mathematical properties:

  • Indefinite Quadratic Targets: Symmetric matrices with varying numbers of positive ($n_+$) and negative ($n_-$) eigenvalues.
  • Role-Asymmetric Monomials: Targets like $x_1^{ k-1}x_2$, where one variable is repeated (the "detector" role) and another appears once (the "value" role).

  • Pure-Power Targets: Simple targets like $x_1^k$.

  • Trained-Block Distillation: A more "naturalistic" setup where small MLPs are trained to mimic a larger, frozen random tanh network.

Key Insights and Ideas

  • The Untied Advantage: Untying isn't just about adding parameters; it changes the "geometry" of the available building blocks. An untied atom can represent an interaction like $a^2 - b^2$ in a single channel, whereas a tied model (limited to $a^2$ or $b^2$) would require two.
  • Structural Matching: Tying is not universally worse. Tied atoms are "structurally matched" to symmetric pure-power targets. If the data only requires detecting a feature and scaling it, tying is perfectly efficient.
  • Monomial Separation: For a degree-$k$ asymmetric monomial ($x_1^{k-1}x_2$), an untied model needs only 1 channel, while a tied model requires $k$ channels.
  • Border Rank and Plateaus: The authors distinguish between exact rank (what is needed for a perfect fit) and border rank (what is needed to get arbitrarily close). They show that even if a tied model lacks the width for an exact fit, it can often reach very low loss by using extremely large weights—a phenomenon that explains "loss plateaus" seen in practice.

Key Results

The theoretical predictions were confirmed through rigorous experiments:

  • Quadratic Width Gap: For indefinite quadratic targets, tied blocks require a width of $n_+ + n_-$, while untied blocks only need $\max(n_+, n_-)$. This results in a clear "gap" in required width, as seen in Figure 2.
Figure 2: Quadratic targets with (n+, n−) positive and negative directions. Left: required widths match the predictions n+ + n−(tied) and max{n+, n−} (untied). Right: loss stays near the target variance below the predicted width and reaches machine precision at the predicted width.
  • Monomial Efficiency: In tests on $x_1^{k-1}x_2$, untied atoms reached machine precision at width $m=1$ for all tested degrees, while tied atoms required significantly more width to achieve the same accuracy.
  • Gated MLP Performance: When comparing actual architectures like ReGLU (untied) and ReLU² (tied) at matched parameter budgets, untied blocks consistently achieved lower loss on asymmetric targets, suggesting that the "GLU success" in LLMs is partly due to this role separation.

The following table summarizes how common blocks fit into this framework:

Table 1: Feed-forward blocks as CP-MLPs. The gate controls which degrees are active through its polynomial signature; the tying pattern controls whether detector and value directions are shared.

Limitations and Future Work

  • Degree-wise Analysis: The current theory analyzes expressivity degree-by-degree (via Taylor coefficients). While this provides necessary bounds, it isn't yet a complete global theory for the full non-linear function.
  • Synthetic Targets: The experiments rely on synthetic polynomial targets. While these are excellent for isolating rank effects, the authors note that bridging the gap to real-world language modeling tasks is a vital next step.
  • Multi-leg Atoms: Future work could extend this theory to "multi-leg" atoms (e.g., blocks that multiply three or more projections) to see if further efficiency gains are possible.

Summary updated

Abstract

Transformer feed-forward blocks combine several design choices: squared activations, gating, and tied versus untied parameter sharing, whose separate contributions to expressivity remain poorly understood. We introduce CP-MLPs, a tensor framework in which each multiplicative unit contributes a rank-1 interaction with three roles: detecting an input feature, reading a value, and writing to the output. Tied gated blocks such as ReLU$^2$ force the detector and value roles to share the same direction; untied gated blocks such as ReGLU and SwiGLU let them differ. We show that this distinction yields exact, degree-wise tensor-rank separations: untied detector-value units are strictly more width-efficient for indefinite quadratic interactions and role-asymmetric monomials, while tied units are structurally matched to symmetric pure-power targets. The width advantage of untying therefore depends on the target structure. Experiments show loss plateaus below the exact-representation width, but these plateaus alone do not establish a positive approximation gap: arbitrarily accurate approximation is governed by border rank. In trained gated MLPs, untied blocks retain an advantage in selected comparisons at matched input-projection budget. Code is available at https://github.com/rarefin/CP-MLPs.
BibTeX
@misc{arefin2026cp,
  author = {Md Rifat Arefin and Farzaneh Heidari and Irina Rish and Guillaume Rabusseau},
  title = {{CP-MLPs: A Tensor-Rank Theory of Tied and Untied MLP Blocks}},
  year = {2026},
  eprint = {2611.13647},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13647},
}
Figure