Mitigating the Feed-Forward Network Bottleneck in Transformers

31 concepts28 papers

When optimizing Transformer architectures, researchers traditionally focus on the quadratic complexity of the self-attention mechanism. However, an asymmetric reality lies under the hood: Feed-Forward Networks (FFNs)—also referred to as Multi-Layer Perceptron (MLP) blocks—typically consume over two-thirds of a Transformer's total parameter budget and floating-point operations (FLOPs).

As attention mechanisms become increasingly optimized through sparse or linear approximations, the static, dense computational cost of FFNs has emerged as the primary bottleneck for both training and inference. To build truly efficient models, we must look beyond attention and fundamentally restructure how Transformers execute non-linear token transformations.

The Root of the Bottleneck: Dense Redundancy

In a standard Transformer layer, the FFN acts as a monolithic block that maps tokens into a high-dimensional intermediate space (often four times the hidden dimension) before projecting them back. This operation is applied uniformly across every single token in a sequence.

This uniform application introduces severe inefficiencies for two reasons:

  1. Activation Sparsity: For any given input token, only a tiny fraction of the neurons in the intermediate layer actually fire. The rest contribute zero or negligible values to the final output.
  2. Token Asymmetry: Not all tokens require the same amount of computational depth to process. A punctuation mark or a highly predictable word receives the same heavy FFN computation as a complex semantic entity.

Because dense FFNs compute every neuron for every token, they incur massive memory access overhead and energy costs. To resolve this, recent research focuses on four primary strategies: exploiting natural activation sparsity, structurally redesigning the underlying matrices, routing tokens conditionally, and targeting FFNs during parameter-efficient fine-tuning.

Exploiting Sparsity via Mixture-of-Experts

If only a small percentage of FFN neurons are active at any given time, we can physically reorganize the dense layer to avoid computing inactive weights.

In MoEfication: Transformer Feed-forward Layers are Mixtures of Experts, Sun et al. demonstrated that standard FFNs can be post-hoc split into a Mixture-of-Experts (MoE) structure. By clustering neurons that frequently fire together into discrete "experts" and utilizing a lightweight router, the model can dynamically compute only 10–30% of the original FFN layer per token with negligible loss in accuracy.

Standard FFN:  [Token] ---> [ Massive Dense Linear Layer (All Neurons Compute) ]
                                                   │
                                                   ▼
MoEfication:   [Token] ---> [ Router ] ──> [ Only Active Expert Clusters Compute ]

This concept of dynamic selection operates along both the neuron and channel dimensions:

  • Edge Fine-Tuning: In resource-constrained environments, Vucetic et al. proposed the Freeze And Reconfigure (FAR) framework. By identifying the most "eager to learn" neurons during a brief priming phase, the authors physically freeze the remaining redundant neurons, drastically cutting down memory access times during backward passes.
  • Channel Selection: Rather than focusing purely on token length, Moon et al. introduced ToaSt, a framework that performs dynamic channel selection within FFNs. This approach treats redundant channels as noise filters, showing that dropping inactive channels can simultaneously boost throughput and model accuracy.

Structural Matrix Redesign and Subspaces

Instead of patching an existing dense layer, another approach replaces the monolithic FFN matrix entirely with structured, parameter-efficient mathematical alternatives.

In Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers, Wei et al. proved that Large Language Models can be trained from scratch by replacing standard FFNs with low-rank and block-diagonal structured matrices. This structural constraint reduces the parameter footprint by up to 17% while maintaining competitive performance. Similarly, for vision tasks, Zhen et al. leveraged Tensor-Train decomposition to compress the storage footprint of FFN layers by over 400x.

Other architectures break the monolithic FFN down into parallel, specialized streams:

  • Subspace Interactions: Tang et al. designed MSCFFN, which splits the high-dimensional FFN space into smaller, parallel subspaces that interact via cross-space operations, achieving a 1.7x speedup over standard dense layers.
  • Head-Specific FFNs: In PartialFormer: Modeling Part Instead of Whole for Machine Translation, Zheng et al. replaced the single massive FFN with an ensemble of smaller, head-specific "Partial-Level Gated FFNs," distributing the parameter budget more granularly across the model.

Conditional Computation and Token Routing

As sequence lengths grow, processing every token through the same heavy FFN layer becomes prohibitively expensive. Conditional computation bypasses this by filtering tokens before they ever reach the FFN.

Ainslie et al. developed CoLT5 to address this specific bottleneck in long-context processing. CoLT5 routes only a small, dynamically selected subset of important tokens to a heavy-duty FFN branch. The remaining "unimportant" tokens are processed by a highly lightweight linear layer. This ensures that the bulk of the model's FFN compute budget is spent exclusively on tokens that drive the core semantics of the document.

FFN-Targeted Fine-Tuning and Reparameterization

While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA traditionally target attention weights, the FFN layer remains an unoptimized memory sink during model adaptation.

To resolve this, Chekalina et al. introduced SparseGrad. By applying Higher Order SVD (HOSVD) to the MLP layers, the authors identified a specific mathematical basis where gradients are naturally sparse. This allows practitioners to update as little as 1% of the MLP parameters during fine-tuning while matching the performance of full parameter updates.

For inference deployment, structural reparameterization allows models to benefit from dense representations during training while executing efficiently on hardware. Xu et al. developed RePaViT, which identifies "channel idle" regions in Vision Transformer FFNs and mathematically compresses them prior to deployment, converting structural redundancy directly into end-to-end latency speedups.

Key Foundational Work

DaViT: Dual Attention Vision Transformers (2022) — contributed 3 concept elements to this direction.
Rethinking the Self-Attention in Vision Transformers (2021) — contributed 2 concept elements to this direction.
Of Non-Linearity and Commutativity in BERT (2021) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_feed_forward_network_overhead_780a6f91,
  title = {feed-forward network overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feed-forward-network-overhead/rdr_780a6f91f5f14f1cad6b961f418a3bad},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_feed_forward_network_overhead_780a6f91_v1,
  title = {feed-forward network overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/feed-forward-network-overhead/rdr_780a6f91f5f14f1cad6b961f418a3bad/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake