The Primacy of Macro Architecture in Vision Models

30 concepts14 papers1 proposal

When Vision Transformers (ViTs) first exploded in popularity, the machine learning community largely attributed their success to the self-attention mechanism. Self-attention allowed for dynamic, data-dependent spatial routing, seemingly unlocking capabilities that static convolutions could not match.

However, a series of architectural post-mortems has revealed a surprising truth: you can rip out the self-attention mechanism entirely, replace it with embarrassingly simple operations like pooling, shifting, or static linear mappings, and still maintain state-of-the-art performance. The real driver of success is not the specific token-mixing mechanism, but rather the macro-architecture—the structural template of normalization layers, residual connections, and feed-forward networks that encapsulates the model.

The MetaFormer Abstraction

To understand why the choice of token mixer is secondary, we must decouple the component that aggregates spatial information across patches (the token mixer) from the overall structural layout. In MetaFormer is Actually What You Need for Vision, Weihao Yu et al. formalized this macro-architecture as a general template called MetaFormer.

Input ──► LayerNorm ──► Spatial Token Mixer ──► (+) ──► LayerNorm ──► Channel FFN ──► (+) ──► Output
   │                                           ▲    │                                 ▲
   └───────────────────────────────────────────┘    └─────────────────────────────────┘

In a standard MetaFormer block, the input passes through a normalization layer, a spatial token mixer, and a residual connection, followed by another normalization layer, a channel-mixing feed-forward network (FFN), and a second residual connection.

By isolating the token mixer within this template, the authors demonstrated that even when the spatial mixing step is replaced with a simple non-parametric pooling operation (PoolFormer) or a literal identity mapping (MetaFormer Baselines for Vision), the model retains a remarkably high performance baseline. The structural layout itself provides a robust optimization landscape and an effective inductive bias for processing visual data, doing the heavy lifting for the network.

The Illusion of the "Secret Sauce"

This structural dominance is further validated by architectures that utilize seemingly primitive spatial operators. For instance, Guangting Wang et al. introduced ShiftViT in When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism. ShiftViT replaces self-attention with a zero-parameter, zero-FLOP shift operation along the spatial dimensions. Despite having no active spatial parameterization, ShiftViT matches or outperforms the Swin Transformer across multiple vision tasks.

Similarly, in the context of hierarchical vision models, Yuxin Fang et al. showed in What Makes for Hierarchical Vision Transformer? that replacing local window attention with simple static linear layers (LINMAPPER) yields highly competitive results. The success of these models stems not from the sophistication of their spatial routing, but from the macro-architectural scaffolding—such as interleaved window processing and cross-window communication—that wraps around them.

From a theoretical perspective, Arda Sahiner et al. leveraged convex duality in Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers to show that the specific topology of how information is routed and mixed between tokens provides the primary inductive bias. In many token-mixing architectures, the performance gap between using linear versus non-linear activations within the mixer is minimal, confirming that the macro-architectural routing topology is what truly matters.

Levelling the Playing Field via Network Engineering

If the macro-architecture is the primary driver of performance, why did early attention-based models outperform older convolutional designs? The answer lies in advanced network engineering and modern training recipes.

When integrated into an optimized macro-architecture, the performance disparities between different token mixers largely evaporate. In UniNeXt: Exploring A Unified Architecture for Vision Recognition, Fangjian Lin et al. demonstrated that a well-designed, unified backbone architecture significantly narrows the performance gap between convolutional, attention-based, and parameter-free mixers. The surrounding macro-architecture acts as an equalizer, unlocking the latent potential of simpler mixers.

As Jifeng Dai et al. observed in Demystify Transformers & Convolutions in Modern Image Deep Networks, advanced network engineering techniques—such as overlapped downsampling, modern training strategies, and transformer-style block layouts—frequently exert a greater influence on final accuracy than the intrinsic design of the spatial token mixer itself. When older spatial mixers are ported into these modern frameworks, they routinely achieve performance comparable to or exceeding newer, more complex designs.

Porting the Structural Template

Because the macro-architecture is a universal design principle, researchers have successfully exported this structural template across various domains and paradigms:

Ultimately, this collective body of research shifts the focus of vision architecture design away from isolated operators. The "secret sauce" of modern vision models is not self-attention, but the structural template that contains it.

Key Foundational Work

Demystify Transformers & Convolutions in Modern Image Deep Networks (2022) — contributed 3 concept elements to this direction.
When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism (2022) — contributed 2 concept elements to this direction.
MetaFormer is Actually What You Need for Vision (2021) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_macro_architectural_influence_c5d76d2b,
  title = {macro-architectural influence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/macro-architectural-influence/rdr_c5d76d2b5ddf43c78ef6f66c776d61f0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_macro_architectural_influence_c5d76d2b_v1,
  title = {macro-architectural influence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/macro-architectural-influence/rdr_c5d76d2b5ddf43c78ef6f66c776d61f0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake