Overcoming the Local Bias of Convolutional Networks

44 concepts46 papers1 proposal

For years, Convolutional Neural Networks (CNNs) served as the bedrock of computer vision. Their success stems from a powerful inductive bias: locality. By sliding a small kernel (typically 3x3 or 5x5) across an image, a convolutional layer assumes that pixels close to one another are highly correlated. This makes CNNs incredibly efficient at extracting low-level features like edges, textures, and local shapes.

However, this local focus introduces a fundamental architectural bottleneck: a single convolutional layer cannot see the big picture. To connect two distant regions of an image, information must propagate through a deep stack of successive layers. While operations like pooling and strided convolutions accelerate this expansion, they do so by discarding spatial resolution. As a result, deep CNNs often struggle with tasks that require a comprehensive understanding of global context or long-range spatial relationships.

The Receptive Field Bottleneck

To understand why local feature extraction fails in complex visual tasks, consider how a network determines the boundaries of an object. As noted by Li et al. in their work on MRCA-UNet, relying strictly on local convolutional features frequently leads to blurred boundaries in medical image segmentation. Without global context, a network cannot determine whether a specific local edge belongs to a target organ or surrounding tissue.

This limitation manifests across several domains:

  • Remote Sensing: Satellite and aerial images cover vast geographic areas where objects are separated by large distances. Wan et al. demonstrate in RDNet that standard convolutions fail to capture the macro-level spatial relationships necessary to interpret large scenes. Even when attention mechanisms are retrofitted onto convolutional backbones, Bandara et al. observe in ChangeFormer that the underlying local constraints still hamper the network's ability to relate distant features for precise change detection.
  • Robotic Navigation and 3D Vision: In object navigation, an agent must continuously fuse its historical trajectory with its immediate surroundings. Jiang et al. point out in NaviFormer that CNNs struggle with these long-range spatio-temporal dependencies, often causing navigation failures. Similarly, for 6D object pose estimation, Tian et al. argue in CTNet that global scene context is vital to resolve spatial ambiguities that local kernels miss.
  • Cross-Modal Fusion: Aligning disparate data streams—such as matching an infrared image with a visible-light image, or an RGB frame with a depth map—requires recognizing non-local structural similarities. Pang et al. show in CAVER that the local connectivity of CNNs limits their capacity for robust cross-modal alignment. To circumvent this, researchers like Hancock et al. have explored alternative structures like graph representation learning (Graph Representation Learning for Infrared and Visible Image Fusion) to capture non-local self-similarity directly.

Shifting to Global Architectures

To bypass the receptive field bottleneck, the research community has increasingly turned to architectures that offer an immediate, global receptive field: Vision Transformers (ViTs) and State Space Models (SSMs).

Vision Transformers and Self-Attention

Transformers eliminate the local constraint entirely through the self-attention mechanism. Instead of searching within a fixed neighborhood, self-attention allows every pixel patch to compute a relationship score with every other pixel patch in the image simultaneously.

The authors of U-MixFormer leverage this capability to dynamically mix multi-scale features across encoders and decoders, ensuring that global context informs local reconstructions. In medical imaging, where structural context is critical, Li et al. introduce FocalUNETR, utilizing focal self-attention to capture long-range spatial dependencies for boundary-aware prostate segmentation. For multi-modal tasks, Zeng et al. employ dual Swin-Transformers in DTMINet to capture long-range contextual dependencies across both RGB and depth modalities.

State Space Models (Mamba)

While Transformers successfully capture global context, they suffer from quadratic computational complexity ($O(N^2)$ relative to the number of pixels). This makes them prohibitively expensive for high-resolution images or long sequences.

State Space Models, particularly the Mamba architecture, solve this by flattening images into sequences and processing them with selective scan mechanisms. This achieves long-range dependency modeling with linear computational complexity ($O(N)$). As surveyed by Liu et al. in Vision Mamba: A Comprehensive Survey and Taxonomy, Mamba offers a highly efficient alternative to Transformers for global context modeling. This efficiency allows architectures like CAMS, developed by Khan et al., to completely abandon both convolutions and self-attention for cardiac image segmentation. Similarly, Iqbal et al. utilize this linear-scaling global context in Pyramid-based Mamba to detect large-scale anomalies that escape local convolutional inspection.

The Hybrid Solution

If global models are so effective, why not abandon convolutions entirely?

The reality is that discarding convolutions completely forces a network to learn basic spatial relationships (such as the fact that adjacent pixels are usually related) entirely from scratch. This often requires massive datasets and prolonged training cycles. Convolutions remain exceptionally efficient at extracting high-frequency, fine-grained details like sharp edges and local textures.

The most pragmatic and powerful trend in modern computer vision is therefore the hybrid architecture, which pairs the local efficiency of CNNs with the global reach of Transformers or Mamba blocks.

Input Image ───► [ Convolutional Layers ] ───► Local Features (Edges, Textures)
                        │
                        ▼
                 [ Mamba / Transformer ] ───► Global Context (Spatial Relationships)
                        │
                        ▼
                 [ Fused Representation ] ───► Precise, Context-Aware Output

This synergy is highly visible across recent developments:

  • Real-Time Deployment: Ding et al. introduce RTMamba for real-time remote sensing semantic segmentation on resource-constrained drones, combining Mamba’s long-range sequence modeling with localized pooling modules to balance speed and accuracy.
  • Complex Boundary Delineation: In brain tumor segmentation, Wang et al. design MambaBTS, fusing multi-scale convolutional kernels with Mamba sequence modeling to capture both fine structural boundaries and macro-level context.
  • Depth and Spatial Reasoning: Wu et al. demonstrate in AMENet that fusing Vision Transformers' global perception with a CNN's spatial feature extraction yields significantly more robust monocular depth estimation in complex scenes.

By understanding that local feature extraction and global context modeling are complementary forces rather than competing paradigms, modern vision architectures achieve a balanced, highly efficient representation of the visual world.

Key Foundational Work

TED-Net: Convolution-Free T2T Vision Transformer-Based Encoder-Decoder Dilation Network for Low-Dose CT Denoising (2021) — contributed 1 concept elements to this direction.
Attentive Convolution: Equipping CNNs with RNN-style Attention Mechanisms (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_local_feature_extraction_limitations_c27d4586,
  title = {local feature extraction limitations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/local-feature-extraction-limitations/rdr_c27d4586d9e049529d2f06c7b46e587a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_local_feature_extraction_limitations_c27d4586_v1,
  title = {local feature extraction limitations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/local-feature-extraction-limitations/rdr_c27d4586d9e049529d2f06c7b46e587a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake