Recent Advances of Local Mechanisms in Computer Vision: A Survey and Outlook of Recent Work

Qiangchang Wang, Yilong YinarXiv 2023 · OpenReview

Abstract

Inspired by the human brain's ability to emphasize discriminative parts of an input while suppressing irrelevant noise, researchers have developed "local mechanisms" to advance computer vision. These techniques focus on target regions to learn discriminative representations and process information selectively for improved efficiency. This survey provides a systematic review of local mechanisms across various tasks, including fine-grained visual recognition, person re-identification, few-shot learning, and Vision Transformers (ViTs). By analyzing the advantages and disadvantages of different categories, we shed light on future research directions in the field.

Introduction: The Roadmap of Locality

Local mechanisms allow models to focus on specific image regions rather than processing everything globally. In modern architectures like Vision Transformers and paradigms like self-supervised learning, these mechanisms are essential for making computer vision more efficient, robust to occlusions, and capable of learning from limited data.

Taxonomy of local mechanisms across various computer vision tasks.

The Problem: Limitations of Global Processing

While global image representations have historically dominated deep learning, they face three significant hurdles:

  1. Computational Waste: Standard ViTs suffer from quadratic complexity; as image resolution increases, memory and processing requirements explode. Processing every pixel equally is inherently inefficient.
  2. Brittleness: Global models often fail when objects are partially occluded or viewed from unusual angles.
  3. Data Hunger: Global features are often dataset-specific. Conversely, local features (like textures or specific part shapes) are more transferable, which is critical for tasks with scarce data, such as Few-Shot Learning.

Categorizing Local Mechanisms

The survey divides local mechanisms into three functional eras:

  • Hard Part Division: The traditional approach of manually cropping images into fixed grids or patches.
  • Soft Part Division: Using attention modules or clustering to automatically identify important regions while suppressing noise.
  • The Transformer Era: The current frontier, focusing on CNN hybrids, window-based attention (e.g., Swin Transformers), and token pruning to discard redundant background information.

Self-Supervised Learning and Masked Prediction

One of the most significant shifts in recent years is the move from global contrastive learning to local masked prediction in self-supervised learning (SSL).

Comparison between (a) global contrastive learning and (b) local masked prediction in self-supervised learning.

As illustrated above, while contrastive learning (a) often relies on global views, Masked Image Modeling (b) forces models to reconstruct missing local patches. This approach has emerged as a powerhouse, demonstrating that predicting local structure is often superior for downstream tasks.

Key Insights: Efficiency and Transferability

  • Locality as Inductive Bias: Local mechanisms act as a "shortcut," allowing models to perform better on smaller datasets by assuming nearby pixels are related. Pure ViTs are often "data hungry" because they lack this inherent bias.
  • Universal Language: Local features are highly transferable. While a global view of a car and a truck differ, their local parts (wheels, lights) are similar, forming the backbone of Zero-Shot Learning.
  • Efficiency via Sparsity: Token pruning can achieve massive speedups (up to 15x in some cases) by identifying and discarding redundant background pixels without sacrificing accuracy.

Future Outlook

The field is moving toward more "knowledgeable" and "selective" models. Future research must address:

  • Top-Down Selectivity: Models should dynamically change which local parts they prioritize based on high-level goals.
  • Memory Retrieval: AI needs better mechanisms to remember and transfer local knowledge across diverse domains.
  • Contextual Reasoning: Balancing local focus with surrounding context remains a challenge; a model must not become so "zoomed in" that it loses vital environmental cues (e.g., recognizing a keyboard is easier when the monitor is visible).

Summary updated

Abstract

Inspired by the fact that human brains can emphasize discriminative parts of the input and suppress irrelevant ones, substantial local mechanisms have been designed to boost the development of computer vision. They can not only focus on target parts to learn discriminative local representations, but also process information selectively to improve the efficiency. In terms of application scenarios and paradigms, local mechanisms have different characteristics. In this survey, we provide a systematic review of local mechanisms for various computer vision tasks and approaches, including fine-grained visual recognition, person re-identification, few-/zero-shot learning, multi-modal learning, self-supervised learning, Vision Transformers, and so on. Categorization of local mechanisms in each field is summarized. Then, advantages and disadvantages for every category are analyzed deeply, leaving room for exploration. Finally, future research directions about local mechanisms have also been discussed that may benefit future works. To the best our knowledge, this is the first survey about local mechanisms on computer vision. We hope that this survey can shed light on future research in the computer vision field.
BibTeX
@misc{wang2023recent,
  author = {Qiangchang Wang and Yilong Yin},
  title = {{Recent Advances of Local Mechanisms in Computer Vision: A Survey and Outlook of Recent Work}},
  year = {2023},
  eprint = {2306.01929},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.01929},
}
Figure