MS-Glance: Non-semantic context vectors and the applications in supervising image reconstruction

Ziqi Gao, Wendi Yang, Yujia Li, Lei Xing, S. Kevin Zhou

Abstract

MS-Glance is a bio-inspired loss function designed to capture the "gist" of an image—its global spatial structure and local textures—without relying on high-level semantic meaning. By mimicking the human visual system's ability to process a scene in a single quick glance, this multi-scale descriptor significantly improves the fidelity of image reconstructions in neural representations and medical imaging. This paper details the methodology of MS-Glance and demonstrates its superiority over traditional metrics like SSIM and LPIPS across diverse datasets.

Introduction

When observing a scene, the human brain processes "global statistics"—such as layout, lighting, and general structure—almost instantly, long before specific objects are identified. This non-semantic context is crucial for visual understanding. However, most current image reconstruction algorithms focus on two extremes: low-level pixel-wise similarity (e.g., L1/L2 loss) or high-level semantic features (e.g., perceptual loss). The middle ground—non-semantic structural information—is often neglected.

Standard metrics like the Structural Similarity Index Measure (SSIM) are limited by their use of Gaussian kernels, which focus heavily on the center of local windows and fail to capture long-range dependencies. MS-Glance addresses these gaps by utilizing a multi-scale approach to capture both the "big picture" and fine details without the computational overhead of complex semantic models.

Methodology

Multi-Scale Glance Vectors

MS-Glance (Multi-Scale Glance) introduces two distinct types of vectors to represent image context:

  1. Global Glance Vectors: These are created by randomly sampling pixels across the entire image based on a perception-driven rule. This mimics a "quick look" where the eye gathers a sparse but wide-reaching set of data points to understand the scene's layout.
  2. Local Glance Vectors: These consist of flattened local image windows, mimicking a "zoom-in" observation to capture fine textures and details.
Conceptual overview of MS-Glance comparing semantic-based and reconstruction-based approaches.

The Glance Index Measure (GlanceIM)

To compare images, the authors propose the Glance Index Measure (GlanceIM), which calculates the inner product of standardized glance vectors. This is mathematically equivalent to a Pearson correlation coefficient. Unlike SSIM, MS-Glance uses a uniform kernel, arguing that global context should not have a specific focus center. This shift also improves training stability, avoiding the "NaN" errors occasionally encountered with Gaussian kernels in this context.

Domain-Specific Priors

For medical imaging, the researchers introduced an "MRI Air Prior." Since the background (air) in an MRI scan contains no diagnostic information, the sampling rule ignores these pixels, forcing the model to concentrate its "glance" on the actual anatomy.

Experimental Results

The model was evaluated on natural images (COCO, CelebA) and medical images (IXI Brain MRI, FastMRI Knee) against standard benchmarks including L1, L2, SSIM, LPIPS, and S3IM.

Implicit Neural Representation (INR) Fitting

In INR fitting tasks using the SIREN architecture, MS-Glance consistently achieved the highest Peak Signal-to-Noise Ratio (PSNR). For IXI Brain MRIs, it reached 39.141 dB, significantly outperforming the SIREN baseline of 37.430 dB.

MRI Restoration and Qualitative Analysis

In 5x accelerated MRI reconstruction, MS-Glance outperformed LPIPS and SSIM. Qualitatively, it avoided the color distortions common with SSIM and the structural "hallucinations" often produced by semantic-heavy perceptual losses like LPIPS. By bypassing semantic labels, the model avoids turning blurry artifacts into fake anatomical features.

Qualitative comparison of reconstruction results across different loss functions and datasets.

Discussion and Conclusion

MS-Glance demonstrates that stochasticity and bio-mimicry can be powerful tools for image reconstruction. Randomly retrieving pixels to form global vectors is computationally efficient and effectively captures the image "gist" without the need for heavy self-attention mechanisms.

While the current implementation involves some inherent randomness due to stochastic sampling, this is mitigated through multiple sampling passes. Future work will explore the application of MS-Glance in higher-level tasks such as image compression, deblurring, and classification, as well as combining it with learned implicit priors.

Summary updated

BibTeX
@inproceedings{gao2025ms,
  author = {Ziqi Gao and Wendi Yang and Yujia Li and Lei Xing and S. Kevin Zhou},
  title = {{MS-Glance: Non-semantic context vectors and the applications in supervising image reconstruction}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00305},
}
Figure