Masking Strategies for Image Manifolds

Hamid Dadkhahi, Marco F. Duarte

Abstract

Modern imaging sensors in power-constrained environments, such as wearable devices, face significant energy challenges. This paper presents Manifold-Aware Pixel Selection (MAPS), a framework comprising two algorithms—MAPS-Global and MAPS-Local—designed to identify the most informative pixel subsets for preserving an image's underlying manifold structure. By sensing only a fraction of pixels, these strategies can reduce power consumption by up to 73% while maintaining high performance in downstream tasks like eye-gaze tracking.

Introduction

In applications ranging from smart eyeglasses to remote environmental monitoring, energy efficiency is paramount. Traditional digital cameras capture full-resolution frames, a process that is inherently energy-intensive. While compressive sensing offers a path toward data reduction, many implementations still require full-scene sensing before compression or necessitate specialized hardware.

This research explores data-dependent masking: the strategic selection of a subset of pixels that sufficiently captures the low-dimensional manifold on which the data resides. By identifying and activating only the pixels critical to defining geometric relationships (such as distances and neighborhoods), sensors can achieve significant power savings without sacrificing the structural integrity of the captured information.

Methodology: Manifold-Aware Pixel Selection (MAPS)

The authors propose two primary algorithms that treat pixel selection as a search for an optimal binary indicator vector. Given the NP-hard nature of finding an absolute optimal mask, greedy heuristic algorithms are employed to iteratively select pixels.

MAPS-Global

Inspired by the Isomap algorithm, MAPS-Global focuses on preserving geodesic distances across the manifold. It utilizes "secants" (normalized differences between neighboring data points) and seeks a mask that minimizes the average distortion of these secants within the masked space.

MAPS-Local

Drawing from Locally Linear Embedding (LLE), MAPS-Local prioritizes the preservation of local weights that describe a point as a linear combination of its neighbors. It employs a cosine similarity metric to ensure that local geometric "cliques" remain intact after masking.

Experimental Setup

The algorithms were evaluated across six diverse image datasets representing various manifold structures, including handwritten digits, rotating heads, and human facial expressions.

Summary of datasets and manifold parameters.

The datasets vary significantly in image dimension and embedding dimensionality, providing a robust testbed for the proposed masking strategies.

Results and Discussion

The MAPS algorithms consistently outperformed traditional variance-based selection (such as PCoA) and random masking. A key finding is that high-variance pixels are not always the most informative for manifold preservation. MAPS often selects pixels that appear "noisy" or dim to the human eye but are mathematically critical for defining the manifold's shape.

Visual comparison of masking strategies across datasets. Note how MAPS-Local and MAPS-Global select distributed, structurally significant pixels compared to variance-based methods like PCoA.

Hardware Impact and Performance

The practical utility of these algorithms was demonstrated using a prototype of computational eyeglasses. By applying a mask of only 200 pixels (out of 1600), the camera's power consumption dropped from 30 mW to 8 mW. Despite this reduction, eye-gaze estimation error remained nearly as low as when using full-frame images. Furthermore, the masks generated from training data showed strong out-of-sample performance, indicating that the selected pixels capture general structural properties of the manifold rather than merely memorizing specific training instances.

Conclusion

MAPS provides a mathematically grounded approach to energy-efficient imaging. While the current implementation requires an initial full-power training phase and utilizes greedy approximations, it sets the stage for more advanced sensing. Future research may explore dynamic masks that adapt to temporal correlations in video sequences or extend the framework to handle datasets comprising a union of multiple distinct manifolds.

Summary updated

BibTeX
@article{dadkhahi2016masking,
  author = {Hamid Dadkhahi and Marco F. Duarte},
  title = {{Masking Strategies for Image Manifolds}},
  journal = {IEEE Trans. Image Process.},
  year = {2016},
  doi = {10.1109/tip.2016.2588327},
}
Figure