Human-Imperceptible Identification with Learnable Lensless Imaging

Thuong Nguyen Canh, Trung Thanh Ngo, Hajime Nagahara

Introduction

As cameras become ubiquitous in smartphones, wearables, and surveillance systems, visual privacy is increasingly at risk. Traditional software-based privacy methods, such as blurring or pixelating an image after capture, are inherently vulnerable; the "clear" image exists momentarily in the system's memory, making it a target for malware or hacking.

Hardware-level protection offers a more robust alternative. Lensless imaging, which replaces a standard lens with a coded mask, captures a "diffraction pattern" or a heavily blurred measurement instead of a clear picture. However, this approach faces a classic trade-off: the more an image is obscured for privacy, the more difficult it becomes for machine learning models to perform tasks like face recognition. This research introduces a learnable lensless imaging system designed to capture images that are unrecognizable to the human eye but remain highly accurate for machine-learning-based identification.

Methodology: Joint Optimization of Hardware and Software

The researchers proposed an end-to-end framework that jointly learns the physical coded pattern of the mask and the weights of a recognition network (ResNet18). By treating the physical imaging process as the first layer of a neural network, the system can be optimized for the specific task of identification while adhering to privacy constraints.

The end-to-end framework jointly optimizes the physical coded mask (H) and the recognition network (R) to produce an identification result (b) from a blurry sensor measurement (y).

Privacy-Preserving Loss Functions

To ensure the captured images were "human-imperceptible," the team introduced four specific loss functions designed to prevent the system from learning a simple, privacy-violating solution:

  1. Similarity Loss ($L_{sim}$): Forces the learned measurement to resemble an image captured with a "fully open" aperture—essentially a featureless blur.
  2. Total Variation (TV) Loss ($L_{tv}$): Encourages complexity in the learned mask, preventing it from collapsing into a simple "pinhole" pattern that would reveal too much detail.
  3. Invertibility Loss ($L_{inv}$): Minimizes the mathematical possibility of reconstructing a clear image from the blurry sensor data.
  4. Restricted Isometry Property (RIP) Loss ($L_{rip}$): This is the most novel addition. In compressive sensing, RIP is a condition that guarantees image reconstruction is possible. Here, the authors intentionally minimize the RIP condition, creating a mathematical "one-way street" that protects privacy by making reconstruction as difficult as possible.

Experimental Results and Discussion

The models were trained and evaluated on three standard large-scale face datasets: MS-Celeb-1M, VGG-Face2, and CASIA-WebFace. The results demonstrated that deep learning models can "see" features in high-dimensional blurry data that the human visual system cannot process.

Visual comparison of target images and the resulting sensor measurements under different loss constraints. Note how the RIP and TV losses produce images that are unrecognizable to humans but maintain high identification accuracy.

Key Findings:

  • High Accuracy: On the MS-Celeb dataset, the learned lensless system achieved 93.23% accuracy using the TV loss and 91.12% using the RIP loss. In contrast, a random, non-optimized mask achieved only ~75% accuracy.
  • The Pinhole Trap: Without privacy constraints, the learned mask naturally evolves toward a pinhole-like pattern. While a pinhole provides the best accuracy (nearly 99%), it reveals the most human-readable information, defeating the purpose of privacy.
  • Human Subjective Testing: In perceptual experiments, ten human observers were asked to identify faces from the captured measurements. While the "pinhole" mask allowed for easy identification, the learned masks (TV and RIP) resulted in human accuracy that was no better than random guessing.
  • Hardware Validation: In real-world tests with a physical prototype using a CMOS sensor and photo-lithography masks, the system maintained strong performance, achieving up to 87% accuracy on the CASIA dataset while keeping images visually obscured.

Conclusion

This work demonstrates that it is possible to decouple machine recognition from human perception at the hardware level. By intentionally violating the Restricted Isometry Property, the researchers created a system where the machine doesn't need a "pretty" picture to function; it only needs consistent statistical patterns. Future work will focus on scaling the system to thousands of identities and improving robustness to varying subject distances and environmental noise.

Summary updated

BibTeX
@article{canh2023human,
  author = {Thuong Nguyen Canh and Trung Thanh Ngo and Hajime Nagahara},
  title = {{Human-Imperceptible Identification with Learnable Lensless Imaging}},
  journal = {IEEE Access},
  year = {2023},
  doi = {10.1109/access.2023.3308069},
}
Figure