PrivateMail: Supervised Manifold Learning of Deep Features with Privacy for Image Retrieval

Praneeth Vepakomma, Julia Balla, Ramesh Raskar

Introduction

Content-based image retrieval (CBIR) is a cornerstone of modern computer vision, enabling users to find similar images within massive databases by comparing "deep features." However, this process often compromises privacy. When a client sends a feature vector to a server, the server can potentially reconstruct the original image or infer sensitive metadata. Existing solutions typically fall into two categories: computationally expensive cryptographic methods or standard Differential Privacy (DP) techniques that often degrade search utility to unusable levels.

Presented at AAAI-2022, PrivateMail introduces a novel framework for secure image retrieval. It leverages a differentially private mechanism for supervised manifold learning, allowing clients to query remote databases with formal privacy guarantees while maintaining high search accuracy.

The PrivateMail Framework

The core of PrivateMail is the Supervised Manifold Learning Query (SMLQ). Unlike unsupervised methods such as PCA, SMLQ incorporates label information to ensure that the resulting low-dimensional embeddings preserve class separation, which is critical for retrieval tasks.

The PrivateMail system architecture illustrates the secure interaction between the client and server premises, utilizing a shared public dataset for alignment.

Iterative Optimization and Privacy

The SMLQ is solved using a majorization-minimization approach that typically converges within 5–7 iterations. To ensure privacy, the authors mathematically derived the $l_2$-sensitivity for the first iteration of the SMLQ and applied Gaussian noise to its output.

A key innovation of PrivateMail is its strategic use of the post-processing property of Differential Privacy. Since DP is immune to subsequent computations on privatized data, only the first iteration incurs a privacy budget ($\epsilon$) cost. All subsequent refinement iterations are "free," allowing the model to achieve a much cleaner embedding than methods that add noise at every step.

Distributed Geometric Alignment

Because the client and server generate embeddings independently, their coordinate systems are naturally misaligned. PrivateMail solves this by using a shared public dataset as an "anchor." Both parties include this public data in their manifold learning process. The client then uses the Kabsch-Umeyama algorithm to perform a rigid transformation, aligning their private embedding space with the server's space.

Visualization of embedding alignment demonstrates how the public dataset acts as an anchor to synchronize the client and server coordinate systems.

To further protect the client's intent, the framework employs dummy queries. The client appends images from non-target classes to the request, preventing the server from inferring the target class based on the returned neighbors. The client then filters these results locally.

Experimental Evaluation

The authors evaluated PrivateMail on three benchmark datasets: CUB-200-2011, Cars196, and CIFAR-100, using ResNet-50 features.

Key Results

  • Utility-Privacy Trade-off: PrivateMail consistently outperformed baselines like DP-PCA and DP-dSNE. On the Cars196 dataset, retrieval performance approached non-private levels even under strict privacy constraints.
  • Convergence and Efficiency: The iterative process is highly efficient, reaching optimal recall in fewer than 10 iterations across all tested datasets.
  • Scalability: By avoiding heavy cryptographic primitives, PrivateMail remains scalable to high-dimensional features and large-scale databases.

Conclusion

PrivateMail demonstrates that supervised manifold learning, when combined with strategic noise placement and public anchoring, can bridge the gap between privacy and utility in image retrieval. By exploiting the post-processing properties of Differential Privacy, it provides a practical framework for secure, high-performance CBIR in distributed environments.

Summary updated

Abstract

Differential Privacy offers strong guarantees such as immutable privacy under any post-processing. In this work, we propose a differentially private mechanism called PrivateMail for performing supervised manifold learning. We then apply it to the use case of private image retrieval to obtain nearest matches to a client’s target image from a server’s database. PrivateMail releases the target image as part of a differentially private manifold embedding. We give bounds on the global sensitivity of the manifold learning map in order to obfuscate and release embeddings with differential privacy inducing noise. We show that PrivateMail obtains a substantially better performance in terms of the privacy-utility trade off in comparison to several baselines on various datasets. We share code for applying PrivateMail at http://tiny.cc/PrivateMail.
BibTeX
@inproceedings{vepakomma2022privatemail,
  author = {Praneeth Vepakomma and Julia Balla and Ramesh Raskar},
  title = {{PrivateMail: Supervised Manifold Learning of Deep Features with Privacy for Image Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i8.20827},
}
Figure