Privacy-Preserving Face Recognition in the Frequency Domain

Yinggui Wang, Jian Liu, Man Luo, Le Yang, Li WangAAAI 2022 · OpenReview

Abstract

Some applications may require performing face recognition (FR) on third-party servers, which could be accessed by attackers with malicious intents to compromise the privacy of users’ face information. This paper advocates a practical privacy-preserving FR scheme without key management realized in the frequency domain. The new scheme first collects the components of the same frequency from different blocks of a face image to form component channels. Only part of the channels are retained and fed into the analysis network that performs an interpretable privacy-accuracy trade-off analysis to identify channels important for face image visualization but not crucial for maintaining high FR accuracy. For this purpose, the loss function of the analysis network consists of the empirical FR error loss and a face visualization penalty term, and the network is trained in an end-to-end manner. We find that with the developed analysis network, more than 94% of the image energy can be dropped while the face recognition accuracy stays almost undegraded. In order to further protect the remaining frequency components, we propose a fast masking method. Effectiveness of the new scheme in removing the visual information of face images while maintaining their distinguishability is validated over several large face datasets. Results show that the proposed scheme achieves a recognition performance and inference time comparable to ArcFace operating on original face images directly.

Introduction

Face recognition (FR) has become ubiquitous, yet its reliance on client-server architectures poses significant privacy risks. When biometric data is transmitted to a cloud server, it becomes vulnerable to breaches. Unlike passwords, biometric data cannot be reset once compromised. Existing solutions like Homomorphic Encryption are computationally prohibitive for real-time use, while perturbation methods like Differential Privacy often degrade accuracy significantly.

The researchers propose a "Goldilocks" solution: a method that is fast, secure against reconstruction attacks, and maintains industry-standard accuracy. This is achieved by operating in the frequency domain to decouple human perception from machine recognition.

Conceptual trade-off between human perception and DNN recognition in the frequency domain.

Methodology: The PPFR-FD Scheme

The core innovation, Privacy-Preserving Face Recognition in the Frequency Domain (PPFR-FD), leverages the Block Discrete Cosine Transform (BDCT).

Frequency Analysis and Channel Selection

By converting images into frequency components, the authors discovered a fundamental gap between machine and human vision. An "Analysis Network" was used to rank these components based on their contribution to recognition versus visualization. Crucially, the DC component—representing average block brightness and containing over 94% of the image's energy—is vital for human vision but largely redundant for deep learning models.

The Masking Pipeline

To secure the remaining high-frequency data, a multi-stage masking process is applied on the client side:

  1. Discard DC: The primary visual information (average brightness) is removed.
  2. Channel Shuffling & Mixing: Frequency channels are pseudo-randomly shuffled and linearly combined (mixed) to "blur" the data into a noise-like state.
  3. Self-Normalization: Images are normalized based on local statistics, preventing attackers from using global patterns for reconstruction.
The proposed PPFR-FD pipeline showing (a) the training phase with the analysis network and (b) the inference phase with the masking steps.

Experimental Results

The model was rigorously tested on massive benchmarks including MS-Celeb-1M for training and LFW, CFP-FP, and AgeDB for evaluation.

  • Accuracy: On the LFW dataset, PPFR-FD achieved 99.68% accuracy, nearly matching the 99.82% of the standard non-private ArcFace model. This demonstrates that the discarded 94% of image energy was not critical for recognition.
  • Efficiency: With an inference time of 2.23ms, the system is suitable for real-time applications and outperforms existing privacy-preserving competitors like CPGAN (21.37ms).
  • Security: The system proved resilient against white-box and GAN-based reconstruction attacks. Even with knowledge of the algorithm, attackers could only produce unrecognizable, fuzzy images.

Conclusion

The PPFR-FD scheme demonstrates that high-accuracy face recognition is possible without exposing human-readable biometric data. By discarding the energy-heavy DC components and scrambling the remaining high-frequency textures, the method provides a robust defense against privacy leaks. Future work will focus on biological template protection (revocability) and securing the initial preprocessing steps on the client device.

Summary updated

Abstract

Some applications may require performing face recognition (FR) on third-party servers, which could be accessed by attackers with malicious intents to compromise the privacy of users’ face information. This paper advocates a practical privacy-preserving FR scheme without key management realized in the frequency domain. The new scheme first collects the components of the same frequency from different blocks of a face image to form component channels. Only part of the channels are retained and fed into the analysis network that performs an interpretable privacy-accuracy trade-off analysis to identify channels important for face image visualization but not crucial for maintaining high FR accuracy. For this purpose, the loss function of the analysis network consists of the empirical FR error loss and a face visualization penalty term, and the network is trained in an end-to-end manner. We find that with the developed analysis network, more than 94% of the image energy can be dropped while the face recognition accuracy stays almost undegraded. In order to further protect the remaining frequency components, we propose a fast masking method. Effectiveness of the new scheme in removing the visual information of face images while maintaining their distinguishability is validated over several large face datasets. Results show that the proposed scheme achieves a recognition performance and inference time comparable to ArcFace operating on original face images directly.
BibTeX
@inproceedings{wang2022privacy,
  author = {Yinggui Wang and Jian Liu and Man Luo and Le Yang and Li Wang},
  title = {{Privacy-Preserving Face Recognition in the Frequency Domain}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=LgodwhAqP4},
}
Figure