Privacy-Preserving Deep Action Recognition: An Adversarial Learning Framework and A New Dataset

Zhenyu Wu, Haotao Wang, Zhaowen Wang, Hailin Jin, Zhangyang WangIEEE TPAMI 2022 · DOI

TL;DR

This paper introduces a novel adversarial learning framework for privacy-preserving action recognition in videos. It tackles the challenge of balancing utility (action recognition) with privacy (preventing leakage of sensitive attributes) by learning to anonymize videos. The authors also introduce PA-HMDB51, a new dataset specifically designed for evaluating this task.

What Problem Does This Paper Address?

Smart cameras are becoming ubiquitous, offering convenience through video analysis. However, this convenience comes at the cost of privacy, as sensitive information like faces, gender, or even nudity can be captured and potentially misused. Traditional methods of anonymization (like blurring or low resolution) often degrade the utility of the video for its intended purpose (e.g., action recognition). This paper addresses the critical need for a method that can perform action recognition effectively while simultaneously obscuring sensitive personal information from the video data. The challenge lies in optimizing this trade-off, especially against a wide range of potential attackers (privacy models) rather than just one.

Methods

The core of the proposed solution is an adversarial learning framework. This framework involves three main components:

  • Anonymization Model ($f_A$): Learns to transform raw video frames into "anonymized" versions.
  • Target Utility Model ($f_T$): Performs the desired task, such as action recognition, on the anonymized video.
  • Privacy Budget Model ($f_B$): Attempts to predict sensitive privacy attributes from the anonymized video.

The framework is trained to maximize the utility task performance while minimizing the privacy leakage, formulated as a minimax optimization problem. To handle the "universal privacy protection" challenge (defending against any potential $f_B$), the authors propose:

  • Model Restarting: Periodically re-initializing the privacy model $f_B$ to prevent the anonymization model $f_A$ from overfitting to a specific attacker.
  • Model Ensemble: Using a collection of multiple privacy models to approximate the space of all possible attackers, making the anonymization more robust.

They also explore three optimization strategies: Gradient Reverse Layer (GRL), Ours-K-Beam (an alternating optimization approach), and Ours-Entropy (which replaces a concave loss with a convex one for better stability).

Datasets

  • UCF101: A standard action recognition dataset used for training and evaluating the utility task.
  • HMDB51: Another action recognition dataset used for training and evaluation.
  • VISPR: A dataset containing images with 68 privacy attributes, used for training privacy prediction models and for cross-dataset evaluation.
  • PA-HMDB51 (Privacy-Annotated HMDB51): A newly constructed dataset introduced by the authors. It consists of 515 videos from HMDB51, with per-frame annotations for five privacy attributes: skin color, face, gender, nudity, and relationship. This dataset is crucial for evaluating both utility and privacy simultaneously on the same video data.

Key Insights and Ideas

  • The "Universal Privacy" Challenge: Simply defending against one specific privacy prediction model is insufficient. True privacy protection requires robustness against any potential attacker model. This is a key differentiator from many prior works.
  • Adversarial Trade-off Optimization: The core idea is to frame privacy preservation as an adversarial game where the anonymization model tries to fool privacy detectors while still allowing the utility model to function.
  • Dataset Scarcity: The lack of datasets with both action and privacy labels is a major bottleneck. The creation of PA-HMDB51 directly addresses this gap.
  • Ours-Entropy's Stability: Replacing the negative cross-entropy loss with negative entropy for the privacy budget task significantly improved training stability, leading to better performance.
  • Model Restarting and Ensemble are Crucial: These techniques are shown to be effective in mitigating "attacker overfitting" and improving the generalization of the privacy protection.

Key Results

  • Improved Trade-off: The proposed Ours-Entropy method, especially with model restarting and ensemble, achieved a significantly better trade-off between action recognition accuracy ($A_T$) and privacy attribute accuracy ($A_B$) compared to naive downsampling and empirical obfuscation methods. For example, on SBU, Ours-Entropy decreased $A_B$ by around 30% with minimal impact on $A_T$.
  • PA-HMDB51 Performance: On the new PA-HMDB51 dataset, the proposed methods showed superior performance, decreasing privacy cMAP by about 8% with minimal harm to utility accuracy when using M=4 ensemble models.
  • Human Study Validation: A human study confirmed that anonymized videos were still recognizable for actions but effectively obscured privacy attributes, validating the learned anonymization transform.

Limitations and Future Work

  • Privacy Budget Definition: The authors acknowledge that the definition of privacy leakage ($J_B$) is not straightforward and can be application-specific. The current ensemble approach is an approximation of the continuous space of attackers. More sophisticated methods for approximating this space are suggested for future work.
  • Dataset Size: The PA-HMDB51 dataset, while novel, is currently limited in size, primarily serving as a testing benchmark. The authors aim to annotate more data to expand it for training in the future.
  • Cost Asymmetry: Different privacy attributes have different protection costs. Future work could explore weighted loss functions or dedicated protectors for more informative attributes.

Summary updated

BibTeX
@article{wu2022privacy,
  author = {Zhenyu Wu and Haotao Wang and Zhaowen Wang and Hailin Jin and Zhangyang Wang},
  title = {{Privacy-Preserving Deep Action Recognition: An Adversarial Learning Framework and A New Dataset}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  doi = {10.1109/tpami.2020.3026709},
}
Figure