Interpretable Privacy Preservation of Text Representations Using Vector Steganography

Geetanjali BihaniAAAI 2022 · DOI

Abstract

Language models often inadvertently learn sensitive social attributes that can be exploited by adversaries. This research proposes Vector Space Steganography (VSS) to hide these associations within text embeddings, aiming for a balance between privacy, model utility, and interpretability.

Introduction

Modern Natural Language Processing (NLP) relies on high-dimensional vector representations. While powerful, these models often capture subtle linguistic patterns that reveal private author attributes, such as identity or political affiliation. Current privacy-preserving methods, including differential privacy and cryptographic approaches, often suffer from high computational costs or reduced model utility. This work introduces Vector Space Steganography (VSS) as a more efficient and interpretable alternative.

Proposed Methodology

Vector Space Steganography (VSS) modifies the internal geometry of text vectors to obfuscate sensitive signals while preserving semantic meaning. The approach involves:

  • Reversing Steganalysis: Utilizing steganalysis criteria to generate steganographic candidates within the vector space.
  • Micro-aggregation: Obfuscating word representations by aggregating these candidates, thereby hiding latent confounds (private signals) while maintaining distributional semantics.
  • Targeted Application: Focusing on specific model layers where sensitive associations are most prominent, rather than applying a blanket fix to the entire model.

Key Findings and Insights

Preliminary research highlights several critical aspects of privacy in language models (LMs):

  1. Architecture Sensitivity: Social associations are model-dependent; different architectures (e.g., bidirectional vs. unidirectional) encode these signals in unique ways, precluding a "universal" scrubbing list.
  2. Layer-wise Leakage: Sensitive information is more prevalent in the lower layers of LMs and diminishes in upper layers due to contextualization. Consequently, privacy interventions should be concentrated at earlier processing stages.
  3. Geometric Obfuscation: By modifying the vector space geometry rather than scrambling data, VSS allows models to maintain performance on downstream tasks while shielding private attributes from adversarial probes.

Research Objectives and Timeline

The proposed research is divided into three main objectives to design, test, and prove the effectiveness of the VSS framework.

Timeline of Research Objectives (RO1-RO3) for the development and validation of the VSS framework.
  • RO1 (Design): Finalization of the VSS algorithm for association obfuscation.
  • RO2 (Utility): Utility testing on standard Natural Language Understanding (NLU) benchmarks to ensure model accuracy is preserved.
  • RO3 (Provability): Evaluation against state-of-the-art reconstruction attacks to establish robust privacy guarantees.

Conclusion

VSS represents a promising middle ground between the "black box" nature of adversarial learning and the heavy computational overhead of cryptography. By leveraging the geometric properties of vector spaces, this research aims to provide a robust and interpretable solution for privacy preservation in modern NLP.

Summary updated

BibTeX
@inproceedings{bihani2022interpretable,
  author = {Geetanjali Bihani},
  title = {{Interpretable Privacy Preservation of Text Representations Using Vector Steganography}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21573},
}
Figure