Sentinel-Guided Zero-Shot Learning: A Collaborative Paradigm without Real Data Exposure

Fan Wan, Xingyu Miao, Haoran Duan, Jingjing Deng, Rui Gao, Yang Long

Abstract

With increasing concerns over data privacy and model copyrights, especially in the context of collaborations between AI service providers and data owners, an innovative SG-ZSL paradigm is proposed in this work. SG-ZSL is designed to foster efficient collaboration without the need to exchange models or sensitive data. It consists of a teacher model, a student model and a generator that links both model entities. The teacher model serves as a sentinel on behalf of the data owner, replacing real data, to guide the student model at the AI service provider's end during training. Considering the disparity of knowledge space between the teacher and student, we introduce two variants of the teacher model: the omniscient and the quasi-omniscient teachers. Under these teachers' guidance, the student model seeks to match the teacher model's performance and explores domains that the teacher has not covered. To trade off between privacy and performance, we further introduce two distinct security-level training protocols: white-box and black-box, enhancing the paradigm's adaptability. Despite the inherent challenges of real data absence in the SG-ZSL paradigm, it consistently outperforms in ZSL and GZSL tasks, notably in the white-box protocol. Our comprehensive evaluation further attests to its robustness and efficiency across various setups, including stringent black-box training protocol.

Introduction

Zero-Shot Learning (ZSL) aims to recognize objects from "unseen" classes by leveraging semantic attributes (e.g., descriptions like "has stripes") and knowledge learned from "seen" classes. Traditionally, this requires AI service providers to have direct access to large-scale labeled datasets. However, modern data privacy regulations (such as GDPR and CCPA) and the proprietary nature of medical or industrial data often prevent data owners from sharing raw images. Simultaneously, AI providers are reluctant to share their proprietary model architectures. This creates a deadlock for collaborative AI development.

The Sentinel-Guided Zero-Shot Learning (SG-ZSL) paradigm addresses this by enabling model training without the exchange of raw data or full model architectures.

Comparison between the traditional ZSL paradigm and the proposed SG-ZSL paradigm.

The SG-ZSL Framework

The SG-ZSL paradigm shifts the collaborative focus from data transmission to "sentinel" guidance. The framework involves three primary components:

  1. The Teacher (Data Sentinel): Hosted by the data owner, this model is pre-trained on sensitive, real-world data. It acts as a gatekeeper, providing feedback on synthetic data without ever exposing the original training samples.
  2. The Generator: Located at the AI service provider's end, the generator produces synthetic visual features based on semantic attributes (e.g., BERT embeddings).
  3. The Student: Also at the provider's end, the student model learns to classify these synthetic features, guided by the Teacher's feedback.

Security Protocols and Teacher Variants

To accommodate different privacy requirements, SG-ZSL supports two protocols:

  • White-Box: The teacher shares both its weights and output scores, allowing for gradient-based optimization. This offers higher performance but requires more transparency from the data owner.
  • Black-Box: The teacher only shares output classification scores (pseudo-labels), maximizing privacy for the data owner as the internal logic of the sentinel remains hidden.

Furthermore, the paradigm considers two teacher types: the Omniscient Teacher (aware of all classes) and the Quasi-Omniscient Teacher (aware only of seen training classes), the latter representing a more realistic scenario where the data owner's knowledge is limited to their specific domain.

Comparison with Existing Paradigms

SG-ZSL represents a significant departure from Inductive ZSL (IZSL) and Transductive ZSL (TZSL). While IZSL requires access to seen training data and TZSL requires both seen and unlabeled unseen data, SG-ZSL operates with zero access to real visual data at the AI provider's end.

Formal comparison of accessible data and weights across IZSL, TZSL, and SG-ZSL.

Experimental Evaluation

The framework was rigorously tested on three standard benchmarks: AWA1 (Animals with Attributes 1), AWA2, and aPY (Attribute Pascal and Yahoo), using BERT embeddings for semantic representation.

Key Findings

  • Performance: In the White-Box protocol, SG-ZSL achieved accuracies of 77.9% (AWA1) and 79.0% (AWA2), rivaling or beating traditional methods that have full access to real data.
  • Transcending the Sentinel: Remarkably, the student model can learn to recognize unseen classes even when the teacher is "Quasi-Omniscient" (i.e., has never seen them). The generator uses semantic maps to explore visual spaces beyond the teacher's original training.
  • Robustness: Even under the strict Black-Box protocol, the model significantly outperformed baseline expectations (e.g., ~33% accuracy on AWA1 vs. ~10% random), proving the viability of high-security collaborative training.

Conclusion and Future Directions

SG-ZSL demonstrates that high-quality synthetic features, refined by a sentinel's feedback, can effectively replace real pixels for ZSL training. While a performance gap remains between White-Box and Black-Box protocols, and communication overhead is a factor, this paradigm offers a robust path forward for privacy-preserving AI collaboration. Future work will focus on more advanced generative architectures and improving optimization under limited Black-Box feedback.

Summary updated

Abstract

With increasing concerns over data privacy and model copyrights, especially in the context of collaborations between AI service providers and data owners, an innovative SG-ZSL paradigm is proposed in this work. SG-ZSL is designed to foster efficient collaboration without the need to exchange models or sensitive data. It consists of a teacher model, a student model and a generator that links both model entities. The teacher model serves as a sentinel on behalf of the data owner, replacing real data, to guide the student model at the AI service provider's end during training. Considering the disparity of knowledge space between the teacher and student, we introduce two variants of the teacher model: the omniscient and the quasi-omniscient teachers. Under these teachers' guidance, the student model seeks to match the teacher model's performance and explores domains that the teacher has not covered. To trade off between privacy and performance, we further introduce two distinct security-level training protocols: white-box and black-box, enhancing the paradigm's adaptability. Despite the inherent challenges of real data absence in the SG-ZSL paradigm, it consistently outperforms in ZSL and GZSL tasks, notably in the white-box protocol. Our comprehensive evaluation further attests to its robustness and efficiency across various setups, including stringent black-box training protocol.
BibTeX
@article{wan2024sentinel,
  author = {Fan Wan and Xingyu Miao and Haoran Duan and Jingjing Deng and Rui Gao and Yang Long},
  title = {{Sentinel-Guided Zero-Shot Learning: A Collaborative Paradigm without Real Data Exposure}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2024},
  url = {https://openreview.net/forum?id=9M3WVaYrW4},
}
Figure