Knowledge Distillation with BERT for Image Tag-Based Privacy Prediction

Chenye Zhao, Cornelia CarageaRANLP 2021 · OpenReview

Abstract

Predicting image privacy is crucial for protecting social media users, yet many are hesitant to upload raw images to third-party services. This paper explores an alternative: predicting privacy using only image tags. We propose a framework that leverages BERT models enhanced by Knowledge Distillation (KD) and semi-supervised learning. Our results show that a distilled BERT model ($BERT_{KD}$) achieves state-of-the-art F1-scores and superior calibration. Notably, the semi-supervised approach allows the model to reach full-dataset performance using only 20% of the available labeled data, addressing the high cost of manual annotation.

Introduction

The rapid growth of social media has created a "privacy paradox." While users express concern over sensitive content being public, they are often unwilling to use automated privacy-prediction tools that require uploading raw image pixels to external servers. Image tags (e.g., "bedroom," "family," "vacation") offer a privacy-preserving alternative for analysis. However, tags are often sparse and lack the semantic context necessary for traditional models like Bag-of-Tags (BoT) or basic CNNs to accurately discern privacy boundaries.

This research addresses these challenges by applying Bidirectional Encoder Representations from Transformers (BERT) to the task of tag-based privacy prediction. By treating image tags as a sequence of text, we can leverage BERT's deep contextual understanding. To further improve efficiency and robustness, we incorporate Knowledge Distillation and semi-supervised learning techniques.

Methodology

The core of our approach is a Knowledge Distillation framework designed to transfer insights from a complex "Teacher" model to a "Student" model.

Knowledge Distillation Framework

We first train a Teacher BERT model on the labeled portion of the dataset. A Student BERT model, sharing the same architecture, is then trained using a composite loss function $L_S$. This loss combines "hard" labels (the ground truth $y$) and "soft" labels ($P_T$) provided by the Teacher. The soft labels represent the probability distribution over classes, capturing the Teacher's uncertainty and providing a richer signal for the Student to learn the decision boundary.

The Knowledge Distillation framework showing the interaction between the Teacher and Student BERT models.

Semi-Supervised Learning

To mitigate the need for extensive manual labeling, we employ the Teacher model to generate pseudo-labels for a large pool of unlabeled data. The Student model is then trained on this augmented dataset, allowing it to benefit from the underlying patterns in the unlabeled tags.

Experimental Setup

We evaluated our framework using the PicAlert dataset, which contains 32,000 Flickr images. These images were manually annotated by external viewers as either "private" (belonging to the personal sphere) or "public." The dataset was split into 22,000 training, 5,000 validation, and 5,000 testing samples. Pre-processing involved filtering rare tags and cleaning noise to ensure high-quality input for the BERT tokenizer.

Results and Discussion

Our experiments demonstrate significant improvements across several key metrics.

Performance and Data Efficiency

The distilled model, $BERT_{KD}$, achieved a private-class F1-score of 0.681, outperforming standard BERT (0.664) and baseline models like TagCNN (0.629). A critical finding is the efficiency of the semi-supervised approach. As shown in Figure 2, the semi-supervised BERT model reaches the performance levels of a fully supervised model using only 20% of the labeled training data (approximately 4,400 samples).

F1-score for the private class as a function of the amount of labeled data (k). The semi-supervised BERT model (red solid line) rapidly approaches the supervised baseline (red dashed line).

Calibration and Robustness

Beyond accuracy, $BERT_{KD}$ exhibited better calibration, reducing Expected Calibration Error (ECE) by up to 2.62%. This ensures that the model's confidence scores are more reliable indicators of actual correctness. Furthermore, the model proved highly robust against adversarial "Synonym-based Attacks," maintaining an F1-score of 0.962 when tags were replaced with similar words, whereas standard models showed significant degradation.

Semantic Insights

Our analysis revealed a strong link between tag emotions and privacy. Tags associated with Trust, Joy, and Disgust (e.g., "innocent," "messy") were predictive of private images, while tags evoking Fear and Surprise (e.g., "caution," "marvel") were more common in public content.

Conclusion

This work demonstrates that BERT, combined with Knowledge Distillation, is a powerful tool for privacy prediction based solely on image tags. Our framework not only improves accuracy but also significantly reduces the reliance on labeled data through semi-supervised learning. Future work will explore multi-modal architectures that combine tag-based analysis with selective image pixel processing to handle cases where tags are missing or ambiguous.

Summary updated

BibTeX
@inproceedings{zhao2021knowledge,
  author = {Chenye Zhao and Cornelia Caragea},
  title = {{Knowledge Distillation with BERT for Image Tag-Based Privacy Prediction}},
  booktitle = {RANLP},
  year = {2021},
  url = {https://openreview.net/forum?id=7f6Oz-jcqcn},
}
Figure