Adversarial Sampling and Training for Semi-Supervised Information Retrieval

Dae Hoon Park, Yi ChangWWW 2018 · DOI · OpenReview

Abstract

Information Retrieval (IR) systems often struggle with messy implicit feedback, characterized by extreme class imbalance and data sparsity. This paper introduces AdvIR, a framework that leverages adversarial perturbations and selective sampling to enhance semi-supervised IR. By generating "hard" training examples and identifying informative non-clicked documents, AdvIR achieves significant performance gains and extreme data efficiency, outperforming baselines like IRGAN with only a fraction of the labeled data.

Introduction

Modern IR systems, such as web search engines and recommendation platforms, rely heavily on implicit user feedback (e.g., clicks). However, this data presents three primary challenges:

  1. Imbalance: Non-clicked documents vastly outnumber clicked ones, yet most are irrelevant and offer little learning signal.
  2. Sparsity: Positive labels (clicks) are rare, hindering model generalization.
  3. Model Vulnerability: Neural IR models are often sensitive to small input perturbations, making them vulnerable to ranking errors.

The AdvIR framework addresses these issues by using adversarial techniques to strengthen positive signals and identify the most informative negative examples.

Methodology

The AdvIR framework consists of several key components designed to improve robustness and efficiency:

Adversarial Training

The model generates adversarial perturbations—calculated noise added to input features—to maximize training loss. By learning to remain accurate under these "attacks," the model develops a smoother and more robust decision boundary.

Adversarial Sampling

Instead of random negative sampling, AdvIR uses a softmax-based approach to identify "difficult" negatives—documents the model incorrectly predicts as highly relevant. This ensures the model learns from the most informative examples.

Selective Virtual Adversarial Training (AdvIR-SVAT)

To reduce the computational cost of standard Virtual Adversarial Training (VAT), the authors propose Selective VAT. This method applies smoothing only to the most informative examples identified through adversarial sampling, maintaining performance while improving efficiency.

Key Results

AdvIR demonstrates superior performance across web search, item recommendation, and question-answering tasks.

Performance comparison of AdvIR variants against IRGAN_ReLU across multiple ranking metrics.

As shown in Figure 1, AdvIR with Adversarial Sampling (AS) consistently outperforms the IRGAN baseline and AdvIR with Uniform Sampling (US). The gains are most notable in high-rank precision; for web search, AdvIR achieved a 21.3% relative improvement in Precision@3 over the ReLU-based IRGAN.

Training convergence and performance vs. labeled data percentage for AdvIR compared to baselines.

The framework's data efficiency is a standout feature. As illustrated in Figure 4, AdvIR achieves higher NDCG@5 scores using only 5% of labeled data than IRGAN does with the full dataset. Furthermore, the learning curves (panels a and b) indicate that AdvIR converges to high precision significantly faster than its competitors, often reaching stability within fewer training epochs.

Conclusion

AdvIR provides a robust solution for semi-supervised information retrieval by combining adversarial training with intelligent sampling. Its ability to achieve state-of-the-art results with minimal labeled data makes it a powerful tool for real-world IR applications where high-quality labels are scarce. Future work may explore the interpretability of adversarial perturbations and address cold-start issues in recommendation systems.

Summary updated

Abstract

Ad-hoc retrieval models with implicit feedback often have problems, e.g., the imbalanced classes in the data set. Too few clicked documents may hurt generalization ability of the models, whereas too many non-clicked documents may harm effectiveness of the models and efficiency of training. In addition, recent neural network-based models are vulnerable to adversarial examples due to the linear nature in them. To solve the problems at the same time, we propose an adversarial sampling and training framework to learn ad-hoc retrieval models with implicit feedback. Our key idea is (i) to augment clicked examples by adversarial training for better generalization and (ii) to obtain very informational non-clicked examples by adversarial sampling and training. Experiments are performed on benchmark data sets for common ad-hoc retrieval tasks such as Web search, item recommendation, and question answering. Experimental results indicate that the proposed approaches significantly outperform strong baselines especially for high-ranked documents, and they outperform IRGAN in NDCG@5 using only 5% of labeled data for the Web search task.
BibTeX
@inproceedings{park2018adversarial,
  author = {Dae Hoon Park and Yi Chang},
  title = {{Adversarial Sampling and Training for Semi-Supervised Information Retrieval}},
  booktitle = {The Web Conference},
  year = {2018},
  doi = {10.1145/3308558.3313416},
}
Figure