Data Augmentation Based on Adversarial Autoencoder Handling Imbalance for Learning to Rank

Qian Yu, Wai LamAAAI 2019 · OpenReview

Abstract

Data imbalance is a key limiting factor for Learning to Rank (LTR) models in information retrieval. Resampling methods and ensemble methods cannot handle the imbalance problem well since none of them incorporate more informative data into the training procedure of LTR models. We propose a data generation model based on Adversarial Autoencoder (AAE) for tackling the data imbalance in LTR via informative data augmentation. This model can be utilized for handling two types of data imbalance, namely, imbalance regarding relevance levels for a particular query and imbalance regarding the amount of relevance judgements in different queries. In the proposed model, relevance information is disentangled from the latent representations in this AAE-based model in order to reconstruct data with specific relevance levels. The semantic information of queries, derived from word embeddings, is incorporated in the adversarial training stage for regularizing the distribution of the latent representation. Two informative data augmentation strategies suitable for LTR are designed utilizing the proposed data generation model. Experiments on benchmark LTR datasets demonstrate that our proposed framework can significantly improve the performance of LTR models.

Introduction

In Information Retrieval, "Learning to Rank" (LTR) models are often hamstrung by two types of data imbalance. First, R-imbalance (Relevance) occurs because for any given query, there are thousands of irrelevant documents but only a handful of highly relevant ones. Models struggle to learn what "highly relevant" looks like because they rarely see it. Second, Q-imbalance (Query) arises when some queries in a training set have hundreds of judged documents, while others have only a few. Queries with more data tend to dominate the model's learning process, leading to poor generalization on "quieter" queries.

Standard fixes like oversampling or SMOTE often lead to overfitting because they don't introduce truly new, "informative" information. The authors argue that a generative approach is needed to create diverse, controllable synthetic training samples that follow the underlying data distribution.

Proposed Methodology

The authors propose an Informative Data Generation Model based on an Adversarial Autoencoder (AAE). This model is designed to generate synthetic data that addresses both relevance and query-level imbalances.

The proposed AAE-based data generation model architecture.

The architecture consists of three main components:

  • The Encoder-Decoder (Disentanglement): The encoder maps document features into a latent representation ($z$). Crucially, the decoder is fed both this latent vector and a specific relevance indicator ($r$). This "disentangles" the document's content from its relevance, allowing the model to take a real document and "reconstruct" what it would look like if it were assigned a different relevance level.
  • The Discriminator (Adversarial Training): To ensure the latent space is well-behaved, a discriminator forces the encoder's output to match a prior distribution (a mixture of Gaussians).
  • Query Type Discovery: To handle Q-imbalance, the authors use GloVe word embeddings to cluster queries into "types." The AAE is conditioned on these query types, ensuring that the generated data preserves the semantic characteristics of specific categories of searches.

Two strategies are proposed: Augment(R), which generates pseudo-pairs at different relevance levels, and Augment(R+Q), which additionally generates samples for under-represented query types by sampling from the learned latent prior.

Experimental Evaluation

The model was evaluated on standard benchmark LTR datasets: MQ2007, MQ2008, and MSLR-WEB10K. These datasets provide a range of relevance levels and query counts to test the robustness of the augmentation strategies.

Key Results

The authors tested their augmentation on five major LTR models: RankSVM, RankNet, ListNet, AdaRank, and LambdaMart.

  • Universal Improvement: Both augmentation strategies significantly outperformed the original datasets and traditional resampling methods across almost all metrics (P@5, NDCG@5, MAP).
  • LambdaMart Excellence: LambdaMart combined with the Augment(R+Q) strategy achieved the best overall performance, improving NDCG@5 by over 3% on the MQ datasets.
  • Robustness: The improvements were statistically significant. The "Augment(R+Q)" strategy was consistently better than "Augment(R)" alone, proving that addressing query-level imbalance is vital for overall ranking performance.

Discussion and Limitations

The "secret sauce" of this approach is disentanglement. By separating relevance from latent features, the model performs "counterfactual" generation—asking, "What would this document look like if it were a perfect match?" Unlike simple oversampling, the AAE generates new feature combinations that follow the underlying data distribution, providing a smoother, more diverse decision boundary.

However, the performance is sensitive to hyperparameters such as the latent dimension and the number of query clusters.

Sensitivity of LTR performance to the latent dimension on MQ2007 and MQ2008 datasets.

As shown in the figure above, LTR P@1 performance peaks at a specific latent dimension (around 10 for these datasets) and drops if the dimension is set too high or too low. Additionally, the study focuses on pre-extracted features rather than end-to-end deep ranking from raw text. Future work could explore the computational overhead of training the AAE and its applicability to raw text inputs.

Summary updated

Abstract

Data imbalance is a key limiting factor for Learning to Rank (LTR) models in information retrieval. Resampling methods and ensemble methods cannot handle the imbalance problem well since none of them incorporate more informative data into the training procedure of LTR models. We propose a data generation model based on Adversarial Autoencoder (AAE) for tackling the data imbalance in LTR via informative data augmentation. This model can be utilized for handling two types of data imbalance, namely, imbalance regarding relevance levels for a particular query and imbalance regarding the amount of relevance judgements in different queries. In the proposed model, relevance information is disentangled from the latent representations in this AAE-based model in order to reconstruct data with specific relevance levels. The semantic information of queries, derived from word embeddings, is incorporated in the adversarial training stage for regularizing the distribution of the latent representation. Two informative data augmentation strategies suitable for LTR are designed utilizing the proposed data generation model. Experiments on benchmark LTR datasets demonstrate that our proposed framework can significantly improve the performance of LTR models.
BibTeX
@inproceedings{yu2019data,
  author = {Qian Yu and Wai Lam},
  title = {{Data Augmentation Based on Adversarial Autoencoder Handling Imbalance for Learning to Rank}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=KQ932ftlOp},
}
Figure