A Deep Relevance Model for Zero-Shot Document Filtering

Chenliang Li, Wei Zhou, Feng Ji, Yu Duan, Haiqing ChenACL 2018 · DOI

Abstract

In the era of massive data streams, the ability to filter documents for emerging topics without extensive labeled training data is critical. This paper introduces DAZER (Deep Relevance Model for Zero-Shot Document Filtering), a model that utilizes conceptual "seed words" to identify relevant documents in unseen categories. By integrating interaction-based representations, category-specific gating, and adversarial learning, DAZER significantly outperforms existing relevance models in zero-shot scenarios, achieving over 30% improvement in ranking accuracy.

Introduction

Traditional supervised learning for document classification requires thousands of labeled examples per category. However, in dynamic environments where new topics emerge daily, such data is often unavailable. While "dataless" classification exists, it typically focuses on multi-class assignment rather than document filtering—a ranking task aimed at extracting the most relevant items from a large corpus.

Existing deep relevance models, primarily designed for web search, often rely on exact keyword matching. These models struggle in zero-shot document filtering where the "query" consists of a few conceptual seed words that may not appear verbatim in the documents. DAZER addresses this by learning high-level relevance signals that generalize across categories.

The DAZER Architecture

The DAZER model is structured into three primary stages: interaction, relevance signal extraction, and relevance aggregation.

The DAZER architecture overview.

1. Interaction-based Representation

Rather than relying solely on cosine similarity or raw word embeddings, DAZER calculates element-wise interactions between document words and the average embedding of the category's seed words. Specifically, it uses element-wise difference to capture vector space "offsets" and element-wise products to capture semantic alignment.

These interactions allow the model to recognize conceptual relationships. For example, the relationship between "atheism" and "atheist" can be mapped to the relationship between "baseball" and "hitter" through vector arithmetic, as shown in the examples below.

Conceptual relationships captured by vector interactions.

2. Category-specific Gating

To adapt to the nuances of different topics, DAZER employs a gating mechanism. A small neural network processes the category embedding to produce a gate vector, which selectively activates specific convolutional filters. This allows the model to dynamically adjust its feature extraction logic based on the target category, acting as a "chameleon" that specializes its internal representations for the task at hand.

3. Adversarial Learning for Generalization

To prevent the model from overfitting to the specific vocabulary of the training categories, DAZER incorporates a Gradient Reversal Layer (GRL). This adversarial component forces the model to learn relevance features that are independent of the specific category labels. By making it difficult for the network to identify the category from the intermediate representations, the model is incentivized to learn universal signals of "relevance."

Experimental Evaluation

The model was evaluated on two benchmarks:

  • 20-Newsgroup (20NG): A standard dataset for topic-based document filtering (e.g., electronics, space).
  • Movie Review (v1.0): A dataset for sentiment-based filtering across five levels of sentiment.

In both experiments, the model was trained on a subset of categories and tested on entirely disjoint, unseen categories to simulate a true zero-shot environment.

Key Results and Discussion

DAZER demonstrated substantial improvements over traditional baselines like BM25 and modern deep learning models such as K-NRM and DRMM.

  • Ranking Performance: DAZER achieved an average Mean Average Precision (MAP) of 0.671 across 14 tasks, a significant lead over the next best model (K-NRM at 0.548).
  • Topic-Specific Gains: In categories like "medicine," DAZER improved MAP by nearly 25% compared to alternative methods.
  • Ablation Study: Removing the element-wise product interaction led to a sharp decline in performance on the 20NG dataset, confirming that high-dimensional interactions are essential for capturing conceptual relevance in topic filtering.

Conclusion and Future Directions

DAZER represents a significant step forward in zero-shot document filtering by moving beyond simple keyword matching to conceptual relevance. While highly effective, the model's performance is sensitive to the choice of seed words. Future work could explore the use of contextual embeddings like BERT or the integration of pseudo-relevance feedback to automatically expand the initial seed word set, further enhancing the model's robustness and accuracy.

Summary updated

BibTeX
@inproceedings{li2018deep,
  author = {Chenliang Li and Wei Zhou and Feng Ji and Yu Duan and Haiqing Chen},
  title = {{A Deep Relevance Model for Zero-Shot Document Filtering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.18653/v1/p18-1214},
}
Figure