Adaptive Feature Discrimination and Denoising for Asymmetric Text Matching

Yan Li, Chenliang Li, Junjun GuoCOLING 2022

TL;DR

This paper introduces ADDAX, a novel model for asymmetric text matching that explicitly identifies and filters out irrelevant features in text pairs. By employing a matching-adapted gating siamese cell for feature discrimination and a locality-constrained hashing denoiser for noise reduction, ADDAX significantly improves performance on various downstream tasks.

What Problem Does This Paper Address?

Asymmetric text matching, where one text is significantly shorter than the other (e.g., a query and a long document), is crucial for many applications like information retrieval and question answering. Existing methods often focus on modeling feature interactions but overlook the critical step of identifying and leveraging discriminative features while discarding noisy or irrelevant ones. This asymmetry poses a challenge because the shorter text might contain less information, making it harder to pinpoint the relevant parts of the longer text. The paper aims to bridge this gap by proposing a method that actively discriminates important features and denoises the representations, especially for the longer document.

Methods

The proposed ADDAX model consists of four main components:

  • BERT-based Context Encoder: Utilizes a pre-trained BERT model to generate contextualized word-level representations for both the query and the document. A single encoder is shared for both to reduce parameters and facilitate feature interaction.
  • Matching-Adapted Gating Siamese Cell (MAGS): This novel component is designed to simulate human intuition in identifying relevant information. It uses a cross-attention mechanism to create "reference representations" of the query based on the document's content. It then computes divergence features and uses a gating mechanism to combine these with original representations, producing "hybrid representations." A highway network further refines these into "discriminative features."
  • Locality-Constrained Hashing Denoiser: Given that longer documents can introduce more noise, this component is applied to the document's discriminative features. It acts as an autoencoder that projects the features into a low-dimensional binary space, preserving relevant semantics while filtering out noise. This process is guided by a reconstruction loss to ensure semantic preservation.
  • MaxSim-based Relevance Predictor: A standard MaxSim operator is used to calculate the final relevance score between the processed query and document representations.

The model is optimized using a triplet-based hinge loss combined with constraints for the hashing denoiser.

Datasets

The paper evaluates ADDAX on four real-world datasets:

  • InsuranceQA: A widely used dataset for question answering.
  • WikiQA: An open-domain answer selection dataset.
  • YahooQA2: A collection derived from Yahoo! Answers, filtered for asymmetric text pairs.
  • MS MARCO: A benchmark dataset for information retrieval.

These datasets are chosen because they exhibit the characteristic asymmetry in text length, with query-document length ratios greater than 3.

Key Insights and Ideas

  • Explicit Feature Discrimination is Key: The paper argues that simply modeling feature interactions isn't enough for asymmetric text matching. Actively identifying and isolating discriminative features is crucial for better relevance learning.
  • Denoising for Long Documents: The asymmetry itself, particularly the length disparity, introduces noise in the longer document. A dedicated denoising mechanism is shown to be effective in mitigating this.
  • Simulating Human Intuition: The MAGS component attempts to mimic how humans subconsciously filter information, using attention and divergence to pinpoint relevant semantic signals.
  • Hashing for Efficient Denoising: Using hashing as a form of dimensionality reduction and noise filtering offers a computationally efficient way to process the potentially large feature space of long documents.

Key Results

ADDAX consistently outperforms state-of-the-art baselines across the evaluated datasets and tasks.

  • On InsuranceQA, ADDAX achieves substantial performance gains, with a relative P@1 gain of 3.23% over the best baseline on YahooQA.
  • On MS MARCO, ADDAX shows significant improvements, with performance gains ranging from 1.25% to 17.40% in terms of MRR@10.
  • Ablation studies reveal that removing the MAGS component leads to the largest performance degradation, highlighting its importance. Excluding the hashing denoiser also results in a notable drop, underscoring the effectiveness of the denoising step.

Limitations and Future Work

  • Limitations: The paper notes that while MAGS is effective for the query side, they did not observe improvement by applying it to the query side when the document is short. They also mention that applying the hashing denoiser to the short query did not yield improvements, suggesting its primary benefit is for longer documents.
  • Future Work: The authors plan to explore feature discrimination and denoising in other asymmetric scenarios, such as document abstractive summarization, caption generation, and more.

Summary updated

BibTeX
@inproceedings{li2022adaptive,
  author = {Yan Li and Chenliang Li and Junjun Guo},
  title = {{Adaptive Feature Discrimination and Denoising for Asymmetric Text Matching}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
}
Figure