Improved Text Matching by Enhancing Mutual Information

Yang Liu, Wenge Rong, Zhang XiongAAAI 2018 · DOI

TL;DR

This paper proposes a novel approach to improve text matching (crucial for tasks like question answering) by reformulating the original text using Generative Adversarial Networks (GANs). By generating semantically equivalent paraphrases, they enhance the mutual information between text pairs, leading to consistently better matching performance across various models.

What Problem Does This Paper Address?

Text matching is a fundamental problem in Natural Language Processing (NLP), underpinning applications like question answering (QA) and information retrieval (IR). Existing methods often focus on building more sophisticated matching models to capture deeper semantic relationships. However, the authors identify a limitation: even advanced models might struggle when the original text (e.g., a question) has limited semantic overlap with the target text (e.g., an answer), despite conveying the same intent. This "semantic gap" hinders effective matching. The paper aims to bridge this gap by enriching the input text itself, rather than solely improving the matching mechanism.

Methods

The core of the proposed method is Question Rewriting using a Generative Adversarial Network (GAN).

  • GAN Framework: A generator (G) is trained to produce paraphrased questions, while a discriminator (D) is trained to distinguish between original and generated questions. This adversarial process encourages G to generate realistic paraphrases.
  • Addressing Non-Differentiability: A key challenge in applying GANs to text is that word generation involves a non-differentiable "argmax" operation. The authors overcome this by employing policy gradients (specifically, using the discriminator's output as a reward signal) to update the generator's parameters. This allows the generator to learn from the discriminator's feedback.
  • Matching Model: After question rewriting, the original question ($q_o$) and the rewritten question ($q_r$) are fed into a matching model along with the answer. The matching model uses:
  • Bidirectional GRUs: To learn context-aware word representations.
  • Fusion Matrix: A bilinear transformation is used to create a matrix representing the interaction between question and answer representations.
  • CNN Feature Extractor: A Convolutional Neural Network (CNN) is applied to the fusion matrix to extract relevant features.
  • Concatenation and MLP: Features from the original question, rewritten question, and the CNN output are concatenated and fed into a Multi-Layer Perceptron (MLP) to produce a final relevance score.
  • Ranking Loss: For tasks involving ranking multiple answers, a LambdaRank loss function is used, which is more suitable for learning relative orderings than simple classification losses.

Datasets

The authors evaluate their method on several datasets:

  • Zhihu: A new, multi-level annotated dataset constructed from the Zhihu platform, featuring questions with answers rated on a five-level scale based on user thumb-ups. This dataset is used for both training and testing.
  • SemEval 2016 Task 3: A dataset for answer selection.
  • WikiQA: Another dataset for answer selection.

Key Insights and Ideas

  • Enhancing Mutual Information: The core idea is that by generating a paraphrase ($q_r$), the mutual information between the question and answer ($I(q_o, A)$) can be enhanced to $I(q_o, q_r, A)$. This is because $q_r$ can capture aspects of the question's intent that might be missed by $q_o$ alone, thus providing a richer signal for matching.
  • Rewriting vs. Better Matching: The paper demonstrates that improving the input text (rewriting) can be as effective, if not more so, than solely improving the matching model itself. This offers a complementary strategy for tackling text matching challenges.
  • Ranking vs. Classification: For tasks with multiple candidate answers, listwise ranking approaches (like LambdaRank) consistently outperform classification-based methods, highlighting the importance of considering the relationships between all answers simultaneously.
  • Visualizing Alignment: The visualization of the fusion matrix provides an intuitive understanding of how the model aligns words between the question and answer, showing that similar word pairs achieve better alignment.

Key Results

  • Consistent Improvement: Across all tested matching models and datasets (SemEval, WikiQA, ZhihuRank), the inclusion of the rewritten question consistently improved performance (measured by MAP, MRR, NDCG, and ERR).
  • ZhihuRank Performance: On the newly introduced ZhihuRank dataset, the "Rewrite + Rank" approach achieved the highest NDCG (0.7723) and ERR (0.6502) among the evaluated systems, outperforming strong baselines like ARC-II and Attentive LSTM.
  • Ranking Advantage: Ranking-based approaches generally outperformed classification-based approaches on all datasets. For example, on ZhihuRank, "Rewrite + Rank" achieved 0.7723 NDCG, while "Rewrite + Classify" achieved 0.7451 NDCG.

Limitations and Future Work

  • Computational Cost: The GAN-based rewriting process adds computational overhead.
  • Quality of Rewriting: The effectiveness of the approach relies on the quality of the generated paraphrases. While policy gradients help, generating truly diverse and semantically rich paraphrases remains an ongoing challenge.
  • Future Work: The authors suggest that this reformulation mechanism is broadly applicable to other text matching scenarios beyond QA, such as paraphrase identification and textual entailment. They also imply that further exploration of GANs for text generation in NLP tasks is warranted.

Summary updated

BibTeX
@inproceedings{liu2018improved,
  author = {Yang Liu and Wenge Rong and Zhang Xiong},
  title = {{Improved Text Matching by Enhancing Mutual Information}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11948},
}
Figure