Deep Manifold Learning for Reading Comprehension and Logical Reasoning Tasks with Polytuplet Loss

Jeffrey Lu, Ivan Rodriguez

TL;DR

This paper introduces "polytuplet loss," a novel loss function for reading comprehension and logical reasoning tasks that prioritizes learning relative answer correctness over absolute accuracy. By framing these tasks as a manifold learning problem, their approach aims to mimic human strategies that focus on comparing answer choices, leading to improved performance on the challenging ReClor benchmark.

What Problem Does This Paper Address?

Reading comprehension and logical reasoning (RCLR) are crucial for extracting information from text, especially for large language models (LLMs). Current approaches often focus on understanding the text and question to determine the absolute truthfulness of each answer choice. However, humans often employ a strategy of comparing answer choices to each other, which can be effective even with incomplete knowledge. This paper addresses the gap by proposing a method that explicitly models this "answer-first" strategy, aiming to improve model performance by learning relative answer correctness.

Methods

The core contribution is the polytuplet loss function, an extension of triplet loss. While triplet loss aims to pull positive examples closer to an anchor and push negative examples further away, polytuplet loss focuses on learning embeddings in a lower-dimensional manifold. Specifically, it projects context, question, and answer embeddings such that the relative distances between answer embeddings and the context embedding are prioritized. This means the model learns to identify which answer is most correct relative to the context, rather than just whether each answer is absolutely correct.

The authors also introduce an index-blind architecture. This design choice flattens the answer choice and batch size dimensions, processing each answer choice independently. This prevents the model from simply memorizing the index of the correct answer and encourages it to learn meaningful embeddings based on the content.

For training, they employ hard and semi-hard negative mining to improve the quality of convergence, selecting negative samples that are close to the decision boundary.

Datasets

The primary dataset used for evaluation is ReClor. This is a challenging reading comprehension and logical reasoning benchmark consisting of multiple-choice questions with accompanying contexts. Each data point includes a context, a question, and four answer choices, where exactly one is correct, and the others are logically or factually inaccurate. The ReClor questions are sourced from past GMAT and LSAT exams.

Key Insights and Ideas

  • Mimicking Human Strategy: The paper's central idea is that focusing on relative answer correctness (comparing choices) can be more effective than solely relying on absolute accuracy, mirroring a common human approach to these tasks.
  • Manifold Learning for Reasoning: Framing RCLR as a manifold learning problem, where embeddings capture relative relationships, offers a novel perspective beyond traditional classification.
  • Interpretable Components: The polytuplet loss and associated architecture are designed to be more interpretable than some existing models by explicitly modeling the comparison of answer choices.
  • Robustness to Incomplete Knowledge: The "answer-first" approach, emphasized by polytuplet loss, theoretically allows models to identify correct answers even with partial understanding of the text.

Key Results

The polytuplet models consistently outperformed baseline models across various pretrained backbones (ALBERT, BERT, DistilBERT).

  • On ALBERT (base), polytuplet loss improved accuracy by 5.6% (from 53.4% to 56.4%).
  • On BERT (base), the improvement was more significant at 11.7% (from 44.6% to 49.8%).
  • On DistilBERT (base), the improvement was 6.5% (from 44.8% to 47.7%).

Similar improvements were observed on larger model complexities (ALBERT xx-large, RoBERTa large), indicating the loss function's effectiveness across different model sizes.

Limitations and Future Work

  • Computing Resources: The authors acknowledge severe limitations in computing resources, which restricted their experiments to smaller pretrained models. This prevented them from testing on state-of-the-art larger models like DeBERTa.
  • Focus on Polytuplet Loss: The analysis did not incorporate logic-targeting pretraining or advanced data augmentation techniques used in prior work, focusing solely on the impact of polytuplet loss and its architecture.
  • Further Investigation Needed: More research is required to fully quantify the benefits of polytuplet loss and to compare it against other state-of-the-art contrastive learning loss functions.
  • Overfitting: While dropout was used, all models showed signs of overfitting due to model complexity and dataset size. Polytuplet models showed a trend of performing optimally at lower dropout rates, suggesting potential inherent resistance to overfitting, but this requires further study.

Overall, the paper presents polytuplet loss as a promising alternative for RCLR tasks, offering a new perspective on how models can learn to reason by comparing answer choices.

Summary updated

BibTeX
@inproceedings{lu2024deep,
  author = {Jeffrey Lu and Ivan Rodriguez},
  title = {{Deep Manifold Learning for Reading Comprehension and Logical Reasoning Tasks with Polytuplet Loss}},
  booktitle = {Ficc},
  year = {2024},
  doi = {10.1007/978-3-031-53963-3\_23},
}
Figure