Polytuplet Loss: A Reverse Approach to Training Reading Comprehension and Logical Reasoning Models

Jeffrey Lu, Ivan Rodriguez

Abstract

This paper introduces Polytuplet Loss, a novel loss function that shifts the focus of Reading Comprehension and Logical Reasoning (RCLR) models from determining absolute answer accuracy to evaluating relative correctness. By training models to project contexts and answers into a shared manifold where the correct choice is pulled toward the context and incorrect ones are pushed away, the authors achieved performance gains of up to 11.7% over standard baselines.

Introduction

Most current machine learning models approach multiple-choice logical reasoning tasks with a "prompt-first" mentality: they try to calculate the absolute probability that a specific answer is true based on the context. However, this requires a very high information threshold—the model must fully understand the logic to "prove" an answer.

The authors argue that humans often use an "answer-first" strategy. If you’ve ever taken a standardized test like the LSAT, you know that you can often identify the correct answer by comparing the choices and eliminating those that are logically inconsistent, even if you don't fully grasp every nuance of the text. This paper aims to bridge this gap by teaching models to prioritize the relative differences between answer choices rather than treating each choice as an independent true/false question.

Methodology

The core contribution is the Polytuplet Loss, an extension of the well-known Triplet Loss used in computer vision.

  1. Manifold Learning: The model projects the "Anchor" (the context and question) and the "Choices" (the potential answers) into a lower-dimensional space (a manifold).
  2. The Loss Function: While Triplet Loss compares one positive and one negative sample, Polytuplet Loss compares the anchor against the correct answer and all incorrect answers simultaneously. It forces the distance between the anchor and the correct answer to be smaller than the distance to any incorrect answer by at least a margin $m$.
  3. Index-blind Architecture: To prevent the model from simply memorizing the position of the correct answer, the authors used an architecture that flattens the batch and answer dimensions. This ensures the model evaluates the relationship between the context and the answer choice without knowing which "slot" (A, B, C, or D) the answer occupies.
  4. Hard and Semi-Hard Negative Mining: The authors implemented a strategy to focus training on "difficult" examples—cases where an incorrect answer is already very close to the anchor or where the correct answer is further away than a distractor.
Visualization of Polytuplet Loss manifold learning, showing the anchor, positive sample, and negative samples separated by margin $m$.

Experimental Setup

The models were evaluated on ReClor, a challenging dataset specifically designed for logical reasoning. Questions are pulled from high-stakes standardized tests like the LSAT and GMAT, totaling 5,138 samples. Unlike standard reading comprehension, ReClor requires identifying logical fallacies, implications, and strengthening/weakening arguments.

Key Results

The authors tested their approach across several Transformer backbones. In every case, adding Polytuplet Loss (CCE + P) outperformed the baseline (CCE only):

Performance comparison of base Transformer models showing significant accuracy improvements with Polytuplet Loss.
  • BERT (base): Jumped from 44.6% to 49.8% (an 11.7% improvement).
  • ALBERT (base): Improved from 53.4% to 56.4%.
  • ALBERT (xxlarge): Reached 67.5% accuracy, a 7.8% improvement over the baseline.
  • RoBERTa (large): Improved from 55.6% to 59.1%.

Key Insights and Discussion

  • Relative > Absolute: For multiple-choice tasks, the model doesn't need to know if an answer is 100% "True" in a vacuum; it only needs to know it is more true than the other three options.
  • Information Subsets: The information required to eliminate an incorrect answer is often a subset of the information required to prove the correct one. By focusing on these differences, the model can succeed even with "incomplete knowledge."
  • Hybrid Loss: The best results came from combining Polytuplet Loss with standard Categorical Cross-Entropy (CCE), suggesting that absolute classification still provides a useful training signal.

Limitations and Future Work

The authors noted that limited hardware prevented testing on the largest state-of-the-art models like DeBERTa. Future work could explore "logic-aware" pretraining and counterfactual data augmentation to further enhance the gains seen with Polytuplet Loss.

Summary updated

BibTeX
@inproceedings{lu2024polytuplet,
  author = {Jeffrey Lu and Ivan Rodriguez},
  title = {{Polytuplet Loss: A Reverse Approach to Training Reading Comprehension and Logical Reasoning Models}},
  booktitle = {Ficc},
  year = {2024},
  doi = {10.1007/978-3-031-53963-3\_23},
}
Figure