Abstract
This paper introduces Polytuplet Loss, a novel loss function that shifts the focus of Reading Comprehension and Logical Reasoning (RCLR) models from determining absolute answer accuracy to evaluating relative correctness. By training models to project contexts and answers into a shared manifold where the correct choice is pulled toward the context and incorrect ones are pushed away, the authors achieved performance gains of up to 11.7% over standard baselines.
Introduction
Most current machine learning models approach multiple-choice logical reasoning tasks with a "prompt-first" mentality: they try to calculate the absolute probability that a specific answer is true based on the context. However, this requires a very high information threshold—the model must fully understand the logic to "prove" an answer.
The authors argue that humans often use an "answer-first" strategy. If you’ve ever taken a standardized test like the LSAT, you know that you can often identify the correct answer by comparing the choices and eliminating those that are logically inconsistent, even if you don't fully grasp every nuance of the text. This paper aims to bridge this gap by teaching models to prioritize the relative differences between answer choices rather than treating each choice as an independent true/false question.
Methodology
The core contribution is the Polytuplet Loss, an extension of the well-known Triplet Loss used in computer vision.
- Manifold Learning: The model projects the "Anchor" (the context and question) and the "Choices" (the potential answers) into a lower-dimensional space (a manifold).
- The Loss Function: While Triplet Loss compares one positive and one negative sample, Polytuplet Loss compares the anchor against the correct answer and all incorrect answers simultaneously. It forces the distance between the anchor and the correct answer to be smaller than the distance to any incorrect answer by at least a margin $m$.
- Index-blind Architecture: To prevent the model from simply memorizing the position of the correct answer, the authors used an architecture that flattens the batch and answer dimensions. This ensures the model evaluates the relationship between the context and the answer choice without knowing which "slot" (A, B, C, or D) the answer occupies.
- Hard and Semi-Hard Negative Mining: The authors implemented a strategy to focus training on "difficult" examples—cases where an incorrect answer is already very close to the anchor or where the correct answer is further away than a distractor.
Experimental Setup
The models were evaluated on ReClor, a challenging dataset specifically designed for logical reasoning. Questions are pulled from high-stakes standardized tests like the LSAT and GMAT, totaling 5,138 samples. Unlike standard reading comprehension, ReClor requires identifying logical fallacies, implications, and strengthening/weakening arguments.
Key Results
The authors tested their approach across several Transformer backbones. In every case, adding Polytuplet Loss (CCE + P) outperformed the baseline (CCE only):
- BERT (base): Jumped from 44.6% to 49.8% (an 11.7% improvement).
- ALBERT (base): Improved from 53.4% to 56.4%.
- ALBERT (xxlarge): Reached 67.5% accuracy, a 7.8% improvement over the baseline.
- RoBERTa (large): Improved from 55.6% to 59.1%.
Key Insights and Discussion
- Relative > Absolute: For multiple-choice tasks, the model doesn't need to know if an answer is 100% "True" in a vacuum; it only needs to know it is more true than the other three options.
- Information Subsets: The information required to eliminate an incorrect answer is often a subset of the information required to prove the correct one. By focusing on these differences, the model can succeed even with "incomplete knowledge."
- Hybrid Loss: The best results came from combining Polytuplet Loss with standard Categorical Cross-Entropy (CCE), suggesting that absolute classification still provides a useful training signal.
Limitations and Future Work
The authors noted that limited hardware prevented testing on the largest state-of-the-art models like DeBERTa. Future work could explore "logic-aware" pretraining and counterfactual data augmentation to further enhance the gains seen with Polytuplet Loss.
Summary updated