Abstract
While large language models (LMs) have achieved human-level performance on many commonsense benchmarks, evidence suggests they often exploit statistical artifacts rather than engaging in genuine reasoning. We present DISCOSENSE, a new benchmark designed to test an AI's understanding of discourse connectives—words like however, because, and for example that define logical relationships between ideas. By employing a novel Conditional Adversarial Filtering (CAF) technique, we developed a dataset where the performance gap between humans (95.4%) and the best-performing models (65.9%) remains substantial. Our findings indicate that discourse-level logic is a critical blind spot for current architectures, yet training on such logic yields significant transfer learning benefits for other reasoning tasks.
Introduction
The rapid advancement of large language models has led to near-perfect scores on established benchmarks like SWAG and HellaSwag. However, these models often fail when faced with subtle logical constraints. A primary example of such constraints is the use of discourse connectives. These words act as logical "glue," signaling how a subsequent statement should relate to the preceding context. For instance, the connective "but" sets an expectation for contrast, while "consequently" implies a causal result. Current models frequently overlook these signals, relying instead on keyword associations. DISCOSENSE aims to expose and measure this "reasoning gap."
Methodology: Conditional Adversarial Filtering
To ensure the benchmark is sufficiently challenging, we developed Conditional Adversarial Filtering (CAF). This method improves upon traditional adversarial filtering by making the generation of distractors (incorrect options) context-aware.
The CAF process involves three key stages:
- Controlled Generation: We fine-tuned a CTRL model using discourse connectives as control codes. This ensures that generated distractors are logically relevant to the specific connective being tested, making them harder to distinguish from the correct answer.
- Iterative Hardening: The process is repeated for approximately 150 iterations. In each round, a discriminator model identifies the "easiest" distractors, which are then replaced by more deceptive ones from the generator.
- Human Verification: To maintain high data quality, we implemented a two-step human consensus process. Only examples where multiple annotators agreed on a single correct answer were retained.
The DISCOSENSE Dataset
The resulting dataset contains 13,056 examples (9,299 for training and 3,757 for testing) derived from the DISCOVERY and DISCOFUSE corpora. It covers 37 distinct discourse connectives across diverse domains like Wikipedia and Common Crawl. By focusing on these connectives, the benchmark forces models to move beyond simple word-matching and consider the structural logic of the text.
Results and Analysis
The performance gap on DISCOSENSE is stark. While humans achieve an accuracy of 95.40%, the state-of-the-art ELECTRA-Large model reaches only 65.87%. Baseline models like BERT and RoBERTa perform even worse, often barely exceeding the 25% random-guess threshold.
Qualitative analysis reveals that models often fall for "abstract associations." For example, in a context about "obesity," a model might incorrectly choose an option containing "diabetes" regardless of whether the connective (e.g., "instead") makes the sentence logically sound.
Transfer Learning
A significant finding of this study is the transferability of discourse-level reasoning. Fine-tuning ELECTRA-Large on DISCOSENSE before evaluating it on HellaSwag improved its score from 86.9% to 91.5%. This suggests that learning to respect discourse connectives instills a form of "logical hygiene" that benefits broader commonsense reasoning capabilities.
Conclusion
DISCOSENSE reveals that discourse-level logic remains a major challenge for modern language models. The benchmark successfully identifies a reasoning gap that traditional datasets miss. Future work will look to expand this approach to multi-hop reasoning across entire paragraphs and explore more ambiguous connectives that require deeper contextual understanding.
Summary updated