DiscoSense: Commonsense Reasoning with Discourse Connectives

Prajjwal Bhargava, Vincent Ng

Abstract

We present DiscoSense, a benchmark for commonsense reasoning via understanding a wide variety of discourse connectives. We generate compelling distractors in DiscoSense using Conditional Adversarial Filtering, an extension of Adversarial Filtering that employs conditional generation. We show that state-of-the-art pre-trained language models struggle to perform well on DiscoSense, which makes this dataset ideal for evaluating next-generation commonsense reasoning systems.

Introduction

In the evolution of Natural Language Processing, Large Language Models (LLMs) have demonstrated remarkable proficiency in standard commonsense reasoning benchmarks. Datasets like SWAG and HellaSwag, once considered difficult, now see models achieving scores exceeding 90%. However, these high scores often mask a lack of deep logical understanding. Researchers suspect that models rely on statistical pattern matching rather than true reasoning.

A fundamental aspect of human reasoning is the ability to navigate discourse relations—the logical "glue" that connects ideas. For example, the sentence "I was hungry" leads to vastly different conclusions depending on whether the following connective is "so" (causal) or "but" (contrastive). Existing benchmarks often fail to isolate these linguistic pivots. To address this, we introduce DiscoSense, a benchmark specifically designed to test a model's grasp of discourse connectives.

The DiscoSense Benchmark

DiscoSense consists of 13,056 context-connective-ending tuples. The task is structured as a multiple-choice challenge: given a context and a specific discourse connective, the model must select the most plausible ending from four options. The dataset covers 37 distinct discourse connectives, such as although, as a result, and nonetheless.

The examples in DiscoSense are derived from the DISCOVERY and DISCOFUSE datasets, which contain naturally occurring sentences from web crawls and Wikipedia. By focusing on these specific markers, DiscoSense forces models to move beyond simple keyword association and into the realm of logical coherence.

Methodology: Conditional Adversarial Filtering

A significant challenge in benchmark creation is preventing models from exploiting "shortcuts" or "easy" distractors. To ensure the difficulty of DiscoSense, we developed Conditional Adversarial Filtering (CAF).

While standard Adversarial Filtering (AF) uses a generator to create distractors and a discriminator to filter them, CAF introduces Conditional Generation. We fine-tuned a CTRL (Controllable Training) model to generate endings specifically conditioned on the discourse connective. This ensures that distractors are not only on-topic but also linguistically sophisticated and relevant to the specific logical relation being tested.

Following the automated generation, we implemented a rigorous two-step human verification process. Student workers reviewed the examples to ensure the ground truth was uniquely correct and that distractors were not accidentally plausible, maintaining high data quality.

Experimental Results

We evaluated several state-of-the-art models, including BERT, RoBERTa, ALBERT, XLNet, ELECTRA, and GPT-2. The results reveal a significant performance gap between humans and machines.

The Performance Gap

While humans achieve an accuracy of 95.4% on DiscoSense, the best-performing model, ELECTRA-Large, reached only 65.87%. This ~30% gap is substantially larger than the 2–5% gaps typically seen on other commonsense benchmarks like PIQA or SWAG. This suggests that DiscoSense successfully isolates a reasoning capability that current LLMs have yet to master.

The "Logic Gap" in Contrastive Reasoning

Error analysis showed that models struggle most with contrastive logic. Connectives like otherwise, however, and although proved much more difficult for models than "compliance" relations like similarly. This indicates that while models can predict what follows a logical sequence, they struggle to understand when a sequence is being subverted or contrasted.

Transfer Learning Value

Interestingly, DiscoSense serves as an effective training tool. Models fine-tuned on DiscoSense prior to being trained on HellaSwag showed improved performance on the latter. This suggests that mastering the nuances of discourse connectives provides a generalizable boost to a model's broader reasoning capabilities.

Conclusion and Future Work

DiscoSense highlights a critical weakness in current language models: the understanding of discourse-level logic. By using Conditional Adversarial Filtering, we have created a benchmark that resists simple statistical shortcuts and demands a deeper level of linguistic comprehension.

Future work will involve expanding the benchmark to include more ambiguous connectives and utilizing even stronger LLMs in the CAF pipeline to maintain the dataset's difficulty. DiscoSense stands as a challenging new frontier for the development of truly "reasoning" AI systems.

Summary updated

Abstract

We present DiscoSense, a benchmark for commonsense reasoning via understanding a wide variety of discourse connectives. We generate compelling distractors in DiscoSense using Conditional Adversarial Filtering, an extension of Adversarial Filtering that employs conditional generation. We show that state-of-the-art pre-trained language models struggle to perform well on DiscoSense, which makes this dataset ideal for evaluating next-generation commonsense reasoning systems.
BibTeX
@inproceedings{bhargava2022discosense,
  author = {Prajjwal Bhargava and Vincent Ng},
  title = {{DiscoSense: Commonsense Reasoning with Discourse Connectives}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=1-2l622kKv},
}
Figure