How Useful is Context, Actually? Comparing LLMs and Humans on Discourse Marker Prediction

Emily Sadlier-Brown, Millie Lou, Miikka Silfverberg, Carla KamACL 2024

TL;DR

This paper investigates how well Large Language Models (LLMs) can predict the discourse marker "actually" compared to humans, using natural conversational speech. They found that while LLMs are catching up, GPT-4 performs on par with or slightly better than humans on this task, highlighting the increasing importance of context for LLMs.

What Problem Does This Paper Address?

Natural language is rich with pragmatic information—the subtle cues speakers use to convey their stance, beliefs, and the relationship between utterances. Discourse markers, like "actually," are key to this pragmatic layer, signaling unexpectedness or reality. Understanding how well AI models can grasp and predict these markers is crucial for developing more human-like conversational AI. While LLMs are becoming incredibly powerful, their ability to handle nuanced pragmatic elements, especially in real-world conversational contexts, is still an active area of research. This paper specifically tackles the question of how much context LLMs need and how its placement affects their ability to predict a common discourse marker, "actually," and compares this to human performance.

Methods

The researchers employed a cloze test methodology, where a word ("actually") was masked in an utterance, and participants (both humans and LLMs) were asked to predict it. They explored four different context conditions:

  • T: Target utterance only.
  • T+N: Target utterance plus the following utterance.
  • P+T: Preceding utterance plus the target utterance.
  • P+T+N: Preceding, target, and following utterances.

They used two types of LLMs: encoder-only models (BERT, RoBERTa, ELECTRA) and generative models (GPT-3.5, GPT-4). For encoder-only models, prediction was straightforward. For generative models, they crafted prompts to elicit predictions. Human participants were recruited via Amazon Mechanical Turk and asked to provide the five most likely English words for the masked slot.

Datasets

The study utilized two main datasets:

  1. Providence Corpus: This dataset consists of 295 naturally-produced spoken utterances containing the discourse marker "actually," sourced from the Providence Corpus of transcribed everyday conversational speech around children. This corpus represents a less-explored text type for computational studies.
  2. Synthetic Examples: To mitigate potential biases and test model robustness, the researchers generated 295 synthetic examples using GPT-3.5. These examples were designed to mimic the structure of the "actually" examples but targeted a variety of other words.

Key Insights and Ideas

  • Context is King (for both humans and LLMs): The study consistently shows that providing surrounding context significantly improves prediction accuracy for "actually" for both humans and LLMs.
  • Preceding Context is Often More Valuable: For humans and the top-performing LLM (GPT-4), the preceding utterance (P+T) provided more benefit than the following utterance (T+N).
  • Generative Models Excel with Context: Generative LLMs like GPT-3.5 and GPT-4 showed larger gains in accuracy from added context compared to encoder-only models, suggesting their architecture is better suited for conditioning on contextual information.
  • Synthetic Data Skews Results: LLMs, particularly GPT-4, vastly outperform humans on synthetic data. This is hypothesized to be because synthetic data, often generated by LLMs themselves, closely mirrors their training distributions, making it more of an overfitting test than a true measure of generalizability.
  • LLMs are Approaching Human-Level Pragmatics: GPT-4's performance on "actually" prediction is on par with, and in some cases exceeds, human performance, indicating significant progress in LLMs' ability to handle pragmatic nuances.

Key Results

  • "Actually" Examples: GPT-4 achieved the highest recall@5 (34.2%) with full context (P+T+N), slightly outperforming humans (31.5%). Humans performed best with preceding context (P+T) at 27.1%.
  • Synthetic Examples: Performance was dramatically higher across the board. GPT-4 achieved an impressive 91.2% recall@5 with full context (P+T+N), while humans reached 51.2%. Notably, three models (GPT-4, BERT, and GPT-3.5) significantly outperformed humans on synthetic data.

Limitations and Future Work

  • Human Subject Limitations: The study acknowledges limitations in controlling for the English proficiency of mTurk participants, their level of attention, and motivation. While efforts were made to filter responses, these factors can influence results.
  • Experimental Design Limitations: Participants could complete multiple examples, potentially leading to a "guessing" strategy where they favored "actually." While mitigated by synthetic data and batching, a slight bias might remain.
  • LLM Training Data: It's impossible to definitively know if the GPT models were trained on the Providence corpus. If they were, their performance might be artificially inflated.
  • Future Work: The authors suggest exploring what contextual cues LLMs might be picking up that humans miss. They also raise the philosophical question of whether emulating human-like (imperfect) pragmatic understanding is a better goal than simply achieving higher accuracy. The stark contrast between natural and synthetic data performance also calls for caution when using synthetic data for comparisons.

Summary updated

BibTeX
@inproceedings{sadlierbrown2024how,
  author = {Emily Sadlier-Brown and Millie Lou and Miikka Silfverberg and Carla Kam},
  title = {{How Useful is Context, Actually? Comparing LLMs and Humans on Discourse Marker Prediction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure