I am a Strange Dataset: Metalinguistic Tests for Language Models

Tristan Thrush, Jared Moore, Miguel Monares, Christopher Potts, Douwe Kiela

TL;DR

Large Language Models (LLMs) are surprisingly poor at metalinguistic self-reference—sentences that make claims about their own structure, such as "This sentence has five words." A new benchmark, "I am a Strange Dataset," reveals that while humans find these tasks trivial, most state-of-the-art models perform near chance, with even GPT-4 struggling to break the 70% accuracy mark.

What Problem Does This Paper Address?

Self-reference is a cornerstone of logic, mathematics, and even theories of consciousness. In linguistics, "metalinguistic" language refers to using language to describe language itself. When you combine these, you get metalinguistic self-reference: statements that refer to their own properties (e.g., word count, parts of speech, or visual arrangement).

While LLMs have mastered many complex reasoning tasks, the authors noticed a glaring gap: models often fail to resolve the recursive logic required to understand a sentence that changes its own truth value as it is being written. For example, if a model is asked to complete "The penultimate word in this sentence is...", the correct answer ("is") depends entirely on the final word chosen. This paper investigates whether current LLMs possess the metalinguistic mastery required to handle these "strange loops."

Figure 1: An example highlighting the challenge pre- sented by our task. All models that we tested on our dataset are close to chance-level.

Methods

The researchers evaluated models using two primary subtasks:

  1. Generation: Models are given a sentence beginning and must choose between two endings. One ending makes the statement true, while the other makes it false.
  2. Verification: Models are presented with a complete statement and must judge its truth value (True/False).

To measure performance, the authors used surprisal-based metrics (comparing the model's internal probability/loss for the "True" vs. "False" options) and Chain-of-Thought (CoT) prompting, where the model explains its reasoning before providing an answer.

Crucially, they included a Non-Self-Referential Control. By replacing "this sentence" with "the following sentence," they could test if the failure was due to the "self" part of the reference or a more fundamental inability to reason about metalinguistic properties like word counts or letter types.

Datasets

The authors introduced "I am a Strange Dataset," a hand-crafted benchmark consisting of 208 examples. The dataset is categorized by 10 specific tags, including:

  • Numerical Operations: Counting words or characters.
  • Sub-Word: Reasoning about syllables or specific letters.
  • Sensory: Visual arrangement of text or emojis.
  • Multi-Channel: Mixing English with code (Python/C) or other languages.
Examples from the dataset showing how different endings flip the truth value of self-referential statements.

They also released "I am an Impossible Dataset," a set of 10 extremely complex examples (like those involving Caesar ciphers or Pi-based word lengths) that stumped even expert human annotators until they were given an explanation.

Key Insights and Ideas

  • The "Self" isn't the only bottleneck: Surprisingly, models performed poorly even on the non-self-referential controls. This suggests that LLMs struggle with basic metalinguistic tasks—like accurately counting the words in a provided string—which are prerequisites for self-reference.
  • Tokenization is a barrier: LLMs process text as "tokens" rather than individual characters. This makes "Sub-Word" tasks (counting letters or syllables) inherently difficult because the model doesn't "see" the text the way a human does.
  • Right for the wrong reasons: CoT reasoning revealed that models often stumble upon the correct True/False answer despite completely flawed logic, highlighting a lack of genuine understanding.
  • Scaling helps, but parity is far off: There is a clear correlation between model size and performance, but the slope is shallow. Parity with the human baseline (89-93%) likely requires models significantly larger or more specialized than current 70B parameter versions.

Key Results

The results show a massive gap between human and machine performance. While untrained human annotators scored between 89% and 93%, most LLMs hovered around 50% (chance level).

The main results table showing that most models perform near chance, with GPT-4 being the only significant outlier.
  • GPT-4 was the only model to consistently beat chance, scoring in the 60-66% range across various metrics.
  • Open-source models (Llama 2, Mistral, etc.) generally struggled, often failing to even output a valid "True" or "False" response in zero-shot settings.
  • Scaling Trend: As shown in the figure below, there is a statistically significant improvement as parameter counts increase, suggesting that metalinguistic capability may be an emergent property of scale.
Figure 7: Parameters to average “I am a Strange Dataset” score across all of the logprobs-based metrics. We only evaluate five 7B models, and two models for each of the other sizes, so computing confidence intervals for each point is not particularly informative. Under the null hypothesis that parameter size has no effect on score, we can compute the p-value for these results as a whole nonparametrically: there are 24 ways that these 4 points can be arranged and in only 1 of the ways do they all increase with the parameter count: p = 1/24 = 0.042.

Limitations and Future Work

The authors acknowledge that the tokenizer is a major limitation; if a model can't "see" characters, it can't reason about them. Furthermore, metalinguistic data is rare in standard web-scraped training sets, meaning models have few opportunities to learn these patterns.

Future work could involve training models on synthetic metalinguistic data or developing architectures that can "see" text at multiple levels of granularity (characters, words, and tokens) to bridge the gap to human-level performance.

Summary updated

Abstract

Statements involving metalinguistic self-reference ("This paper has six sections.") are prevalent in many domains. Can current large language models (LLMs) handle such language? In this paper, we present "I am a Strange Dataset", a new dataset for addressing this question. There are two subtasks: generation and verification. In generation, models continue statements like "The penultimate word in this sentence is" (where a correct continuation is "is"). In verification, models judge the truth of statements like "The penultimate word in this sentence is sentence." (false). We also provide minimally different metalinguistic non-self-reference examples to complement the main dataset by probing for whether models can handle metalinguistic language at all. The dataset is hand-crafted by experts and validated by non-expert annotators. We test a variety of open-source LLMs (7B to 70B parameters) as well as closed-source LLMs through APIs. All models perform close to chance across both subtasks and even on the non-self-referential metalinguistic control data, though we find some steady improvement with model scale. GPT 4 is the only model to consistently do significantly better than chance, and it is still only in the 60% range, while our untrained human annotators score well in the 89-93% range. The dataset and evaluation toolkit are available at https://github.com/TristanThrush/i-am-a-strange-dataset.
BibTeX
@inproceedings{thrush2024i,
  author = {Tristan Thrush and Jared Moore and Miguel Monares and Christopher Potts and Douwe Kiela},
  title = {{I am a Strange Dataset: Metalinguistic Tests for Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.482},
}
Figure