I am a Strange Dataset: Metalinguistic Tests for Language Models

Tristan Thrush, Jared Moore, Miguel Monares, Christopher Potts, Douwe Kiela

Introduction

Self-reference is a cornerstone of logic, mathematics, and human linguistics. "Metalinguistic self-reference" occurs when language describes its own properties—for instance, "This sentence has five words." While humans verify or complete such structures with ease, they create "strange loops" that challenge the linear processing of modern Large Language Models (LLMs).

Researchers have introduced "I am a Strange Dataset," a benchmark designed to test whether LLMs can navigate these puzzles. The study reveals a significant capability gap: while humans solve these problems with ~90% accuracy, most current LLMs perform near chance, with even state-of-the-art models like GPT-4 struggling to reach the 70% mark.

Methods

The evaluation suite includes open-source models (Llama 2, Mistral, Mixtral) and closed-source models (GPT-3.5, GPT-4, Claude 2). Performance was measured across two primary subtasks:

  1. Generation: Comparing the model's likelihood (logprobs) for a true ending versus a false ending given a sentence prefix.
  2. Verification: Judging the truth value (True/False) of completed statements using Zero-Shot (ZS), Few-Shot (FS), and Chain-of-Thought (CoT) prompting.

To isolate the difficulty of self-reference, the authors also utilized a Non-Self-Referential Control. This baseline tests whether failures stem from a lack of basic metalinguistic skills (like counting) or specifically from the recursive nature of "strange loops."

The "I am a Strange Dataset" Benchmark

The dataset consists of 200 core evaluation examples, hand-crafted by experts and categorized into 10 distinct linguistic tags. These categories range from numerical operations to sub-word logic and sensory descriptions.

Categories and examples from the "I am a Strange Dataset" benchmark.

The benchmark also includes a control set of 95 sentence pairs and an "Impossible Dataset" featuring 10 extremely complex examples (e.g., Caesar ciphers) that challenge even expert human annotators.

Key Results and Scaling

The results highlight a stark contrast between human and machine performance. While untrained human annotators achieved scores between 89% and 93%, most LLMs hovered around the 50-55% chance level.

Performance of various LLMs across generation and verification tasks.

GPT-4 emerged as the clear outlier, achieving 66% on the Validation CoT metric—the highest among all tested models. However, even this performance remains far below the human baseline. A clear scaling trend exists where larger models perform better, but the slow rate of improvement suggests that simply increasing parameter counts may not be enough to reach human-level proficiency in this domain.

Discussion and Limitations

A primary hurdle identified is tokenization. Because LLMs process text as tokens rather than raw characters, they struggle with sub-word tasks like counting letters. Furthermore, metalinguistic statements are rare in standard training corpora, leading to data scarcity.

The study also found that in Chain-of-Thought evaluations, models often arrived at the correct label through hallucinated logic, indicating that their "reasoning" in self-referential contexts is fragile. Future work may require specialized training data or architectural changes that allow models to more effectively "look back" at their own output buffers.

Summary updated

Abstract

Statements involving metalinguistic self-reference ("This paper has six sections.") are prevalent in many domains. Can large language models (LLMs) handle such language? In this paper, we present "I am a Strange Dataset", a new dataset for addressing this question. There are two subtasks: generation and verification. In generation, models continue statements like "The penultimate word in this sentence is" (where a correct continuation is "is"). In verification, models judge the truth of statements like "The penultimate word in this sentence is sentence." (false). We also provide minimally different metalinguistic non-self-reference examples to complement the main dataset by probing for whether models can handle metalinguistic language at all. The dataset is hand-crafted by experts and validated by non-expert annotators. We test a variety of open-source LLMs (7B to 70B parameters) as well as closed-source LLMs through APIs. All models perform close to chance across both subtasks and even on the non-self-referential metalinguistic control data, though we find some steady improvement with model scale. GPT 4 is the only model to consistently do significantly better than chance, and it is still only in the 60% range, while our untrained human annotators score well in the 89-93% range. The dataset and evaluation toolkit are available at https://github.com/TristanThrush/i-am-a-strange-dataset.
BibTeX
@inproceedings{thrush2024i,
  author = {Tristan Thrush and Jared Moore and Miguel Monares and Christopher Potts and Douwe Kiela},
  title = {{I am a Strange Dataset: Metalinguistic Tests for Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.482},
}
Figure