Inductive Linguistic Reasoning with Large Language Models

Raghav Ramji, Keshav RamjiSubmission to ICLR 2025 · Rejected

Introduction

While Large Language Models (LLMs) excel at processing high-resource languages like English, they often struggle with "low-resource" or extinct languages where training data is scarce. Evaluating these models on their linguistic reasoning capabilities is crucial for understanding the gaps in their skills during large-scale adoption. This paper focuses on linguistic reasoning—the ability to deduce the grammatical rules of an unknown language from just a few examples, similar to the challenges faced by participants in the International Linguistics Olympiad (IOL).

Standard approaches, such as few-shot prompting or Chain-of-Thought (CoT), frequently fail in these scenarios because the models lack sufficient context to "anchor" their reasoning. To address this, researchers have developed a novel two-stage analogical prompting method. By tasking models to first identify a language's family and self-generate similar puzzles in related languages, this approach significantly boosts the ability of LLMs to solve complex linguistic puzzles, achieving state-of-the-art results.

The 2-Stage Analogical Prompting Framework

The authors introduce a framework designed to elicit a model's latent parametric knowledge of language families, using it to create auxiliary demonstrations that guide the reasoning process.

2-Stage Analogical Prompting Framework

The framework operates in two distinct stages:

  1. Analogical Generation (Stage 1): Given a test puzzle in a target language (e.g., Montenegrin), a generator model identifies the language family (e.g., Slavic) and generates similar "analogical" puzzles in related, higher-resource languages it knows well (e.g., Croatian).
  2. Deduction (Stage 2): A deducer model—which can be the same as the generator or a different one—is provided with both the original target language examples and the newly generated analogical examples. It uses this expanded context to perform inductive reasoning and translate the test phrase.

This two-stage process mimics human "System 2" thinking: slow, deliberate reasoning where one recalls similar known problems to solve a new, unknown one. The act of generating related puzzles acts as a "warm-up," forcing the model to surface its latent knowledge of syntax and morphology before tackling the specific target problem.

Datasets and Evaluation

The researchers evaluated their method on two primary benchmarks designed to test out-of-distribution linguistic reasoning:

  • modeLing: A dataset of 272 "Rosetta Stone" style translation puzzles across 19 languages, specifically designed to be unseen by models during training.
  • LINGOLY: A larger benchmark featuring 1,133 problems in 94 languages, covering diverse task types like grammatical patterns, match-ups, and computational linguistics errors.

Key Insights and Results

The analogical prompting approach yielded substantial improvements over standard reasoning techniques. Zero-shot performance on these tasks is near 0%, confirming that the models must truly reason in-context rather than relying on memorized data.

Performance Gains with Analogical Prompting
  • Significant Performance Gains: Analogical prompting improved GPT-4o’s performance by +8.1% (from 59.19% to 67.28%) and Llama-3.1-405B’s by +5.9% (from 65.81% to 71.69%) over standard CoT approaches.
  • State-of-the-Art on LINGOLY: The method achieved SOTA results across nearly all difficulty levels (Breakthrough to Round 2), even surpassing previous high-water marks set by models like Claude-3 Opus.
  • Cross-Model Synergy: The study explored combinations of different models acting as generators and deducers. The strongest overall performance (71.69% Exact Match) was achieved when GPT-4o acted as the "generator" and Llama-3.1-405B acted as the "deducer."
  • The "Weak-to-Strong" Boost: Interestingly, frontier models like GPT-4o sometimes performed better when using analogical examples generated by a specialized (but technically "weaker") multilingual model like Aya-35B than they did using their own self-generated examples. This suggests that multilingual specialization provides unique reasoning anchors that generalist models can leverage.
  • Handling Language Isolates: For languages with no known relatives (isolates), models were surprisingly creative, often generating "synthetic" or geographically proximate languages to create a comparative framework for reasoning.

Limitations and Future Work

While highly effective, the approach has limitations. The primary evaluation metric, "Exact Match," is binary and unforgiving; alternative metrics like ChrF2 and BLEU can be noisy for short translation strings. Additionally, the current method only handles text-based puzzles, whereas many real-world linguistics puzzles involve images, diagrams, or maps requiring multimodal capabilities. Finally, there is currently no automated way to verify if the generated analogical puzzles are linguistically accurate, relying instead on the model's internal consistency. Future work may focus on human-in-the-loop verification and expanding the method to handle more typologically unique languages.

Summary updated

Abstract

Evaluating large language models (LLMs) on their linguistic reasoning capabilities is an important task to understand the gaps in their skills that may surface during large-scale adoption. In this work, we investigate the abilities of such models to perform abstract multilingual reasoning through the lens of linguistic puzzles on extremely low-resource languages. As these translation tasks involve inductive and deductive reasoning from reference instances, we examine whether diverse auxiliary demonstrations can be automatically induced from seed exemplars, through analogical prompting. We employ a two-stage procedure, first generating analogical exemplars with a language model, and then applying them in-context along with provided target language exemplars. We explore various combinations of language models as analogical generators and reasoning agents, testing different model sizes and specialized multilingual LLMs. Our results on the modeLing dataset show that analogical prompting is effective in eliciting models' knowledge of language grammar similarities, boosting the performance of GPT-4o by as much as 8.1\% and Llama-3.1-405B by 5.9\% over chain-of-thought approaches. These gains are realized with self-generated analogical demonstrations as well as those generated by weaker multilingual models. We also report several findings about interesting phenomena which drive linguistic reasoning performance, suggesting that such puzzles are a valuable benchmark for new reasoning methods.
BibTeX
@misc{ramji2024inductive,
  author = {Raghav Ramji and Keshav Ramji},
  title = {{Inductive Linguistic Reasoning with Large Language Models}},
  year = {2024},
  eprint = {2412.17819},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.17819},
}
Figure