A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics

Qing Li, Siyuan Huang, Yining Hong, Yixin Zhu, Ying Nian Wu, Song-Chun Zhu

Introduction

While current state-of-the-art AI models excel at recognizing patterns like handwritten digits, they often fail miserably when asked to solve longer math problems or handle larger numbers than those seen during training. To test if AI can truly generalize like humans, researchers have introduced HINT (Handwritten arithmetic with INTegers), a minimalist yet challenging benchmark. The findings prove that simply "scaling up" models isn't enough to achieve human-like reasoning.

The Challenge of Systematic Compositionality

Humans possess a remarkable cognitive ability called systematic compositionality. If you know how to add two numbers and understand the concept of parentheses, you can solve an infinitely long expression, even if you’ve never seen one that long before.

Current AI models, however, tend to "cheat" by memorizing patterns. When faced with "out-of-distribution" data—such as a math problem that is longer (syntactic extrapolation) or involves larger results (semantic extrapolation) than the training set—even powerful models like Transformers often fall apart. Previous benchmarks mixed these challenges together, making it hard to pinpoint why a model failed. The authors of this paper aim to decouple these failures into three distinct levels:

  1. Perception: Turning raw signals (images) into symbols.
  2. Syntax: Understanding how symbols structurally combine (the rules of arithmetic).
  3. Semantics: Executing the actual logic to find a result.

The HINT Dataset

To isolate these variables, the authors introduced the HINT dataset, consisting of 1 million handwritten arithmetic expressions. The images are sourced from the CROHME dataset of handwritten math symbols. The training set is deliberately limited to 10 operators and results under 100.

Where HINT truly shines is in its evaluation splits, designed to test specific types of generalization:

  • I (Interpolation): New images, but the same length and value range as training.
  • SS: New combinations of syntax and semantics within the training range.
  • LS (Length/Syntax): Expressions longer than those in training.
  • SL (Semantics): Expressions resulting in numbers much larger than 100.
  • LL: Both longer expressions and larger numbers.
Examples of the HINT dataset splits, showing training and various test categories (I, SS, LS, SL, LL).

Additionally, a Few-shot Split tests whether models can learn entirely new operators (like "max" or "average") from just a few examples.

Methods

The researchers benchmarked several standard "sequence-to-sequence" (seq2seq) architectures. To handle the visual component, they used a ResNet-18 backbone to encode handwritten images into embeddings. These embeddings were then fed into various decoders:

  • RNNs (LSTM/GRU): Traditional recurrent models, with and without attention.
  • Transformers: The current industry standard, including variants with Relative Positional Encoding and Universal Transformers.
  • GPT-3: Tested in a zero-shot setting using Chain of Thought (CoT) prompting.

Key Insights and Results

Perception is the Easy Part Models reached nearly 90% accuracy on the "I" (Interpolation) split. This suggests that modern AI is highly capable at "seeing" (perception), but the breakdown happens during the "thinking" phase (syntax/semantics).

The "Scaling" Wall In one of the most striking findings, the authors show that simply increasing the size of the dataset or the model does almost nothing to help with extrapolation.

Test accuracy on the hardest split (LL) scales logarithmically with model and data size, indicating a fundamental limitation.

To reach 90% accuracy on the most difficult "LL" split via scaling alone, the authors estimate you would need a model with $10^{33}$ parameters—orders of magnitude larger than anything that exists today.

Relative Position Matters Transformers using relative positional encoding performed significantly better on longer sequences than those using absolute positions, though they still lagged far behind human performance. The Universal Transformer with relative positional encoding emerged as the top performer among trained models. However, while it achieved ~88% on perception-heavy tasks, its performance plummeted to near zero on samples where the final answer exceeded the training limit of 100.

GPT-3 Needs to "Think" Standard zero-shot GPT-3 failed at HINT, but using "Chain of Thought" prompting boosted its accuracy from 8.6% to 27.6%. Forcing the model to decompose the problem step-by-step helps it handle the semantic logic.

The Human-AI Gap in Few-Shot Learning In the few-shot learning category, humans achieved an average of 86.7% accuracy when learning new symbols. In contrast, the best AI model only managed 41.7%, struggling particularly to apply new operators to complex, multi-step expressions.

Conclusion and Future Work

The authors acknowledge that HINT is "minimalist." The expressions are context-free and spatially simple (linear strings), whereas real-world language and logic are often context-dependent and visually complex.

Nonetheless, the HINT dataset stands as a clear signal to the AI community: we cannot simply "scale" our way to true intelligence. Future research will likely need to move toward Neural-Symbolic approaches—hybrids that combine the "fuzzy" pattern recognition of neural networks with the "hard" logical rules of symbolic AI—to achieve genuine systematic generalization.

Summary updated

Abstract

Inspired by humans' exceptional ability to master arithmetic and generalize to new problems, we present a new dataset, HINT, to examine machines' capability of learning generalizable concepts at three levels: perception, syntax, and semantics. In HINT, machines are tasked with learning how concepts are perceived from raw signals such as images (i.e., perception), how multiple concepts are structurally combined to form a valid expression (i.e., syntax), and how concepts are realized to afford various reasoning tasks (i.e., semantics), all in a weakly supervised manner. Focusing on systematic generalization, we carefully design a five-fold test set to evaluate both the interpolation and the extrapolation of learned concepts w.r.t the three levels. Further, we design a few-shot learning split to determine whether or not models can rapidly learn new concepts and generalize them to more complex scenarios. To comprehend existing models' limitations, we undertake extensive experiments with various sequence-to-sequence models, including RNNs, Transformers, and GPT-3 (with the chain of thought prompting). The results indicate that current models struggle to extrapolate to long-range syntactic dependency and semantics. Models exhibit a considerable gap toward human-level generalization when evaluated with new concepts in a few-shot setting. Moreover, we discover that it is infeasible to solve HINT by merely scaling up the dataset and the model size; this strategy contributes little to the extrapolation of syntax and semantics. Finally, in zero-shot GPT-3 experiments, the chain of thought prompting exhibits impressive results and significantly boosts the test accuracy. We believe the HINT dataset and the experimental findings are of great interest to the learning community on systematic generalization.%
BibTeX
@inproceedings{li2023minimalist,
  author = {Qing Li and Siyuan Huang and Yining Hong and Yixin Zhu and Ying Nian Wu and Song-Chun Zhu},
  title = {{A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=kIPyTuEZuAK},
}
Figure