A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics

Qing Li, Siyuan Huang, Yining Hong, Yixin Zhu, Ying Nian Wu, Song-Chun Zhu

Humans have an uncanny ability to learn a new concept—say, a mathematical operator—and immediately apply it to incredibly complex problems. We don't just memorize patterns; we understand the underlying "grammar" of how concepts combine. In machine learning, this is known as systematic generalization, and despite the recent hype around Large Language Models (LLMs), it remains a formidable challenge.

A recent paper from ICLR 2023 introduces HINT (Handwritten arithmetic with INTegers), a minimalist yet rigorous benchmark designed to expose exactly where our current models fall short in perception, syntax, and semantics.

TL;DR

The HINT dataset tasks models with solving handwritten arithmetic expressions using only the final result as supervision. The study reveals that while models excel at visual perception, they struggle to extrapolate to longer expressions or larger numbers, and simply scaling up model size is not a viable path to human-level reasoning.

What Problem Does This Paper Address?

Most existing benchmarks for systematic generalization are "synthetic"—they use clean, digital text and often conflate different types of learning. For example, if a model fails to solve a long sentence, is it because it didn't understand the words (semantics) or because the sentence structure was too complex (syntax)?

The authors argue that true intelligence requires three interweaving levels of understanding:

  1. Perception: Recognizing raw signals (e.g., turning pixels of a handwritten '5' into the concept of five).
  2. Syntax: Understanding how concepts structurally combine (e.g., knowing that parentheses change the order of operations).
  3. Semantics: Realizing the functional meaning (e.g., knowing that '+' means addition).

HINT forces models to learn all three simultaneously in a weakly supervised manner—meaning the model only sees the handwritten image and the final answer, never the intermediate steps or the symbolic "translation."

Methods

The researchers formulated HINT as a sequence-to-sequence (seq2seq) problem.

  • Perception Layer: A ResNet-18 backbone processes sliding window patches of handwritten expressions.
  • Reasoning Layer: They benchmarked several architectures, including:
  • RNNs: LSTMs and GRUs (with and without attention).
  • Transformers: Vanilla versions, versions with relative positional encoding, and Universal Transformers (which share weights across layers).
  • LLMs: GPT-3 (text-davinci-002) was tested on symbolic versions of the task using zero-shot and "Chain-of-Thought" (CoT) prompting.
Figure 1: The seq2seq framework applied to an example in HINT. ¡SOS¿: start-of-sentence tokens. ¡EOS¿: end-of-sentence tokens. A sliding window segments the handwritten expression into a sequence of images, which are then separately encoded by ResNet-18. The expected output is a sequence of digits in reverse order.

Datasets

The HINT dataset contains ~1 million samples derived from the CROHME handwritten math symbol collection. To test generalization, the authors created five distinct test splits:

  • I (Interpolation): Same expression lengths and value ranges as training, but new handwritten images.
  • SS (Syntax & Semantics Interpolation): New expressions, but within the training length and value range.
  • LS (Long Syntax): Longer expressions than seen in training.
  • SL (Super Large Semantics): Expressions resulting in numbers larger than 100 (the training cap).
  • LL (Long & Large): Both longer expressions and larger results.

They also included a few-shot learning split where models must learn entirely new concepts (like a "max" operator or new digits) from just a few examples.

Key Insights and Ideas

  • Perception is "Solved": Even without direct labels for individual symbols, models naturally learned to cluster handwritten digits and operators. A t-SNE visualization shows clear, meaningful clusters based on visual appearance and syntactic roles.
Figure 2: The t-SNE visualization of the embeddings (the outputs of ResNet-18) of handwritten images using the Transformer rel. univ. model. The image embeddings form clear clusters for each concept based on visual appearance. In addition, these clusters reflect the concepts’ syntactic roles: The majority of digits are towards the bottom, operators are around the center, and parentheses are near the top.
  • The "Scaling Law" Wall: One of the most striking findings is that scaling doesn't help extrapolation. The authors calculated that to reach 90% accuracy on the hardest "LL" split by just adding more data and parameters, you would need a model with $10^{33}$ parameters trained on $10^{15}$ examples—an astronomical impossibility.
Figure 3: Scaling trends w.r.t. model size and dataset size when training Transformer rel. uni. on the test subset LL with symbol inputs.
  • Syntax vs. Semantics: Transformers with relative positional encoding are much better at handling long sequences (syntax) than vanilla versions, but they still hit a wall when the result of the math exceeds what they saw in training (semantics).

Key Results

  • Performance Gap: The best-performing model (Universal Transformer with relative positional encoding) achieved only 54.3% average accuracy on HINT. While it was nearly perfect on interpolation (Split I), it plummeted to ~19% on the hardest extrapolation tasks (Split LL).
  • GPT-3 & CoT: On symbolic inputs, standard zero-shot GPT-3 managed only 8.6% accuracy. However, using Chain-of-Thought prompting ("Let's think step by step") boosted this to 27.6%. While impressive, it still lags far behind human-level reasoning on long-range dependencies.
  • Human vs. Machine: In few-shot tests, humans achieved 86.7% accuracy, easily generalizing new operators to complex formulas. The best machine models struggled to break 41.7%.

Limitations and Future Work

The authors acknowledge that while HINT is syntactically complex, the visual environment is still "clean" (white backgrounds, clear symbols). Future work could involve more complex spatial relations or context-dependent meanings (where a symbol's meaning changes based on its surroundings).

Ultimately, the paper concludes that current seq2seq models are "rule-memorizers" rather than "rule-extractors." To bridge the gap to human-level generalization, we likely need new architectures—perhaps neural-symbolic hybrids—that can explicitly represent and manipulate abstract rules.

Summary updated

Abstract

Inspired by humans' exceptional ability to master arithmetic and generalize to new problems, we present a new dataset, Handwritten arithmetic with INTegers (HINT), to examine machines' capability of learning generalizable concepts at three levels: perception, syntax, and semantics. In HINT, machines are tasked with learning how concepts are perceived from raw signals such as images (i.e., perception), how multiple concepts are structurally combined to form a valid expression (i.e., syntax), and how concepts are realized to afford various reasoning tasks (i.e., semantics), all in a weakly supervised manner. Focusing on systematic generalization, we carefully design a five-fold test set to evaluate both the interpolation and the extrapolation of learned concepts w.r.t. the three levels. Further, we design a few-shot learning split to determine whether or not models can rapidly learn new concepts and generalize them to more complex scenarios. To comprehend existing models' limitations, we undertake extensive experiments with various sequence-to-sequence models, including RNNs, Transformers, and GPT-3 (with the chain of thought prompting). The results indicate that current models struggle to extrapolate to long-range syntactic dependency and semantics. Models exhibit a considerable gap toward human-level generalization when evaluated with new concepts in a few-shot setting. Moreover, we discover that it is infeasible to solve HINT by merely scaling up the dataset and the model size; this strategy contributes little to the extrapolation of syntax and semantics. Finally, in zero-shot GPT-3 experiments, the chain of thought prompting exhibits impressive results and significantly boosts the test accuracy. We believe the HINT dataset and the experimental findings are of great interest to the learning community on systematic generalization.
BibTeX
@inproceedings{li2023minimalist,
  author = {Qing Li and Siyuan Huang and Yining Hong and Yixin Zhu and Ying Nian Wu and Song-Chun Zhu},
  title = {{A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=kIPyTuEZuAK},
}
Figure