A HINT from Arithmetic: On Systematic Generalization of Perception, Syntax, and Semantics

Qing Li, Siyuan Huang, Yining Hong, Yixin Zhu, Ying Nian Wu, Song-Chun Zhu

Abstract

Inspired by humans' remarkable ability to master arithmetic and generalize to unseen problems, we present a new dataset, HINT, to study machines' capability of learning generalizable concepts at three levels: perception, syntax, and semantics. Learning agents are tasked to reckon how concepts are perceived from raw signals such as images (i.e., perception), how multiple concepts are structurally combined to form a valid expression (i.e., syntax), and how concepts are realized to afford various reasoning tasks (i.e., semantics), all in a weakly supervised manner. With a focus on systematic generalization, we carefully design a five-fold test set to evaluate both the interpolation and the extrapolation of learned concepts w.r.t. the three levels. We further design a few-shot learning split to test whether models could quickly learn new concepts and generalize them to more complex scenarios. To understand existing models' limitations, we conduct extensive experiments with various sequence-to-sequence models, including RNNs, Transformers, and GPT-3 (with the chain of thought prompting). The results suggest that current models still struggle in extrapolation to long-range syntactic dependency and semantics. Models show a significant gap toward human-level generalization when tested with new concepts in a few-shot setting. Moreover, we find that it is infeasible to solve HINT by simply scaling up the dataset and the model size; this strategy barely helps the extrapolation over syntax and semantics. Finally, in zero-shot GPT-3 experiments, the chain of thought prompting shows impressive results and significantly boosts the test accuracy. We believe the proposed dataset together with the experimental findings is of great interest to the community on systematic generalization.

Introduction

Humans have a remarkable ability to learn a concept once and apply it in infinitely complex ways—a trait known as systematic compositionality. If a student understands the operation "1 + 1" and the variable concept "x," they should intuitively grasp the structure of "1 + x." Current AI benchmarks often struggle to isolate why models fail: is it a failure of perception (seeing the input), syntax (understanding the grammar), or semantics (knowing what symbols do)? The HINT (Handwritten arithmetic with INTegers) dataset provides a minimalist yet rigorous testbed to evaluate whether models truly generalize or merely memorize patterns within a specific distribution.

Methodology

The researchers treat HINT as a sequence-to-sequence (seq2seq) problem, requiring models to map raw pixel data to final numerical results under weak supervision. The architecture consists of two primary components:

  1. Perception: A ResNet-18 backbone acts as the visual encoder, converting a sliding window of handwritten image pixels into a sequence of embeddings.
  2. Reasoning: These embeddings are processed by various decoders, including RNNs (LSTM, GRU), Transformers (vanilla and Universal), and GPT-3.

Crucially, models are only provided with the final result of an expression (e.g., "42") during training. They must autonomously infer the latent structures—mapping specific image segments to digits and learning the underlying mathematical tree—without direct supervision on intermediate steps.

The seq2seq framework used to process handwritten expressions into numerical results.

The HINT Dataset

The HINT dataset consists of 1 million arithmetic expressions using handwritten symbols from the CROHME dataset. To rigorously test generalization, the evaluation is split into five categories:

  • I (Interpolation): Same complexity as training, using new images.
  • SS (Syntax & Semantics Interpolation): New combinations of known lengths and values.
  • LS (Length Extrapolation): Expressions longer than those seen in training.
  • SL (Semantic Extrapolation): Expressions resulting in numbers larger than the training cap (100).
  • LL (Length & Semantic Extrapolation): Both longer expressions and larger values.

Key Insights and Results

Perception vs. Logic

Perception appears to be the most manageable hurdle. Models achieved approximately 88% accuracy on the "I" subset. Visualizations indicate that even without direct labels, the ResNet-18 encoder successfully learns to cluster images by their symbolic identity (digits vs. operators). However, accuracy drops precipitously on the LS, SL, and LL subsets, suggesting models are "blind" to logic that falls outside their training range.

The Scaling Wall

A significant finding of the study is that systematic generalization cannot be solved by brute force. The authors calculated that reaching 90% accuracy on the LL subset would require a model with roughly $10^{33}$ parameters trained on $10^{15}$ examples based on current scaling trends. This suggests that simply increasing model size or data volume provides diminishing returns for true extrapolation.

Scaling laws show that increasing model size or data provides diminishing returns for extrapolation tasks.

Model Performance and Prompting

The Universal Transformer with relative positional encoding emerged as the strongest standard model, averaging 53.1% accuracy across image-based tests. While standard models struggled with semantic extrapolation (~11% on SL), GPT-3 showed promise when paired with Chain-of-Thought (CoT) prompting. Asking the model to "think step-by-step" boosted zero-shot accuracy from 8.6% to 27.6%, though it still faced challenges with long-range dependencies.

Conclusion

HINT reveals a substantial gap between current neural architectures and human-level systematic generalization. While models excel at interpolation, they hit a "wall" when faced with longer or more complex semantic structures. Future work may need to move beyond pure end-to-end deep learning toward neural-symbolic approaches that can better bridge the gap between raw perception and rigid logical reasoning.

Summary updated

Abstract

Inspired by humans' remarkable ability to master arithmetic and generalize to unseen problems, we present a new dataset, HINT, to study machines' capability of learning generalizable concepts at three levels: perception, syntax, and semantics. Learning agents are tasked to reckon how concepts are perceived from raw signals such as images (i.e., perception), how multiple concepts are structurally combined to form a valid expression (i.e., syntax), and how concepts are realized to afford various reasoning tasks (i.e., semantics), all in a weakly supervised manner. With a focus on systematic generalization, we carefully design a five-fold test set to evaluate both the interpolation and the extrapolation of learned concepts w.r.t. the three levels. We further design a few-shot learning split to test whether models could quickly learn new concepts and generalize them to more complex scenarios. To understand existing models' limitations, we conduct extensive experiments with various sequence-to-sequence models, including RNNs, Transformers, and GPT-3 (with the chain of thought prompting). The results suggest that current models still struggle in extrapolation to long-range syntactic dependency and semantics. Models show a significant gap toward human-level generalization when tested with new concepts in a few-shot setting. Moreover, we find that it is infeasible to solve HINT by simply scaling up the dataset and the model size; this strategy barely helps the extrapolation over syntax and semantics. Finally, in zero-shot GPT-3 experiments, the chain of thought prompting shows impressive results and significantly boosts the test accuracy. We believe the proposed dataset together with the experimental findings is of great interest to the community on systematic generalization.
BibTeX
@inproceedings{li2023hint,
  author = {Qing Li and Siyuan Huang and Yining Hong and Yixin Zhu and Ying Nian Wu and Song-Chun Zhu},
  title = {{A HINT from Arithmetic: On Systematic Generalization of Perception, Syntax, and Semantics}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=kIPyTuEZuAK},
}
Figure