Modern deep learning models are notoriously data-hungry. To master a language task, a standard sequential model typically requires millions or billions of tokens. The reason for this inefficiency is fundamental: sequential models treat language as a flat stream of tokens. To perform tasks accurately, they must infer the underlying hierarchical rules of grammar, syntax, and morphology entirely from scratch.
A powerful alternative approach bypasses this trial-and-error discovery by explicitly injecting structural priors into the learning process. By equipping models with explicit linguistic scaffolds—such as dependency trees, morphological paradigms, or syntactic skeletons—we provide a lower-entropy, higher-density signal. Instead of wasting parameter capacity and data volume learning basic grammar, the model can focus immediately on the downstream task.
Syntactic Skeletons as Data Force Multipliers
When a model is forced to learn language purely from word co-occurrences, it requires massive datasets to recognize long-distance dependencies or complex grammatical roles. Introducing an explicit syntactic structure acts as a force multiplier for data efficiency.
For example, Adhiguna Kuncoro et al. explored this in Syntactic Structure Distillation Pretraining For Bidirectional Encoders. By distilling structural syntactic knowledge from a Recurrent Neural Network Grammar (RNNG) into a bidirectional transformer (BERT), the authors injected an explicit hierarchical bias. When evaluated on semantic role labeling, this structure-distilled model achieved significant error reductions while using only 5% of the task-specific training data required by standard baselines.
Flat Sequence: [ The ] [ cat ] [ that ] [ chased ] [ the ] [ mouse ] [ ran ] [ away ]
|_______|
(Long-distance dependency)
Syntactic Prior: (S (NP (NP The cat) (SBAR that chased the mouse)) (VP ran away))
|__________________________________________|
|
(Immediate structural link)
Similarly, Yuyu Zhang et al. developed the Neural Variational Language Model in Language Modeling with Shared Grammar. By treating grammar as a shared latent resource across corpora, their model learns structural rules from a small annotated dataset and transfers that intuition to unannotated text. As a result, the model achieves competitive perplexity using only 20% of the data required by purely sequential models.
Even in specialized generation and editing tasks, syntax drastically reduces data requirements. Jing Jiang et al. demonstrated in Can Syntax Help? Improving an LSTM-based Sentence Compression Model for New Domains that combining a recurrent network with explicit part-of-speech and dependency features allowed the system to generalize to new domains while using 250 times less data than purely neural sequence-to-sequence baselines.
Invariant Anchors for Cross-Domain and Cross-Lingual Transfer
Data scarcity is particularly acute when transferring models across domains (e.g., from movie reviews to financial text) or across languages. Vocabulary and surface-level token distributions change dramatically between domains, rendering flat sequence representations highly fragile. Syntactic structures, however, remain remarkably invariant.
In cross-domain sentiment classification, Kai Zhang et al. introduced Graph Adaptive Semantic Transfer (GAST) in Graph Adaptive Semantic Transfer for Cross-domain Sentiment Classification. By mapping sentences to syntactic dependency graphs, the framework focuses on domain-invariant relationships (like modifier-noun pairs) rather than raw text sequences. This structural focus allows the model to match or outperform sequence-only models while using a small fraction of the target domain's training data.
This principle scales to cross-lingual comprehension. In Structural Contrastive Pretraining for Cross-Lingual Comprehension, Nuo Chen et al. aligned the core "structural backbones" (Subject-Verb-Object structures and key entities) of parallel sentences across different languages. By prioritizing these high-signal semantic relationships during pretraining, their model outperformed systems trained on over 30 times more data that relied on standard token-level alignment.
Language A: [ Subject A ] ---> [ Verb A ] ---> [ Object A ]
| | |
(Structural Contrastive Alignment: Invariant Backbone)
| | |
Language B: [ Subject B ] ---> [ Verb B ] ---> [ Object B ]
The utility of structural anchors extends beyond natural languages to programming languages. In Syntax and Domain Aware Model for Unsupervised Program Translation, Fang Liu et al. leveraged Abstract Syntax Trees (ASTs) to translate code between languages without parallel corpora. Because the underlying logic of a loop or conditional statement is preserved in the AST structure even when the syntax changes, the model achieves high translation accuracy without requiring massive monolingual code repositories.
Symbolic Constraints in Extreme Low-Resource Regimes
In extreme low-resource scenarios, data is measured in dozens of examples rather than thousands. In this regime, purely statistical models fail because they cannot reliably separate signal from noise. Structural and symbolic constraints solve this by establishing a strict mathematical floor for model behavior.
Consider task-oriented dialogue systems. Arash Eshghi et al. demonstrated in Bootstrapping incremental dialogue systems from minimal data: the generalisation power of dialogue grammars that combining an incremental semantic grammar (DS-TTR) with reinforcement learning allows a system to achieve high task success rates after training on as few as five example dialogues. The grammar acts as a rigid constraint, handling human speech patterns like self-corrections and hesitations natively, so the statistical learning component only needs to optimize high-level dialogue strategies.
Raw Input: "I want a flight to Boston— no, sorry, Chicago."
Without Prior: Model must learn via thousands of examples that "no, sorry" negates "Boston".
With Grammar: The symbolic grammar automatically updates the semantic tree:
Destination: Boston -> (Self-Correction) -> Chicago.
Model only learns the final dialogue state.
We see a similar effect in morphology. In Unsupervised Morphology Learning with Statistical Paradigms, Hongzhi Xu et al. targeted morphological segmentation by leveraging the linguistic concept of paradigms (consistent sets of suffixes like -ed, -ing, and -s). By using these structural paradigms to prune spurious segmentations, their model outperformed complex log-linear and high-dimensional embedding models using nothing more than a raw word list, completely bypassing the need for massive training corpora.
Finally, structural constraints can simplify complex generation tasks into simpler sub-problems. Lingpeng Kong showed in Transforming Dependencies into Phrase Structures that treating a fast dependency parse as a hard constraint for a structured prediction model allows the system to generate rich phrase-structure trees with minimal training data. Because the dependency skeleton already captures the primary syntactic information of the sentence, the model trained on a small fraction of annotated data can match the performance of traditional, data-intensive parsers.
Key Foundational Work
Corpus-Based Acquisition of Relative Pronoun Disambiguation Heuristics (1992) — contributed 1 concept elements to this direction.
Bayesian Inference of Regular Grammar and Markov Source Models (1989) — contributed 1 concept elements to this direction.