Entity Abstraction for Structural Generalization in Language Models

19 concepts11 papers1 proposal

When a language model processes a sentence like "Alice traveled to Paris," it faces two fundamentally different types of information: the underlying structural syntax of a travel narrative, and the highly specific, open-ended identities of the actor and the destination. Standard neural networks treat all tokens equally, forcing the model to allocate parameter capacity to memorizing rare proper nouns rather than mastering generalizable linguistic patterns.

Entity abstraction solves this by replacing specific named entities or domain terms with generic, type-based placeholders (e.g., [PERSON] traveled to [LOCATION]) during training. This simple normalization shifts the network's focus from what specific content is being discussed to how the language is structurally organized. By decoupling syntax from surface-level vocabulary, entity abstraction acts as a powerful regularizer that improves data efficiency, enables cross-domain transfer, and prevents overfitting to rare words.


Taming the Long Tail of Vocabulary

The primary bottleneck in many natural language processing (NLP) tasks is the out-of-vocabulary (OOV) problem and data sparsity. Named entities form an infinite, long-tailed distribution. By collapsing thousands of unique names into a handful of semantic categories, we drastically reduce the effective vocabulary size that a model must manage.

As Di Lu et al. demonstrate in Entity-aware Image Caption Generation, category-based placeholders mitigate the OOV problem for rare names, allowing language models to remain robust against infrequent data points while focusing on structural patterns. Similarly, Thomas Scialom et al. utilize named-entity placeholders in Self-Attention Architectures for Answer-Agnostic Neural Question Generation to prevent models from generating incorrect or hallucinated entities, forcing the architecture to prioritize underlying syntactic patterns.

This vocabulary reduction is particularly vital in low-resource settings where data sparsity makes memorizing unique words impossible. For instance, when restoring highly fragmented ancient cuneiform tablets, Ethan Fetaya et al. show in Restoration of Fragmentary Babylonian Texts Using Recurrent Neural Networks that replacing proper nouns, locations, and numerical values with generic category tokens allows the model to prioritize learning syntactic structures and formulaic patterns over memorizing unique, low-frequency vocabulary.

Original:  "King Nebuchadnezzar marched to Babylon with 500 soldiers."
Abstracted: "[TITLE] [PERSON] marched to [LOCATION] with [NUMBER] [NOUN]."

To formalize this architectural separation, Rizwan Parvez et al. in Building Language Models for Text with Named Entities split the language modeling task into a "type model" and an "entity composite model." By training the type model on a corpus where entities are replaced by their broader categories, the effective vocabulary size shrinks dramatically. In a subsequent study, Md Rizwan Parvez demonstrated in Building Language Models for Text with Named Entities that this reduction yields highly robust and accurate type predictions, proving that isolating entity types allows the network to learn general text patterns much more easily.


Learning Reusable Templates and Rhetorical Patterns

Beyond reducing vocabulary size, entity abstraction acts as a structural shield that prevents models from overfitting to specific keywords, enabling them to learn reusable templates that transfer across domains.

Consider the challenge of domain adaptation. If a model memorizes that a specific drug name belongs in a high-performing medical advertisement, it cannot transfer that persuasive structure to an ad about a different treatment. In The Automated Copywriter: Algorithmic Rephrasing of Health-Related Advertisements to Improve their Performance, Brit Youngmann et al. replace specific domain entities with generic tokens during training. This abstraction strategy forces the model to focus on the structural and rhetorical patterns of the text rather than specific factual content. The authors extend this framework in Algorithmic copywriting: automated generation of health-related advertisements to improve their performance, showing that abstracting domain-specific terms facilitates the creation of generalized templates that remain effective across entirely different subject areas within the same broad domain.

A similar principle applies to summarization tasks. Lu Wang et al. show in Domain-Independent Abstract Generation for Focused Meeting Summarization that replacing domain-dependent terms with generic tags allows a system to capture the general linguistic structure of summaries. Consequently, a model trained on one specific context can effectively summarize discussions in unrelated fields without requiring new training data.

In scientific extreme summarization, where human annotations are highly expensive, Yuning Mao et al. demonstrate in CiteSum: Citation Text-guided Scientific Extreme Summarization and Domain Adaptation with Limited Supervision that replacing specific entity names or author citations with a generic reference token prevents the model from overfitting to specific names. This encourages the learning of generalizable summarization patterns that adapt well across different scientific disciplines.


Disentangling Structure from Content for Transformations

In tasks where the goal is to manipulate the structure of a sentence while preserving its core constraints, entity abstraction isolates the entities so the neural network can focus entirely on the transformation logic. The specific entities are treated as passive passengers that do not affect the underlying grammatical rules.

[Input Message]  ──>  Entity Abstraction  ──>  [Abstracted Template]
                             │                               │
                      Extract Entities               Apply Transformation
                             │                               │
                             ▼                               ▼
[Saved Entities] ───────────────────────────>  Re-substitution & Output

For example, in natural language interfaces for databases, Srini Iyer et al. show in Learning a Neural Semantic Parser from User Feedback that mapping natural language questions to SQL queries becomes significantly more generalizable when specific entity mentions are replaced with type-based placeholders. The model learns the mapping from structural phrases to SQL syntax, and the specific entities are simply substituted back into the generated query before execution to maintain the constraints of the original request.

This decoupling is also crucial when changing the perspective of a sentence. In Converting the Point of View of Message Spoken to Virtual Assistants, Gunhee Lee et al. replace names and identifiers with placeholders during training. This allows the model to master the grammatical shifts required for point-of-view conversion (e.g., transforming "Tell Bob I love him" into "Joe says he loves you") independently of the specific names involved, ensuring the system generalizes perfectly to unseen names.

Finally, in classification tasks, Mark G. Lee et al. demonstrate in Integrating Question Classification and Deep Learning for improved Answer Selection that replacing named entities with generic placeholder tokens focuses the model's attention on the structural role of an entity rather than its unique surface form. Instead of getting bogged down by low-frequency words, the model learns the relationship between the question type and the presence of a relevant entity class, significantly reducing noise and improving selection accuracy.

Key Foundational Work

Domain-Independent Abstract Generation for Focused Meeting Summarization (2013) — contributed 2 concept elements to this direction.
Learning a Neural Semantic Parser from User Feedback (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_entity_abstraction_b76f0cd5,
  title = {entity abstraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/entity-abstraction/rdr_b76f0cd590b0478696bc60611e9c0c53},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_entity_abstraction_b76f0cd5_v1,
  title = {entity abstraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/entity-abstraction/rdr_b76f0cd590b0478696bc60611e9c0c53/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake