Compositional Label Generation via Sequence Decoding

37 concepts23 papers1 proposal

In standard machine learning, labeling is traditionally treated as a discrete classification problem. Given an input, a model computes a softmax distribution over a fixed, predefined vocabulary of atomic classes. This "closed-world" assumption works well for simple tasks, but it breaks down entirely when the label space is combinatorial, open-ended, or characterized by a heavy long tail.

For instance, in highly inflected languages like Sanskrit, morphological combinations can yield tens of thousands of valid tags (Gupta et al. in [Evaluating Neural Morphological Taggers for Sanskrit]). In syntactic parsing frameworks like Combinatory Categorial Grammar (CCG), the set of potential categories is theoretically infinite (Kogkalidis et al. in [Constructive Type-Logical Supertagging with Self-Attention Networks]). Similarly, real-world metadata like social media hashtags (Yu et al. in [Generating Hashtags for Short-form Videos with Guided Signals]), source code variable names (Jiang in [Unleashing the Power of Generative Model in Recovering Variable Names from Stripped Binary]), or descriptive topic labels (Alokaili et al. in [Automatic Generation of Topic Labels]) evolve constantly. A standard classifier can never predict a label it has not explicitly seen during training.

To break free from this constraint, a powerful paradigm shift has emerged: Generative Label Synthesis. Instead of treating a label as an indivisible black box, these approaches decompose labels into structured sequences of primitive sub-components, allowing models to construct entirely new labels token-by-token.

From Atomic IDs to Compositional Sequences

The core mechanism of generative label synthesis relies on linearizing complex target categories into sequences of tokens, characters, or logical operators. By doing so, the task shifts from standard classification to sequence generation or structured prediction.

Traditional Classification:
Input  ───► [ Neural Network ] ───► Class ID #34201 (Monolithic Black Box)

Generative Label Synthesis:
Input  ───► [ Neural Network ] ───► [Token 1] ───► [Token 2] ───► [Token 3] (Novel Composition)

For example, Bhargava et al. demonstrate this by breaking down complex CCG supertags into primitive grammatical operators using an encoder-decoder architecture ([Supertagging with CCG primitives]). Instead of choosing from a massive list of rare categories, their model generates the tag sequentially.

In a completely different domain, Huang et al. ([Tag Recommendation by Word-Level Tag Sequence Modeling]) and Wang et al. ([Microblog Hashtag Generation via Encoding Conversation Contexts]) model multi-word tags and hashtags as sequences of constituent words rather than atomic identifiers. This sequence-based decoding leverages internal semantic structures and word-level cues, enabling recommendation systems to synthesize entirely new tags for emerging topics.

Why It Works: Learning the Grammar of the Label Space

The fundamental reason this approach works is that it allows the model to learn the internal grammar or logic of the label space itself. When a model learns how primitive components interact, it gains the ability to assemble those components into novel configurations that were completely absent from the training data.

Consider how humans describe colors. We do not just memorize a fixed list of colors; we compose modifiers. Goodman et al. showed that sequence-based decoding allows a model to generate productive language like "faded teal" by combining its learned understanding of saturation and hue, even without direct training on that specific phrase ([Learning to Generate Compositional Color Descriptions]).

This compositional capability enables true zero-shot generalization across diverse modalities:

Productive Synthesis in Natural Language Generation

Generative label synthesis is particularly powerful for tasks where the ideal descriptor must be synthesized from scratch rather than extracted from a document.

In unsupervised topic modeling, traditional methods pick the most frequent words or rank existing Wikipedia titles as labels. However, Popa et al. ([BART-TL: Weakly-Supervised Topic Label Generation]) and Alokaili et al. ([Automatic Generation of Topic Labels]) use sequence-to-sequence transformers to synthesize completely novel, human-comprehensible descriptive phrases from topic word distributions. This mirrors earlier work by Mehdad et al., who utilized semantic aggregation to construct labels that do not appear verbatim in the source text ([Towards Topic Labeling with Phrase Entailment and Aggregation]).

Similarly, when expanding taxonomies, Zeng et al. found that constructing concept names token-by-token outperforms extraction methods, as it allows the system to generate valid concepts that are rare or entirely missing from the source corpus ([Enhancing Taxonomy Completion with Concept Generation via Fusing Relational Representations]).

This generative flexibility also solves classic machine translation and data augmentation challenges:

Key Foundational Work

Productive Generation of Compound Words in Statistical Machine Translation (2011) — contributed 2 concept elements to this direction.
PROVERB: The Probabilistic Cruciverbalist (1999) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 22 papers
  2. Version 1 · 2026-10-04 · 27 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_generative_topic_labeling_ddb6fb5a,
  title = {generative topic labeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-topic-labeling/rdr_ddb6fb5a56db4364a4ca00e2931aa9ef},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_generative_topic_labeling_ddb6fb5a_v2,
  title = {generative topic labeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-topic-labeling/rdr_ddb6fb5a56db4364a4ca00e2931aa9ef/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake