Steering Text Generation via Vocabulary Biasing

25 concepts21 papers1 proposal

Autoregressive language models generate text token by token, conditioning each new word on the preceding sequence. While highly effective for local fluency, this sequential approach often struggles to maintain global thematic consistency. Without an overarching anchor, decoders can drift off-topic, generate repetitive or generic responses (such as "I see" or "Me too"), or fail to incorporate domain-specific terminology.

To address this, a prominent research direction focuses on vocabulary biasing and topic-constrained decoding. Instead of relying solely on the model's internal sequential context, these techniques inject global semantic constraints directly into the decoding process, steering the model toward specific themes, domains, or keywords.

The Core Mechanism: Logit and Probability Manipulation

The most direct way to guide a decoder is by modifying its output probability distribution at each generation step. Normally, a language model outputs a vector of unnormalized log-probabilities (logits) over the entire vocabulary. These logits are passed through a softmax layer to yield the final token probabilities.

By adding a bias vector or a prior probability distribution directly to these logits before the softmax operation, we can systematically alter the landscape of what the model is likely to say next.

$$\text{P}(w_t) = \text{softmax}(\text{logits} + \mathbf{b})$$

Where $\mathbf{b}$ is a vocabulary-wide bias vector that assigns positive values to target words and zero or negative values elsewhere.

Standard Logits:  [ cat: 1.2,  run: 0.4,  the: 3.5,  bonds: -2.1 ]
Topic Bias (Fin): [ cat: 0.0,  run: 0.0,  the: 0.0,  bonds:  4.5 ]
------------------------------------------------------------------
Biased Logits:    [ cat: 1.2,  run: 0.4,  the: 3.5,  bonds:  2.4 ]

This mechanism acts as a soft constraint. It does not rigidly force the model to output a specific word at an inappropriate time. If the language model's internal state strongly demands a grammatical token (like a punctuation mark or a preposition), the underlying logit for that token will still dominate. However, when the model faces a choice between multiple semantically valid words, the bias tips the scale toward the desired topic.

This foundational approach has been adapted across various tasks:

  • Dialogue Systems: Xing et al. introduced a topic-specific bias distribution to the standard word generation probability, significantly reducing generic responses in favor of content-rich replies.
  • Domain Adaptation: Zhou et al. utilized a category-based probability boost to ensure generated advertising keywords adhered strictly to a target commercial domain.
  • Machine Translation: Wei et al. applied a target-vocabulary bias to favor words aligning with the source sentence's global topic space, preventing errors like premature sequence termination.
  • Multimodal Generation: Xu et al. added a bias probability to encourage the selection of characters semantically related to visual keywords extracted from an input image.
  • Knowledge Integration: Chen et al. derived a vocabulary bias vector from an external recommender system's hidden state, nudging a dialogue decoder toward domain-relevant entities.
  • Stylistic Control: Hoang demonstrated that modifying the final softmax layer with an agent-specific bias term can successfully mimic the unique vocabulary preferences and word frequencies of individual human translators.

Shifting Internal Representations and Latent Spaces

While modifying the final softmax layer is computationally efficient, an alternative strategy injects thematic guidance deeper within the network architecture. Instead of altering the final vocabulary probabilities directly, these methods modulate the hidden states or latent variables that feed into the output layer.

For instance, Wang et al. developed a method to shift the hidden states of a Transformer decoder using a bias vector calculated from a document's overall topic distribution. By altering the internal representations before they reach the output layer, the model's attention mechanisms naturally align with the overarching theme.

Other approaches leverage generative latent spaces to enforce consistency:

  • Stochastic Latent Structures: Li et al. combined deterministic hidden states with a stochastic latent structure model to capture underlying semantic frameworks, reducing the production of loose or irrelevant content.
  • Discrete Codebooks: Yoo et al. integrated a latent topic distribution variable to govern the selection of discrete codebook entries, enforcing global semantic consistency across generated components.

Training Objectives for Semantic Adherence

To ensure that decoders actually respond to these biases without suffering from "latent variable collapse"—a common failure mode where the autoregressive decoder learns to ignore external conditioning signals—researchers employ specialized loss functions and training objectives.

One strategy relies on explicit optimization constraints. Xu et al. utilized a log-likelihood loss that explicitly encourages the decoder to incorporate words associated with identified latent topics. To enforce precise conversational trajectories, Yin et al. introduced a divergence-based loss function that pulls the decoder's internal representations toward the centroid of a target topic cluster during training.

Alternatively, adversarial or discriminator-based objectives can be used. Tang et al. introduced a topic discriminator that penalizes the generator if the output text deviates from the intended semantic distribution. This architectural constraint forces the decoder to remain faithful to the provided latent code throughout the entire sequence.

[Latent Code] ──> [Decoder] ──> [Generated Text] ──> [Topic Discriminator]
                                                            │
                                                   (Penalizes deviation
                                                    from target topic)

Applications Across Domains

The ability to bias vocabulary distributions provides a robust, interpretable lever for steering text generation across a wide variety of NLP applications:

  • Abstractive Summarization: Conditioning decoders on global document themes derived from external modeling techniques (like Latent Dirichlet Allocation) helps maintain thematic consistency in extreme or abstractive summarization tasks (Narayan et al.; Wang et al.).
  • In-Context Learning & Translation: In statistical machine translation, topic-relevant vocabulary rewards act as a soft constraint to nudge models toward domain-consistent terminology (Gong et al.). In modern large language models, Aycock and Bawden demonstrated that augmenting prompts with keywords derived from the most probable topic serves as a stochastic proxy for a bilingual lexicon, narrowing down domain-specific terminology.
  • Specialized Text Generation: Vocabulary biasing is highly effective when text must align with specific non-textual attributes. This includes biasing decoders toward acoustic event keywords in audio captioning (Han et al.), aspect-specific product features in personalized review generation (Ni et al.), or sentiment-heavy words via targeted copy mechanisms (Li et al.).

Key Foundational Work

Topic Aware Neural Response Generation (2017) — contributed 2 concept elements to this direction.
Cache-based Document-level Statistical Machine Translation (2011) — contributed 1 concept elements to this direction.
Deep Recurrent Generative Decoder for Abstractive Text Summarization (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_topic_constrained_decoding_affb9c56,
  title = {topic-constrained decoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/topic-constrained-decoding/rdr_affb9c5613c34b5d83505f7310f72411},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_topic_constrained_decoding_affb9c56_v1,
  title = {topic-constrained decoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/topic-constrained-decoding/rdr_affb9c5613c34b5d83505f7310f72411/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake