Standard language models are often "black boxes": you provide an input, and the model generates a response based on the most likely distribution of words it saw during training. While fluent, this process lacks a steering wheel. If you need a summary to be exactly two sentences long, or a medical report to prioritize a specific diagnosis, a standard model offers no easy way to enforce those constraints.
To solve this, researchers have developed a paradigm of controllable generation that uses discrete attribute tags—specialized markers prepended to the input—to act as "knobs" for the model’s behavior.
The Intuition: Prepending a Global State
The core idea is deceptively simple: if you want a model to change its output style, tell it what style to use before it writes a single word. By prepending a discrete token like <Length_Short> or <Grade_5> to the source text, we provide a "global state" that the model can attend to throughout the entire generation process.
In Controllable Abstractive Summarization, Angela Fan and colleagues demonstrated that these markers allow users to regulate high-level attributes like length and entity focus without changing the underlying model architecture. Because of the self-attention mechanism in Transformers, every word the model generates "looks back" at these initial control tokens. The tokens shift the probability distribution of the vocabulary, making "concise" words more likely when a <Short> tag is present.
Multi-Dimensional Steering
The power of this approach lies in its compositionality. You are not limited to a single knob; you can stack them to achieve precise, multi-faceted control.
For instance, the ACCESS model by Benoît Sagot and Louis Martin uses multiple tokens to control sentence simplification. A user can simultaneously specify the desired amount of paraphrasing, the reduction in sentence length, and the difficulty of the vocabulary. Similarly, Dang Nguyen and Tianyi Zhou introduced MCTune, which uses linguistic metadata—such as noun density and word count—as discrete tags. They found that training models to follow these structural constraints actually improves the model's overall reasoning and quality, as it forces the model to better plan its output.
Grounding Generation in Clinical and Technical Truth
In high-stakes domains like medicine, "style" is secondary to "fact." Researchers have adapted attribute-guided generation to ensure clinical accuracy. In PromptMRG, Haibo Jin and Hao Chen use a classification branch to predict specific medical attributes from an image. These attributes are converted into discrete tokens and injected into the text decoder. This ensures the generated report is "diagnosis-driven," acting as a hard constraint that prevents the model from hallucinating a healthy lung when the classifier detects an opacity.
This mechanism extends to technical domains as well. Colin B. Clement and the team behind PyMT5 use control tokens to switch between different "modes" of code generation, such as translating a docstring into a Python method or vice versa. By signaling the target format (e.g., <Docstring_to_Code>), a single model can master multiple distinct tasks that would otherwise require separate specialized networks.
From Black-Box to Symbolic Planning
While simple tags work for style, complex tasks often require a "divide and conquer" strategy. Amit Moryossef and Yoav Goldberg proposed separating planning from realization. In their framework, a symbolic planner first decides the order and structure of information (the "plan"), which is then fed as a sequence of control signals to a neural model.
This hybrid approach addresses a fundamental weakness of neural networks: their tendency to lose track of the "big picture." By using discrete tags to represent a structural plan, we gain the best of both worlds—the linguistic fluency of deep learning and the predictable, verifiable structure of symbolic logic. Whether it is generating diverse news headlines using keyphrases as signals (Dayiheng Liu) or guiding a chatbot's recommendations through predicted attribute slots (Shirley Anugrah Hayati), discrete tags transform the generative process from a random walk into a guided journey.