In the evolving landscape of Natural Language Processing (NLP), Large Language Models (LLMs) have demonstrated remarkable fluency. However, "steering" these models to follow specific structural constraints remains a significant challenge. While existing constrained generation techniques allow users to specify which keywords should appear, they rarely offer control over where those keywords should be placed. Researchers from Hitachi have addressed this gap by proposing a task-independent method to control both the inclusion and the specific relative position of keywords in generated text.
The Challenge of Structural Control
Effective communication often depends on the strategic placement of information. For instance, a news summary is most effective when an "eye-catching" keyword appears in the lead sentence, while supplementary details are relegated to the middle or end. Current models lack a native mechanism for this level of structural precision. Furthermore, models often fail to include requested keywords if they cannot find a statistically probable location for them. The authors argue that by specifying a target position, the model is actually better equipped to integrate these words naturally into the narrative flow.
Methodology: Length and Position Tokens
The core innovation of this research is the introduction of task-independent Control Tokens. This method was validated using two distinct architectures: BART (for summarization) and GPT-2 (for story generation).
1. Length Tokens ([LENGTHn])
The model quantizes the target text length into units of five words. For example, a target output of 50–54 words is represented by the token [LENGTH50]. This provides the model with a structural "budget" before generation begins.
2. Position Tokens ([POSITIONn])
Rather than using absolute word indices (which are difficult for humans to estimate and for models to generalize), the authors used relative positions. The text is divided into 10% increments. A token like [POSITION0] indicates the first 10% of the text, while [POSITION90] indicates the final 10%.
The Workflow
During the training phase, the model is exposed to keywords extracted from ground-truth text, paired with tokens representing their actual positions and the total length of the reference. This allows the model to learn the statistical relationship between these markers and the resulting syntax. At inference, a user provides a source document or prompt followed by a control string, such as: [LENGTH50] [SEP] miracle dog [POSITION0] fundraising [POSITION80].
Experimental Evaluation
The method was evaluated across three diverse benchmarks to test its versatility:
- CNN/DailyMail: Multi-sentence news summarization.
- XSum (Extreme Summarization): Highly compressed, single-sentence summaries.
- ROCStories: Five-sentence commonsense stories, testing creative generation without a source document.
Key Insights
- The Synergy of Length and Position: A critical finding was that position control is significantly less effective without length control. Without knowing the total length of the output, the model cannot accurately calculate where a specific percentage (e.g., the "70% mark") actually falls.
- Position as an Anchor: Providing a position token unexpectedly increased the keyword inclusion rate. By assigning a specific "slot" for a word, the model is less likely to "forget" the constraint compared to simply providing a list of keywords with no placement instructions.
- Stylistic Flexibility: Case studies demonstrated that shifting a keyword's position naturally alters the model's tone. Moving a phrase like "true miracle dog" from the beginning to the end of a summary shifted the output from a "hook-style" news lead to a chronological conclusion.
Results and Performance
The proposed method (+Pos+Len) consistently outperformed keyword-only baselines:
- Keyword Inclusion: On the CNN/DM dataset, the inclusion rate for three keywords rose from 39.1% (keyword-only) to 70.6% with position and length tokens.
- Positional Accuracy: In the XSum task, the model successfully placed keywords in the exact 10% bucket requested 64.1% of the time, more than doubling the performance of the baseline (28.7%).
- ROUGE Scores: Summaries generated with position control achieved higher ROUGE scores, suggesting that following positional constraints leads to structural choices that more closely mirror human-written references.
Limitations and Future Directions
Despite the success of the method, the authors noted a "Back of the Text" problem: the model is significantly more accurate at placing keywords at the beginning of a text than at the end. This is likely because, as generation progresses, the model becomes increasingly constrained by the context of the words it has already generated, leaving less room to force in a specific keyword late in the sequence.
Additionally, the current experiments often rely on "Oracle" data (where target lengths and positions are known from a reference). Future work will focus on models that can automatically determine the optimal keywords and positions to maximize reader engagement without requiring manual user input for every token.
Summary updated