Controlling Sequence-to-Sequence Models - A Demonstration on Neural-based Acrostic Generator

Liang-Hsin Shen, Pei-Lun Tai, Chao-Chung Wu, Shou-De LinEMNLP 2019 · OpenReview

Abstract

An acrostic is a form of writing that the first token of each line (or other recurring features in the text) forms a meaningful sequence. In this paper we present a generalized acrostic generation system that can hide certain message in a flexible pattern specified by the users. Different from previous works that focus on rule-based solutions, here we adopt a neural- based sequence-to-sequence model to achieve this goal. Besides acrostic, users are also allowed to specify the rhyme and length of the output sequences. Based on our knowledge, this is the first neural-based natural language generation system that demonstrates the capability of performing micro-level control over output sentences.

Introduction

Neural text generators, particularly Sequence-to-Sequence (Seq2Seq) models like the Transformer, are excellent at producing fluent, human-like text. However, they are notoriously difficult to control. Imposing "hard constraints"—such as specific word placements or rhyming schemes—is non-trivial because these models operate primarily on probabilistic distributions.

Traditional approaches to acrostics relied on rule-based systems or synonym replacement, which often resulted in unnatural or clunky text. This research, presented at EMNLP 2019, demonstrates that standard Seq2Seq models can be trained to follow "micro-level" constraints simply by appending control signals to the input, allowing for both creative expression and strict structural obedience.

Methodology

The authors utilize a Transformer-based Seq2Seq model with a modified input representation. Instead of training on standard pairs of consecutive lines ($S_1 \to S_2$), they append explicit control signals to the input sequence.

The Transformer-based architecture showing how control signals (Position, Rhyme, Length) are appended to the input embedding to guide the generation process.

These signals define three key constraints:

  1. Acrostic Signal: Specifies the token and its intended position (e.g., 1 Telling).
  2. Rhyme Signal: A formal tag representing a specific rhyme group (e.g., IYN).
  3. Length Signal: A digit representing the desired number of tokens in the line.

The training format follows the pattern: Input Line || Position/Token || Rhyme || Length -> Output Line. By training on millions of these pairs, the model's attention mechanism naturally learns to map these arbitrary tags to the structure of the generated output.

System Implementation

The system is implemented as a web-based demonstration consisting of a front-end for user input and a back-end server for iterative generation.

Overview of the system architecture, illustrating the interaction between the front-end website for pattern specification and the back-end server's generation loop.

Users can specify a theme, a hidden message, and a "Hidden Pattern" via a grid interface. The back-end then generates the lyrics line-by-line, using the previously generated line as context for the next, while adhering to the user-defined constraints.

Experimental Setup

The researchers evaluated the system using two large-scale lyric datasets:

  • Chinese Lyrics: 651,339 training pairs with a 50,000-character vocabulary.
  • English Lyrics: 1,000,000 training pairs with a 50,000-word vocabulary.

Results and Discussion

The system achieved remarkably high accuracy in satisfying user constraints across both languages:

Condition Accuracy (Chinese) Accuracy (English)
Word/Char Position 99.38% 98.21%
Rhyme 99.31% 97.67%
Sentence Length 99.90% 99.85%

Key insights from the study include:

  • Implicit Constraint Learning: The Transformer architecture can learn hard rules without specialized loss functions if provided with structured input signals.
  • Micro-level Control: This work proves that fine-grained positional accuracy is possible within a standard generative framework.
  • Language Variation: English proved slightly more difficult than Chinese, likely due to stricter grammatical rules compared to the more flexible structure of Chinese lyrics.

Conclusion

This research demonstrates a robust method for imposing micro-level constraints on Seq2Seq models. By integrating control signals directly into the input, the system can generate fluent, rhyming acrostics with flexible hidden patterns, marking a significant step forward in controllable natural language generation.

Summary updated

Abstract

An acrostic is a form of writing that the first token of each line (or other recurring features in the text) forms a meaningful sequence. In this paper we present a generalized acrostic generation system that can hide certain message in a flexible pattern specified by the users. Different from previous works that focus on rule-based solutions, here we adopt a neural- based sequence-to-sequence model to achieve this goal. Besides acrostic, users are also allowed to specify the rhyme and length of the output sequences. Based on our knowledge, this is the first neural-based natural language generation system that demonstrates the capability of performing micro-level control over output sentences.
BibTeX
@inproceedings{shen2019controlling,
  author = {Liang-Hsin Shen and Pei-Lun Tai and Chao-Chung Wu and Shou-De Lin},
  title = {{Controlling Sequence-to-Sequence Models - A Demonstration on Neural-based Acrostic Generator}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=qDFf3O7o6Z},
}
Figure