Abstract
We propose a novel conditioned text generation model. It draws inspiration from traditional template-based text generation techniques, where the source provides the content (i.e., what to say), and the template influences how to say it. Building on the successful encoder-decoder paradigm, it first encodes the content representation from the given input text; to produce the output, it retrieves exemplar text from the training data as "soft templates," which are then used to construct an exemplar-specific decoder. We evaluate the proposed model on abstractive text summarization and data-to-text generation. Empirical results show that this model achieves strong performance and outperforms comparable baselines.
Introduction
Neural text generation models, such as those used for summarization and translation, frequently struggle with a "genericness" problem. While modern encoder-decoder architectures are highly fluent, they often produce bland or uncontrolled outputs. Historically, template-based systems were used to ensure structural precision, but these systems lack the scalability required for diverse datasets.
Recent "exemplar-based" neural models have attempted to bridge this gap by retrieving similar training examples and concatenating them to the input. However, simply attending to an example primarily aids in content selection rather than structural guidance. This paper introduces Exemplar-based Adaptive Decoding (ADADEC), a method that treats exemplars as "soft templates" to guide the stylistic and structural aspects of generation by dynamically reshaping the decoder's parameters.
Methodology
The core innovation of ADADEC is the dynamic construction of decoder weight matrices for every input, rather than using fixed weights. The process follows a four-step pipeline:
- Retrieval: For a given input, the system uses Information Retrieval (Bag-of-Words and Cosine similarity) to find the most similar example in the training set.
- Coefficient Computation: An encoder processes this exemplar to produce a vector, which is then used to calculate a set of coefficients ((\lambda)).
- Dynamic Parameter Construction: The decoder's weight matrices are constructed as a weighted sum of several "basis" matrices, guided by the (\lambda) coefficients.
- Rank-1 Constraints: To maintain parameter efficiency, basis matrices are constrained to be rank-1 (the outer product of two vectors). This allows the model to simulate a massive mixture of experts without a proportional increase in parameters.
By modifying the decoder's internal logic based on the exemplar, the model is forced to learn structural patterns rather than merely copying words.
Experimental Setup
The authors evaluated ADADEC on three major benchmarks:
- Annotated Gigaword: Abstractive sentence summarization of news snippets.
- New York Times (NYT) Annotated Corpus: Long-form abstractive summarization.
- WikiBio: A data-to-text task generating the first sentence of a Wikipedia biography from structured infobox data.
Results and Discussion
ADADEC demonstrated consistent improvements over standard sequence-to-sequence baselines across all tasks. On the Gigaword dataset, ADADEC achieved a ROUGE-L score of 34.7, significantly outperforming the standard Seq2Seq baseline of 33.5.
In the WikiBio data-to-text task, the model reached a BLEU score of 43.6 (vs. 42.5 baseline). When combined with "Attention-based Exemplars" (ATTEXP), which helps the model copy specific relevant words, performance further increased to 44.1. Qualitative analysis confirmed that swapping exemplars could force the model to change sentence tense or structural complexity, proving the decoder truly adapts to the exemplar's style.
Conclusion
ADADEC offers a robust framework for separating content and style in text generation. By using exemplars as soft templates to tune decoder parameters, it achieves structural consistency without the brittleness of manual templates. Future work may address the retrieval bottleneck by using neural rerankers and optimizing the inference overhead associated with semi-parametric retrieval.
Summary updated