Text Generation with Exemplar-based Adaptive Decoding

Hao Peng, Ankur P. Parikh, Manaal Faruqui, Bhuwan Dhingra, Dipanjan Das

Abstract

Neural text generation models are fluent but often lack structural consistency, while template-based systems are precise but unscalable. This paper introduces ADADEC (Exemplar-based Adaptive Decoding), a framework that bridges this gap. ADADEC uses retrieved exemplars to dynamically reparameterize the decoder, creating "soft templates" that guide the style and structure of generated text. Evaluated on summarization and data-to-text tasks, ADADEC consistently outperforms standard sequence-to-sequence models and strong exemplar-based baselines.

Introduction

The standard encoder-decoder paradigm in neural text generation excels at fluency but struggles with consistency, often producing generic responses or hallucinating facts due to a lack of structural guidance. Conversely, traditional template-based systems offer high precision but are brittle and difficult to scale to open-domain tasks. Recent "retrieve-and-edit" approaches attempt to use exemplars (similar training examples) by concatenating them to the input. However, this often fails to capture the stylistic and structural nuances of the exemplar. ADADEC proposes a more fundamental integration: using the exemplar to adapt the model's parameters themselves.

Methodology: Adaptive Decoding

The core of ADADEC is the dynamic construction of the decoder's weight matrices based on a retrieved exemplar. This process transforms the exemplar into a stylistic blueprint for the generation task.

Algorithm 1: The four-step process for constructing the adaptive decoder.

The process follows four main steps:

  1. Retrieval: For a given input, the system finds the most similar source in the training set using Bag-of-Words and cosine similarity, then retrieves its corresponding target (the "exemplar").
  2. Encoding: A small RNN processes the exemplar into a vector representation.
  3. Coefficient Computation: This vector determines a set of coefficients ($\lambda$).
  4. Adaptive Parameter Construction: The decoder's weight matrices are constructed as a weighted sum of several "basis" matrices, scaled by these coefficients.

To maintain efficiency, the authors employ Rank-1 constraints, decomposing large basis matrices into outer products of vectors. This allows for an expressive adaptive decoder without a significant increase in parameter count compared to a standard LSTM.

Experimental Setup

The framework was evaluated on three benchmarks:

  • Annotated Gigaword: Abstractive sentence summarization (short headlines).
  • New York Times (NYT): Multi-sentence summarization with long source documents.
  • Wikibio: Generating Wikipedia biography opening sentences from infobox tables.

Results

ADADEC demonstrates superior performance across all tasks, effectively separating "what" to say (content from the encoder) from "how" to say it (style from the adaptive decoder).

Performance comparison on the Annotated Gigaword dataset.

On the Gigaword dataset, ADADEC achieved a ROUGE-1 score of 37.3, matching state-of-the-art models that use complex handcrafted features. In the NYT task, it reached a ROUGE-L of 26.4, outperforming previous extractive and abstractive models. For Wikibio, the model achieved 43.6 BLEU, which increased to 44.1 when combined with attention over the exemplar (ATTEXP). Qualitative analysis confirmed that the model successfully adopts the syntactic structures of its exemplars, such as adjusting brevity or phrasing based on the retrieved example.

Discussion and Future Work

While ADADEC provides a powerful mechanism for structural guidance, it currently relies on a static retrieval process. Future improvements could involve neural rerankers for better exemplar selection or integrating multiple exemplars to offer a broader range of stylistic choices. Additionally, while the rank-1 trick manages parameter growth, the retrieval step remains a potential bottleneck for large-scale real-time deployments.

Summary updated

BibTeX
@inproceedings{peng2019text,
  author = {Hao Peng and Ankur P. Parikh and Manaal Faruqui and Bhuwan Dhingra and Dipanjan Das},
  title = {{Text Generation with Exemplar-based Adaptive Decoding}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=Bkb06X-OWB},
}
Figure