COD3S: Diverse Generation with Discrete Semantic Signatures

Nathaniel Weir, João Sedoc, Benjamin Van Durme

Abstract

We present COD3S, a novel method for generating semantically diverse sentences using neural sequence-to-sequence (seq2seq) models. Conditioned on an input, seq2seq models typically produce semantically and syntactically homogeneous sets of sentences and thus perform poorly on one-to-many sequence generation tasks. Our two-stage approach improves output diversity by conditioning generation on locality-sensitive hash (LSH)-based semantic sentence codes whose Hamming distances highly correlate with human judgments of semantic textual similarity. Though it is generally applicable, we apply COD3S to causal generation, the task of predicting a proposition's plausible causes or effects. We demonstrate through automatic and human evaluation that responses produced using our method exhibit improved diversity without degrading task performance.

Introduction

Neural sequence-to-sequence (seq2seq) models are notorious for "playing it safe." When tasked with open-ended generation—such as predicting the cause or effect of an event—standard decoding methods like beam search tend to produce candidates that are nearly identical in meaning, differing only by minor syntactic tweaks. In the field of causal generation (e.g., "The glass fell because..."), this lack of variety is a major hurdle. Ideally, a model should explore multiple plausible realities, such as "it was pushed," "it was slippery," or "the shelf broke."

While previous research has addressed syntactic diversity, achieving true semantic diversity—generating fundamentally different ideas—remains difficult. COD3S (COnstrained Decoding with Semantic Sentence Signatures) addresses this by treating the semantic space as a searchable hierarchy of discrete "bins."

Methodology: The COD3S Framework

The authors propose a two-stage approach that discretizes the semantic space to force the model into distinct conceptual regions.

1. Semantic Signatures via LSH

The framework uses Sentence-BERT (SBERT) to map sentences into a high-dimensional vector space. To make this space searchable and discrete, Locality-Sensitive Hashing (LSH) is applied, converting the SBERT vector into a short binary signature (e.g., 16 bits). Because LSH preserves cosine similarity, sentences with similar meanings are mapped to the same or similar bitstrings.

2. Two-Stage Decoding

During training, target sentences are prefixed with their own binary signatures, teaching the model to associate specific bit patterns with specific semantic content. At inference time, the process is split into two distinct steps:

  • Stage 1 (Signature Inference): The model first predicts the most likely semantic signatures (bins) for a given input.
  • Stage 2 (Conditional Sequence Inference): For each predicted signature, the model performs a separate beam search to generate the actual sentence, conditioned on that specific bitstring.
The COD3S framework: (a) Mapping sentences to discrete signatures via SBERT and LSH; (b) The two-stage inference process predicting signatures then sequences.

To ensure outputs are both diverse and relevant, a bidirectional Maximum Mutual Information (MMI) objective is used for reranking, penalizing generic or "bland" responses.

Experimental Evaluation

The model was trained on CausalBank, a corpus of 10 million causal sentence pairs, and evaluated on the COPA (Choice of Plausible Alternatives) benchmark.

Key Results

  • Superior Diversity: COD3S outperformed standard seq2seq models and diversity-promoting baselines (like DPC) across lexical (BLEU-based) and semantic (SBERT-based) metrics.
  • Human Evaluation: Amazon Mechanical Turk studies showed that humans rated the top-10 results from COD3S as containing more high-quality, distinct causal completions than baseline models.
  • Signature Fidelity: The 16-bit signatures maintained a Spearman correlation of .652 with human judgments of semantic similarity, confirming that these discrete bins capture meaningful nuances.
Performance comparison on the COPA benchmark across various perplexity (PPL) and oracle metrics, showing COD3S variants against baselines.

Discussion and Limitations

One of the primary advantages of COD3S is the interpretability of its semantic bins. Each bin can be represented by a "medoid"—a representative sentence from the training set—allowing researchers to preview the semantic space the model intends to explore.

However, the model faces a "signature bottleneck": performance is heavily dependent on the first stage of predicting the correct signatures. If the predicted signature is a poor fit for the input, the resulting sentence may be semantically strained. Future work may explore more sophisticated sampling methods for signatures to further refine the transition from continuous latent spaces to discrete semantic bins.

Summary updated

Abstract

We present COD3S, a novel method for generating semantically diverse sentences using neural sequence-to-sequence (seq2seq) models. Conditioned on an input, seq2seq models typically produce semantically and syntactically homogeneous sets of sentences and thus perform poorly on one-to-many sequence generation tasks. Our two-stage approach improves output diversity by conditioning generation on locality-sensitive hash (LSH)-based semantic sentence codes whose Hamming distances highly correlate with human judgments of semantic textual similarity. Though it is generally applicable, we apply COD3S to causal generation, the task of predicting a proposition's plausible causes or effects. We demonstrate through automatic and human evaluation that responses produced using our method exhibit improved diversity without degrading task performance.
BibTeX
@inproceedings{weir2020cod,
  author = {Nathaniel Weir and João Sedoc and Benjamin Van Durme},
  title = {{COD3S: Diverse Generation with Discrete Semantic Signatures}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=sUbD0jZfOY},
}
Figure