Introduction
Standard neural sequence-to-sequence (seq2seq) models often suffer from a "homogeneity" problem. In open-ended tasks like causal reasoning, typical decoding methods such as beam search tend to produce candidates that are nearly identical in meaning, even if they use different words. For instance, given the input "the man was hungry," a model might output "he wanted food" and "he wanted to eat," failing to explore more diverse but equally plausible causal paths.
COD3S (COnstrained Decoding with Semantic Sentence Signatures) is a novel framework designed to force models to explore distinct semantic neighborhoods. By conditioning generation on discrete binary "signatures" derived from sentence embeddings, the model can produce a wide array of semantically diverse responses.
Methodology: Discrete Semantic Signatures
The COD3S framework utilizes a two-stage approach to bridge the gap between continuous semantic embeddings and discrete token generation.
- Semantic Embedding and Discretization: Sentences are first mapped into a high-dimensional space using Sentence-BERT (SBERT), where cosine distance correlates with semantic similarity. These continuous vectors are then converted into short binary strings (signatures) using Locality-Sensitive Hashing (LSH). This ensures that sentences with similar meanings share similar binary signatures.
- Two-Stage Decoding: During inference, the model first performs a beam search to identify the $k$-best signatures, enforcing a Hamming distance threshold to ensure semantic diversity. In the second stage, the model performs a conditional beam search for each signature to generate the most likely sentence belonging to that specific semantic "bin."
To further refine relevance, the authors implement a Maximum Mutual Information (MMI-bidi) objective, which encourages the model to select signatures and sentences that are uniquely tied to the specific input rather than generic common phrases.
Evaluation and Key Results
The model was trained on CausalBank, a dataset of 10 million cause-and-effect pairs, and evaluated on an out-of-distribution split of the COPA (Choice of Plausible Alternatives) benchmark.
Quantitative results show that COD3S significantly outperforms standard seq2seq models in semantic diversity. Using SBERT-based cosine distance as a metric, COD3S produced nearly 10 distinct ideas out of 10 attempts, whereas standard beam search often dropped to fewer than 2 distinct ideas as similarity thresholds tightened.
Human evaluations via Amazon Mechanical Turk confirmed these findings. Judges rated the diverse outputs from COD3S as nearly as plausible as the top outputs from standard models. Notably, the 10th-best output from COD3S was frequently rated as more plausible than the 3rd-best output from a standard model, demonstrating the framework's ability to surface high-quality, non-obvious candidates.
Discussion and Future Work
While COD3S successfully addresses semantic homogeneity, it faces challenges in signature inference. Predicting the "correct" semantic signature is difficult, as the model's confidence across different signatures is often flat. Additionally, despite the MMI objective, the model occasionally defaults to generic predicates.
Future research may explore alternative methods for signature inference and apply the COD3S framework to other open-ended domains such as story generation or image captioning. The bit-sequence hierarchy inherent in LSH signatures offers a promising path for finer-grained control over semantic generation.
Summary updated