Abstract
This paper investigates the capacity of neural networks to reconstruct fluent sentences from unordered sets of words, a task known as "bag-to-sentence" transformation. While traditional NLP models are typically order-sensitive, humans can often derive meaning from scrambled text. We evaluate several architectures, including RNNs, pooling methods, and self-attention-based encoders, across three scenarios: normal, noisy, and missing-word bags. Our results show that self-attention mechanisms, specifically the proposed AttM model, effectively capture semantic dependencies to organize words. AttM outperforms traditional baselines and even surpasses human performance on large bags of words, although humans remain more robust when information is missing or noisy.
Introduction
A fundamental question in natural language processing is the extent to which word order is necessary for understanding. Most modern AI models are designed to process text sequentially. However, many real-world applications—such as generating summaries from keywords, reordering phrases in machine translation, or assistive communication—require generating coherent text from unordered or partially ordered information. This study explores whether neural networks can emulate the human ability to organize a "bag of words" into a fluent sentence.
Task Formulation
The researchers define the bag-to-sentence problem through three scenarios of increasing difficulty:
- Normal: The input is a perfectly shuffled bag containing all words from the original sentence.
- Noise: The bag includes additional, irrelevant words.
- Missing: The bag is incomplete, lacking essential words for the final sentence.
Methodology
The study utilizes an encoder-decoder framework with various encoding strategies:
- Baselines: Standard LSTMs (RNNs), simple Max-Pooling of word embeddings, and iterative Memory-based attention.
- Self-Attention: The core innovation involves using self-attention as an order-invariant encoder. By omitting positional encodings, the model must relate words based solely on semantic dependencies (e.g., subject-verb relationships).
- Proposed Models: Two variants are introduced: AttP (self-attention with pooling) and AttM (self-attention with memory-based fusion).
- Decoder: A pointer-based decoder is used to either generate words from a fixed vocabulary or "copy" them directly from the input bag.
Results and Analysis
The self-attention models, particularly AttM, consistently outperformed other architectures across BLEU, ROUGE-L, and Perfect Matching Rate (PMR) metrics.
As the size of the input bag increases, the combinatorial complexity grows exponentially. Interestingly, AttM outperformed humans on the "Normal" scenario for bags containing 13-20 words. While humans excel at small-scale linguistic puzzles, the neural model is more effective at managing the vast search space of larger bags.
In terms of robustness, models handle "Noise" (filtering out extra words) much better than "Missing" information. Reconstructing a sentence with missing words requires "hallucinating" contextually appropriate tokens, a task where humans still maintain an edge due to broader world knowledge and common-sense reasoning.
Conclusion
This empirical study demonstrates that self-attention is a powerful tool for encoding unordered semantic information. While neural models show superior performance in handling large-scale combinatorial ordering, bridging the gap in robustness—specifically in scenarios with missing information—remains a key challenge for future research. Incorporating external knowledge bases could be a promising direction for improving the model's ability to infer missing context.
Summary updated