In modern natural language processing, we often treat every problem as a sequence-to-sequence task. Whether we are translating a sentence or extracting facts, we feed a sequence into a model and expect a sequence to come out. However, many real-world tasks—like tagging an image, identifying keyphrases in a document, or predicting the semantic components (sememes) of a word—do not have a natural order. They are sets.
When we force these unordered sets into a sequential format (e.g., alphabetical or frequency-based), we introduce a "false order bias." This bias forces models to learn artificial dependencies, where the prediction of one label depends on its arbitrary position rather than its semantic relevance.
The Problem of False Order Bias
Standard autoregressive models, such as the Transformer or RNNs, are trained using a maximum likelihood estimation (MLE) that penalizes the model based on the exact position of tokens. As Xuancheng Ren and colleagues point out in their work on Sememe Prediction, this creates a mismatch: the model is penalized for predicting a correct label if it appears in the "wrong" order.
This serialization creates several issues:
- Spurious Causality: As noted by Xianlong Luo in the SADD framework, generative models develop a "spurious causal dependency" where the model believes a subsequent extraction depends on a previous one, even when the items are independent in the source text.
- Optimization Difficulty: Forcing a model to memorize a specific permutation (like alphabetical order) adds an unnecessary layer of complexity. Jiacheng Ye argues in One2Set that this forces the model to waste capacity on learning the list structure rather than the underlying semantic relationships.
- Brittle Generalization: If the sorting criteria change, the model's performance often collapses. Pengcheng Yang demonstrated this sensitivity in Seq2Set, showing that standard Seq2Seq models are "handcuffed" by arbitrary label orders.
Transitioning from Sequences to Sets
To solve this, researchers have moved toward "predict-and-assign" paradigms that treat the output as a collection of elements rather than a chain.
Bipartite Matching and Alignment
One of the most effective strategies is to allow the model to predict a set of labels in any order and then use a matching algorithm to align them with the ground truth. In One2Set, Jiacheng Ye uses a bipartite matching mechanism (the Hungarian algorithm) during training. This ensures the model is only penalized for the content of its predictions, not their sequence. Similarly, Arnau Ramisa introduced Predicted Label Alignment (PLA), which dynamically matches ground truth labels to the model's predictions on the fly, leading to faster convergence and fewer duplicate predictions.
Permutation-Invariant Rewards
Another approach involves Reinforcement Learning (RL). Pengcheng Yang developed Seq2Set, which uses a permutation-invariant reward function. Instead of checking if the model predicted "Apple, Banana," the RL agent checks if the set {Apple, Banana} was recovered, regardless of the order. This allows the model to capture complex label correlations without being restricted by a fixed sequence.
Probabilistic Set Modeling
Some researchers attempt to redefine the probability of a set itself. Kechen Qin and the team at Northeastern University proposed set-RNN, which defines the probability of a set as the sum of the probabilities of all possible permutations of its elements. This mathematically frees the model to discover the most "natural" order for a specific input.
Adaptive and Non-Autoregressive Alternatives
If a sequence must be used, can we make the order less arbitrary? Jiangnan Li suggests Dynamic Label-Order Learning (DLOL), where the model learns to predict "difficult" labels first. By front-loading the hard cases, the model avoids "shortcut learning" and improves its recall of rare labels.
Alternatively, some researchers argue for abandoning autoregression entirely for these tasks. Frederick Liu introduced EncT5, which strips the autoregressive decoder from T5 and replaces it with a non-autoregressive head. This removes the sequential dependency altogether, making the model more parameter-efficient and better suited for multi-label classification.
Impact Across Domains
The shift toward set-based generation has shown significant gains across diverse fields:
- Information Extraction: Zinong Yang applied these concepts to Comparative Opinion Quintuple Extraction, outperforming traditional pipeline methods by treating facts as a set.
- Knowledge Graphs: In Sememe Prediction, Xuancheng Ren used a "Label Distributed" seq2seq model to expand semantic knowledge bases more accurately than human amateurs.
- Keyphrase Generation: Jiacheng Ye demonstrated that set-based models are not only more accurate but also over six times faster during inference than sequential ones.
By recognizing that many NLP outputs are unordered sets rather than rigid sequences, this research direction provides a more mathematically sound and robust way to train generative models for complex classification and extraction tasks.