In traditional multi-label classification, we treat labels as a flat, independent set. If an image contains both a "forest" and a "river," a standard model makes two separate decisions: Is there a forest? and Is there a river? While computationally efficient, this "Binary Relevance" approach ignores a fundamental truth of the real world: labels are rarely independent. A "river" is far more likely to appear if a "forest" is already present, and almost certain not to appear if the model has already identified a "desert."
A growing body of research suggests that the most effective way to capture these dependencies is to stop treating multi-label classification as a set-prediction problem and start treating it as a sequence generation task.
The Chain Rule of Labels
The core intuition behind this shift is the probabilistic chain rule. As Jesse Read and colleagues demonstrated in their seminal work on Classifier Chains, we can decompose the joint probability of a label set into a product of conditional probabilities.
Instead of predicting all labels at once, we predict them one by one. Each subsequent prediction is conditioned not only on the input data but also on all previously predicted labels. This transforms the task into a "conversation" where the model uses its own prior decisions as context to refine its future guesses. Biao Liu recently extended this logic in PCLP, using structural causal models to show that this sequential decomposition allows models to learn a "generative prior" of label correlations.
From Binary Chains to Neural Sequences
While early methods like Classifier Chains used a literal chain of binary classifiers, modern research leverages the power of Sequence-to-Sequence (Seq2Seq) architectures. In this paradigm, an encoder (like a CNN or Transformer) processes the input, and an autoregressive decoder generates a linearized string of labels.
Jiang Wang and the team behind CNN-RNN were among the first to show that this allows a model to explicitly filter out semantically incompatible combinations. For instance, if a model predicts "indoor," the recurrent state effectively lowers the probability of "clouds" for the next step.
This mechanism is particularly powerful in specialized domains:
- Medication Recommendation: Rui Wu developed COGNet, which uses a "copy-or-predict" mechanism to decide whether to carry over medications from a patient's history or generate new ones based on current symptoms.
- Pedestrian Attribute Recognition: Jiandong Jin and Xiao Wang introduced SequencePAR, treating attributes like "skirt" and "female" as a sequence to capture the strong semantic correlations that independent classifiers often miss.
- Entity Typing: Qing Liu and Xianpei Han used this sequential logic in their Label Reasoning Network to perform deductive reasoning, where earlier, broader labels (e.g., "Person") serve as premises for more specific subsequent labels (e.g., "Artist").
Solving the Threshold Problem
One of the most persistent "headaches" in multi-label classification is determining the size of the output set. Standard models output a probability for every possible label, requiring researchers to manually tune a threshold (e.g., "only keep labels with >0.5 probability").
Sequential decoding solves this elegantly. By treating the output as a sequence, the model can learn to generate a special "End-of-Sequence" (EOS) token. As Hongyin Zhu notes in the SpecExplorer paper, this allows the model to dynamically determine the correct number of labels for each specific input. The model stops when it is "done," rather than when an arbitrary numerical threshold is met.
Why It Works: Latent Structure
Beyond just capturing correlations, reframing classification as generation forces the model to build a more expressive latent space. Iftitahu Ni’mah and Samaneh Khoshrou argue in KeyGen2Vec that the autoregressive objective forces the encoder to produce embeddings that account for the conditional nature of the labels.
When a model knows it must predict a sequence, its internal representation of the input must store not just "what is there," but "how what is there relates to everything else." This results in embeddings that are more robust to noise and better at generalizing to rare, "long-tail" labels, as the model can rely on the context of common labels to "reason" its way toward rarer ones.