Abstract Tag recommendation is a critical task for content discovery on social media and technical platforms. Traditional approaches treat tags as either independent items for ranking or as sequential tokens for autoregressive generation. However, tag sets are inherently orderless yet highly interdependent. This paper introduces Sequence-Oblivious Generation (SOG), a novel framework that leverages a Transformer encoder to iteratively generate tags. By employing a 1-to-M training scheme and early multi-modal fusion, SOG captures complex tag relations without imposing artificial sequential constraints. Experiments on Instagram and Stack Overflow datasets demonstrate that SOG significantly outperforms state-of-the-art ranking and autoregressive baselines.
Introduction
When users tag content on platforms like Instagram or Stack Overflow, the resulting tag sets exhibit two distinct characteristics: orderlessness and inter-dependency. For instance, the tags #sunset and #beach are functionally identical regardless of their order, yet the presence of one significantly increases the likelihood of the other.
Existing models typically fall into two categories that fail to capture both traits simultaneously. Ranking models treat tags as independent items, ignoring inter-dependencies. Conversely, Autoregressive (AR) models treat tags as a sequence, forcing an arbitrary order that makes predictions overly dependent on the immediately preceding token rather than the entire set.
Methodology: Sequence-Oblivious Generation
The SOG framework addresses these limitations through three primary innovations:
1. Encoder-based Iterative Generation
Instead of a standard Transformer decoder, SOG utilizes a BERT encoder. The process is treated as iterative "pseudo-relevance feedback": the model takes the initial context, appends a [MASK] token, and predicts a tag. This tag is then fed back into the input for the next iteration. Because it uses a bidirectional encoder, the [MASK] token attends to all previously generated tags equally, avoiding the "last token" bottleneck common in sequential models.
2. 1-to-M Training Scheme
While standard models use a 1-to-1 mapping (predicting the next specific tag in a ground-truth sequence), SOG employs a 1-to-M scheme. It is trained to predict any of the remaining ground-truth tags at any given step. This is implemented via a KL-divergence loss that encourages the model to distribute probability across the entire set of remaining correct tags, reinforcing the orderless nature of the task.
3. Early Multi-modal Fusion
For multi-modal contexts like Instagram, SOG converts images into captions and metadata (time, location) into text strings. These are fed into BERT simultaneously, allowing the self-attention mechanism to perform early fusion. This enables the model to learn complex synergies, such as how a specific image might require different tags depending on the time of day.
Experiments and Results
The authors evaluated SOG on two datasets: a multi-modal Instagram dataset (87k posts) and a text-only Stack Overflow dataset (81k posts). SOG consistently outperformed all baselines:
- Instagram (P@1): SOG achieved 51.85%, significantly higher than the best ranking (41.78%) and autoregressive (37.34%) models.
- Stack Overflow (P@1): SOG reached 67.87%, compared to 58.72% for ranking and 62.30% for AR baselines.
Ablation studies highlighted that user-written descriptions (text) remain the strongest signal, though multi-modal fusion provided substantial gains. Notably, the authors found that greedy search outperformed beam search, as the goal is relevance of individual items rather than the likelihood of a specific sequence.
Conclusion
SOG demonstrates that treating tags as a unique, syntax-free class of language requires moving away from traditional sequential generation. By leveraging bidirectional encoders and order-agnostic training, SOG effectively captures the semantic density of tag sets. Future work may explore direct visual processing within the Transformer and applying sequence-oblivious approaches to other set-prediction tasks like recipe ingredient prediction.
Summary updated