Leveraging Order-Free Tag Relations for Context-Aware Recommendation

Junmo Kang, Jeonghwan Kim, Suwon Shin, Sung-Hyon Myaeng

Abstract Tag recommendation is a critical task for content discovery on social media and technical platforms. Traditional approaches treat tags as either independent items for ranking or as sequential tokens for autoregressive generation. However, tag sets are inherently orderless yet highly interdependent. This paper introduces Sequence-Oblivious Generation (SOG), a novel framework that leverages a Transformer encoder to iteratively generate tags. By employing a 1-to-M training scheme and early multi-modal fusion, SOG captures complex tag relations without imposing artificial sequential constraints. Experiments on Instagram and Stack Overflow datasets demonstrate that SOG significantly outperforms state-of-the-art ranking and autoregressive baselines.

Introduction

When users tag content on platforms like Instagram or Stack Overflow, the resulting tag sets exhibit two distinct characteristics: orderlessness and inter-dependency. For instance, the tags #sunset and #beach are functionally identical regardless of their order, yet the presence of one significantly increases the likelihood of the other.

Existing models typically fall into two categories that fail to capture both traits simultaneously. Ranking models treat tags as independent items, ignoring inter-dependencies. Conversely, Autoregressive (AR) models treat tags as a sequence, forcing an arbitrary order that makes predictions overly dependent on the immediately preceding token rather than the entire set.

Comparison of tag relation modeling: Ranking (independent), Autoregressive (sequential), and SOG (order-free inter-dependent).

Methodology: Sequence-Oblivious Generation

The SOG framework addresses these limitations through three primary innovations:

1. Encoder-based Iterative Generation

Instead of a standard Transformer decoder, SOG utilizes a BERT encoder. The process is treated as iterative "pseudo-relevance feedback": the model takes the initial context, appends a [MASK] token, and predicts a tag. This tag is then fed back into the input for the next iteration. Because it uses a bidirectional encoder, the [MASK] token attends to all previously generated tags equally, avoiding the "last token" bottleneck common in sequential models.

The SOG framework uses a BERT encoder for iterative tag generation, incorporating multi-modal context and previously generated tags.

2. 1-to-M Training Scheme

While standard models use a 1-to-1 mapping (predicting the next specific tag in a ground-truth sequence), SOG employs a 1-to-M scheme. It is trained to predict any of the remaining ground-truth tags at any given step. This is implemented via a KL-divergence loss that encourages the model to distribute probability across the entire set of remaining correct tags, reinforcing the orderless nature of the task.

3. Early Multi-modal Fusion

For multi-modal contexts like Instagram, SOG converts images into captions and metadata (time, location) into text strings. These are fed into BERT simultaneously, allowing the self-attention mechanism to perform early fusion. This enables the model to learn complex synergies, such as how a specific image might require different tags depending on the time of day.

Experiments and Results

The authors evaluated SOG on two datasets: a multi-modal Instagram dataset (87k posts) and a text-only Stack Overflow dataset (81k posts). SOG consistently outperformed all baselines:

  • Instagram (P@1): SOG achieved 51.85%, significantly higher than the best ranking (41.78%) and autoregressive (37.34%) models.
  • Stack Overflow (P@1): SOG reached 67.87%, compared to 58.72% for ranking and 62.30% for AR baselines.

Ablation studies highlighted that user-written descriptions (text) remain the strongest signal, though multi-modal fusion provided substantial gains. Notably, the authors found that greedy search outperformed beam search, as the goal is relevance of individual items rather than the likelihood of a specific sequence.

Conclusion

SOG demonstrates that treating tags as a unique, syntax-free class of language requires moving away from traditional sequential generation. By leveraging bidirectional encoders and order-agnostic training, SOG effectively captures the semantic density of tag sets. Future work may explore direct visual processing within the Transformer and applying sequence-oblivious approaches to other set-prediction tasks like recipe ingredient prediction.

Summary updated

Abstract

Tag recommendation relies on either a ranking function for top-$k$ tags or an autoregressive generation method. However, the previous methods neglect one of two seemingly conflicting yet desirable characteristics of a tag set: orderlessness and inter-dependency. While the ranking approach fails to address the inter-dependency among tags when they are ranked, the autoregressive approach fails to take orderlessness into account because it is designed to utilize sequential relations among tokens. We propose a sequence-oblivious generation method for tag recommendation, in which the next tag to be generated is independent of the order of the generated tags and the order of the ground truth tags occurring in training data. Empirical results on two different domains, Instagram and Stack Overflow, show that our method is significantly superior to the previous approaches.
BibTeX
@inproceedings{kang2021leveraging,
  author = {Junmo Kang and Jeonghwan Kim and Suwon Shin and Sung-Hyon Myaeng},
  title = {{Leveraging Order-Free Tag Relations for Context-Aware Recommendation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=LWC49N2QPP},
}
Figure