MARS: Matching Attribute-aware Representations for Text-based Sequential Recommendation

Hyunsoo Kim, Junyoung Kim, Minjin Choi, Sunkyung Lee, Jongwuk LeearXiv 2024 · arXiv

TL;DR

MARS (Matching Attribute-aware Representations for Sequential Recommendation) introduces a fine-grained approach to text-based recommender systems by encoding items and user histories into multiple attribute-specific vectors (e.g., title, brand, category). By matching user preferences and item features attribute-by-attribute using a late-interaction mechanism, it achieves up to a 29% performance boost over state-of-the-art baselines in both fine-tuned and zero-shot settings.

What Problem Does This Paper Address?

Sequential recommendation systems try to predict a user's next choice based on their past interactions. While traditional models rely on abstract item IDs, recent trends favor text-based models powered by Pre-trained Language Models (PLMs). Text-based models are excellent because they leverage real-world knowledge and can handle "cold-start" items (new items with no interaction history).

However, existing text-based models suffer from an information bottleneck: they compress an entire item description or a user's entire purchase history into a single, coarse-grained vector. This single-vector approach fails to capture nuanced, multi-faceted user interests.

Figure 1: A motivating example of two user sequences. The green and pink boxes represent different user interests that are highly relevant to target items.

As illustrated in Figure 1, two users might have highly overlapping purchase histories, but their underlying motivations differ. User A cares about matching a specific product type (Keyboards), while User B is loyal to a specific brand (Logitech). Single-vector models blur these details together, whereas MARS aims to preserve and match these fine-grained, attribute-level preferences.

Methods

MARS tackles this bottleneck with two core components: Attribute-aware Text Encoding and Attribute-wise Interaction Matching.

Figure 2: Model architecture of MARS. Each item consists of three attributes, e.g., title, brand, and category. For simplicity, we only display attribute values, omitting their keys.
  1. Attribute-aware Text Encoding: Instead of relying on a single token (like [BOS]) to represent an item, MARS structures item data into key-value text pairs (e.g., "Brand: Apple", "Category: Keyboard"). The text is passed through a PLM (Longformer-base). MARS then applies average pooling over the specific hidden states corresponding to each attribute's tokens. This yields a set of distinct vectors for each item—one for its title, one for its brand, and one for its category. The user sequence encoder mirrors this process, generating attribute vectors across the user's timeline.
  2. Attribute-wise Interaction Matching: To score a candidate item against a user's history, MARS uses a late-interaction mechanism called Maximum Similarity (MaxSim), inspired by dense retrieval models like ColBERT. For a given attribute (e.g., Brand), it computes the maximum cosine similarity between the candidate item's brand vector and all brand vectors in the user's history. The final recommendation score is simply the sum of these maximum similarities across all attributes.

To keep training efficient, the authors employ a two-stage training strategy: they first update and freeze the item representations before fully optimizing the sequence matching parameters.

Datasets

The model was evaluated using five distinct sub-datasets from the standard Amazon review dataset (v2):

  • Scientific (Industrial and Scientific)
  • Pantry (Prime Pantry)
  • Inst. (Musical Instruments)
  • Arts (Arts, Crafts & Sewing)
  • Office (Office Products)

The datasets were filtered using a standard "5-core" approach (retaining users and items with at least five interactions). For zero-shot evaluation, the model was pre-trained on a separate combination of 8 Amazon categories (such as Automotive, Electronics, and Home & Kitchen) and evaluated on the five target datasets without any fine-tuning.

Key Insights and Ideas

  • De-bottlenecking the Embedding Space: Retaining separate vectors for different item facets prevents distinct semantic concepts (like brand identity vs. product category) from diluting each other during vector compression.
  • The Power of Late Interaction in RecSys: Borrowing the MaxSim operator from information retrieval allows the recommender to dynamically align a candidate item with the exact moment in a user's history where they exhibited a matching preference.
  • Superior Zero-Shot Transfer: Because MARS maps fine-grained text semantics rather than rigid item IDs, its learned matching patterns transfer remarkably well to entirely unseen product domains.

Key Results

MARS consistently achieved state-of-the-art performance across all benchmarks, outperforming ID-based, hybrid, and text-based baselines.

  • Fine-Tuned Performance: MARS achieved substantial improvements over the previous best text-based model (Recformer), showing a 24.43% improvement in Recall@10 and a 29.26% improvement in NDCG@10 on the Office dataset.
  • Zero-Shot Performance: In zero-shot testing, MARS outperformed all text-based competitors with an average performance gain of 13.11%, demonstrating robust domain transfer.
  • Ablation Validation: Replacing the attribute-aware vectors with a standard single [BOS] vector or replacing the MaxSim operator with a simple mean pooling strategy caused noticeable performance drops, validating both core architectural choices.

Limitations and Future Work

The authors point out that MARS focuses strictly on the attributes of individual items. It does not currently incorporate macro-level relational side-information, such as the exact time intervals between purchases or co-purchase network graphs. Integrating these structural, sequence-level relationships alongside fine-grained textual attributes represents a promising direction for future research.

Summary updated

Abstract

Sequential recommendation aims to predict the next item a user is likely to prefer based on their sequential interaction history. Recently, text-based sequential recommendation has emerged as a promising paradigm that uses pre-trained language models to exploit textual item features to enhance performance and facilitate knowledge transfer to unseen datasets. However, existing text-based recommender models still struggle with two key challenges: (i) representing users and items with multiple attributes, and (ii) matching items with complex user interests. To address these challenges, we propose a novel model, Matching Attribute-aware Representations for Text-based Sequential Recommendation (MARS). MARS extracts detailed user and item representations through attribute-aware text encoding, capturing diverse user intents with multiple attribute-aware representations. It then computes user-item scores via attribute-wise interaction matching, effectively capturing attribute-level user preferences. Our extensive experiments demonstrate that MARS significantly outperforms existing sequential models, achieving improvements of up to 24.43% and 29.26% in Recall@10 and NDCG@10 across five benchmark datasets. Code is available at https://github.com/junieberry/MARS
BibTeX
@misc{kim2024mars,
  author = {Hyunsoo Kim and Junyoung Kim and Minjin Choi and Sunkyung Lee and Jongwuk Lee},
  title = {{MARS: Matching Attribute-aware Representations for Text-based Sequential Recommendation}},
  year = {2024},
  eprint = {2409.00702},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.00702},
}

Loading related content...

Figure