Label Attention Network for Temporal Sets Prediction: You Were Looking at a Wrong Self-Attention

Alexey ZaytsevOpenReview 2025 · OpenReview

TL;DR

This paper introduces LANET, a novel Label-Attention Network for predicting future sets of labels based on historical event sequences. LANET excels by first aggregating historical event data into time- and set-aware representations before feeding them into a Transformer encoder, significantly outperforming existing methods by up to 65% in weighted F1 score.

What Problem Does This Paper Address?

Many real-world scenarios involve sequences of events, where each event is characterized by a timestamp and a set of associated labels. Predicting the set of labels for a future event, known as temporal sets prediction, is crucial for applications like e-commerce (predicting items a user will buy next) and finance (predicting transaction types). The core challenge lies in effectively capturing both the temporal dependencies between events and the intricate relationships within label sets. Existing models often struggle to jointly model these two aspects due to limitations in how they represent historical information, leading to suboptimal predictions.

Methods

The proposed solution is the Label-Attention Network (LANET). Its key innovation lies in its input processing stage:

  • Time- and Set-Aware Aggregation: Before entering the main neural network, historical event sequences are transformed. For each label, the model aggregates embeddings of all timestamps where that label appeared in a set. This creates a "time-aware" component. This is then combined with the standard label embeddings to form a joint representation.
  • Transformer Encoder: These aggregated, time- and set-aware representations are then fed into a standard Transformer encoder. The self-attention mechanism within the Transformer allows the model to learn complex interactions between these joint label representations, effectively capturing both temporal and label dependencies.
  • Prediction Layer: Finally, an output layer uses the updated representations from the Transformer to predict a confidence score for each label being part of the next event's label set.

This approach differs from prior work by focusing on a sophisticated input aggregation strategy that explicitly encodes temporal and set structure information before the self-attention mechanism, rather than relying solely on the attention mechanism to discover these relationships.

Datasets

The authors evaluated LANET on four publicly available datasets:

  • Dunnhumby-Carbo (DC): Transactional data from a retail store, where sets are products purchased in a transaction.
  • Mimic III: Medical records from intensive care units, where sets are disease classification codes associated with a hospital admission.
  • Instacart: User product order data, where sets are items purchased in a single order.
  • Synthea: Simulated electronic health record data, similar to Mimic III.

Key Insights and Ideas

  • Input Representation is Key: The paper strongly suggests that how historical data is pre-processed and represented is more critical for temporal sets prediction than the choice of the core architecture (e.g., Transformer). LANET's success stems from its novel aggregation of time and label information.
  • Time vs. Label Importance: Ablation studies indicate that while time information is important, LANET's performance is more heavily influenced by its ability to capture label interdependencies. Even when time information is removed, LANET still performs well, highlighting the strength of its label interaction modeling.
  • Attention for Causal Explanations: The authors explored using LANET's attention weights to interpret label relationships, demonstrating potential for generating causal explanations for predictions, particularly in the Instacart dataset.

Key Results

LANET consistently outperformed four established baseline models (SFCNTSP, DNNTSP, GPTopFreq, TCMBN) across all four datasets and all evaluated metrics.

  • Significant Improvement: LANET achieved top performance, with improvements of up to 65% in weighted F1 score compared to the closest competitor (TCMBN) on the DC dataset.
  • Robustness: The model demonstrated strong performance even when time information was removed, indicating its robust handling of label relationships.
  • Hyperparameter Sensitivity: The study showed that LANET's performance is sensitive to embedding size and the number of attention heads, with an optimal number of encoder layers identified.

Limitations and Future Work

  • Dataset Size and Label Vocabulary: LANET shows consistently strong results on datasets with label vocabularies of 100-200. Its adaptation to recommendation settings with thousands of items or very small vocabularies (5-20) remains an open question.
  • Event Sequence Contraction: The paper acknowledges that reducing event sequences to temporal sets by grouping events within a time period might introduce unexpected findings and warrants further study.
  • Future Directions: The authors suggest exploring LANET's applicability in recommendation systems and its potential for self-supervised learning to generate valuable representations for downstream tasks.

Summary updated

Abstract

Most user-related data can be represented as a sequence of events associated with a timestamp and a collection of categorical labels. For example, the purchased basket of goods and the time of buying fully characterize the event of the store visit. Anticipation of the label set for the future event called the problem of temporal sets prediction, holds significant value, especially in such high-stakes industries as finance and e-commerce. A fundamental challenge of this task is the joint consideration of the temporal nature of events and label relations within sets. The existing models fail to capture complex time and label dependencies due to ineffective representation of historical information initially. We aim to address this shortcoming by presenting the framework with a specific way to aggregate the observed information into time- and set structure-aware views prior to transferring it into main architecture blocks. Our strong emphasis on input arrangement facilitates the subsequent efficient learning of label interactions. The proposed model is called Label-Attention NETwork, or LANET. We conducted experiments on four different datasets and made a comparison with four established models, in- cluding SOTA, in this area. The experimental results suggest that LANET provides significantly better quality than any other model, achieving an improvement up to 65% in terms of weighted F1 metric compared to the closest competitor. Moreover, we contemplate causal relationships between labels in our work, as well as a thorough study of LANET components’ influence on performance. We provide an implementation of LANET to encourage its wider usage.
BibTeX
@misc{zaytsev2025label,
  author = {Alexey Zaytsev},
  title = {{Label Attention Network for Temporal Sets Prediction: You Were Looking at a Wrong Self-Attention}},
  year = {2025},
  url = {https://openreview.net/forum?id=eEdyNbygu1},
  howpublished = {OpenReview},
}
Figure