TAPES: Task-Aligned Joint Predictive Pretraining for Transaction Event Sequences

Ruobing Zuo, Weiyu Han, Huilin He, Haoyu Jiang, Ze Zhao, Jie Sheng, Xiao Tan, Shuai Chen, Tianyi ZhangarXiv 2026 · arXiv

In the world of digital finance and e-commerce, user behavior is captured as a sequence of transaction events. Whether it's predicting if a user will churn, assessing credit risk, or recommending a product, the underlying data is the same. However, the "best" way to represent these sequences for one task might be completely different for another.

A new paper, TAPES: Task-Aligned Joint Predictive Pretraining for Transaction Event Sequences, introduces a framework that moves away from "one-size-fits-all" pretraining. Instead, it automatically selects and jointly models the most relevant prediction targets for a specific downstream task.

TL;DR

TAPES is a two-stage pretraining framework that identifies the most useful predictive signals for a specific downstream task and models their joint dependencies using a shared latent variable. By aligning self-supervised pretraining with task-specific needs, it achieves state-of-the-art performance across diverse financial and e-commerce benchmarks.

What Problem Does This Paper Address?

Most sequence pretraining methods rely on generic objectives, like predicting the next event's raw values. The authors identify two major gaps in this approach:

  1. Semantic Mismatch: A single transaction field (e.g., "Amount") can be viewed through multiple lenses: its absolute value, its change from the previous transaction, or its deviation from the user's historical average. Not all of these "views" are equally helpful for every task. For example, "deviation from mean" might be critical for fraud detection but less relevant for age classification.
  2. Independent Predictions: Standard models often predict multiple targets (like amount and category) independently. This ignores the conditional co-occurrence—the fact that certain combinations of outcomes are more likely given a specific history.
Figure 1: Motivation for TAPES. (a) Different transforma- tions of the same transaction field yield prediction targets with distinct semantics, illustrated by amount level, adjacent change, and historical deviation. (b) Two historical contexts share the same target-wise conditional marginals but exhibit different joint outcome patterns, motivating joint prediction. Event values, target combinations, and distributions are il- lustrative; bin denotes target-specific discretization.

Methods

TAPES (Task-Aligned Joint Predictive Pretraining) operates in two distinct stages:

Stage 1: Task-Aligned Target-Set Selection

The framework defines a "target space" using field-operator pairings. A field (like Amount) is combined with an operator (like "Adjacent Change") to create a specific prediction target. TAPES then runs "mini-pretraining" trials on different candidate sets of these targets. It uses a small portion of the downstream labels to evaluate which target set produces the most useful representations via a simple linear probe.

Stage 2: Shared-Latent Joint Predictive Pretraining

Once the best target set is selected, the model undergoes full pretraining. Crucially, it doesn't just sum up independent losses for each target. Instead, it uses a joint likelihood model parameterized by a shared discrete latent variable. This allows the model to capture dependencies between targets—learning, for instance, how a specific category of purchase often coincides with a specific range of price changes.

Figure 2: Overview of Task-Aligned Joint Predictive Pretraining for Transaction Event Sequences (TAPES). (1) Stage 1: Task- guided target-set selection. Field–operator combinations capture predictive semantics beyond raw field values, and short pretraining trials identify a target set useful for the downstream task. (2) Stage 2: Shared-latent joint pretraining. The selected targets are jointly modeled through a shared discrete latent variable to learn representations of transaction histories. Candidate sets and probabilities are illustrative.

Datasets

The authors evaluated TAPES on a comprehensive suite of benchmarks:

  • Public Benchmarks: Five datasets covering credit default (AlfaBattle), age classification (Retail X5), product purchase (MBD-mini), churn prediction (Rosbank Churn), and e-commerce purchase (Taobao).
  • Industrial Dataset: TransRisk, a massive private dataset containing over 29 million transaction-level examples for real-world risk assessment.

Key Insights and Ideas

  • Predictability $\neq$ Utility: The authors demonstrate that just because a model can predict a certain property of the next event well doesn't mean that property helps the downstream task. Alignment is key.
  • The Power of Joint Modeling: By modeling the joint distribution of targets rather than marginals, the encoder learns a richer representation of the historical context.
  • Efficiency in Low-Label Regimes: Pretraining with TAPES is particularly effective when downstream labels are scarce. On the AlfaBattle dataset, TAPES using only 40% of the labels outperformed a model trained from scratch on 100% of the labels.

Key Results

TAPES consistently outperformed representative baselines, including tree ensembles (XGBoost, CatBoost), supervised sequence models (GRU, Transformer), and other self-supervised methods (CoLES, TabBERT).

  • State-of-the-Art Performance: TAPES achieved the best primary score on 4 out of 5 public benchmarks and the best average rank (1.20) across all tasks.
  • Industrial Impact: On the 29M-example TransRisk dataset, TAPES improved AUROC by 1.51 percentage points over the supervised baseline and significantly boosted risk retrieval metrics (Recall@top1%).
  • Robustness: Ablation studies confirmed that both the target selection and the joint modeling components contribute significantly to the final performance gains.
Figure 3: Development performance at different labeled-data fractions for downstream fine-tuning on AlfaBattle, Retail X5, MBD-mini, and Rosbank Churn. Curves show means over fine-tuning seeds; shaded regions indicate ± one sample standard deviation.

Limitations and Future Work

While TAPES is powerful, the authors acknowledge some areas for improvement:

  • Search Efficiency: Evaluating multiple candidate target sets in Stage 1 adds computational overhead. Future work could explore more efficient ways to search this semantic space.
  • Dependency Characterization: While joint modeling helps, more research is needed to understand exactly which types of target dependencies provide the most benefit for specific data conditions.

Overall, TAPES provides a principled way to bridge the gap between self-supervised pretraining and the specific needs of downstream financial and e-commerce applications.

Summary updated

Abstract

Transaction event sequences support diverse prediction tasks in online commerce and digital financial services. The same transaction records can yield prediction targets with different semantics, including values, changes, and relations to past events. These targets impose different constraints on history representations, but their predictability alone does not establish the downstream utility of a target set. Moreover, target-wise prediction losses do not explicitly model conditional co-occurrence among targets. To address these challenges, we propose TAPES, a task-aligned joint predictive pretraining framework for transaction sequences. TAPES constructs a finite collection of candidate target sets from admissible field-operator combinations. It selects a target set according to the downstream utility of representations learned through short joint pretraining trials under matched budgets. It then reinitializes the encoder and pretrains it on the selected targets through a joint likelihood parameterized by a shared discrete latent variable, before downstream supervised adaptation. Experiments in a real-world transaction-risk setting and on multiple public financial benchmarks spanning diverse downstream tasks demonstrate that TAPES outperforms representative baselines. Ablation studies further validate the contributions of predictive target selection and joint modeling.
BibTeX
@misc{zuo2026tapes,
  author = {Ruobing Zuo and Weiyu Han and Huilin He and Haoyu Jiang and Ze Zhao and Jie Sheng and Xiao Tan and Shuai Chen and Tianyi Zhang},
  title = {{TAPES: Task-Aligned Joint Predictive Pretraining for Transaction Event Sequences}},
  year = {2026},
  eprint = {2611.04747},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04747},
}
Figure