XicorAttention: Time Series Transformer Using Attention with Nonlinear Correlation

Daichi Kimura, Tomonori Izumitani, Hisashi KashimaarXiv 2025 · DOI · arXiv

Abstract

Standard Transformer models utilize dot-product attention, which effectively measures linear correlation between queries and keys. However, real-world time series data often exhibit complex nonlinear dependencies that linear measures fail to capture. We present XicorAttention, a novel attention mechanism that replaces the dot-product with Chatterjee’s rank correlation coefficient ($\xi$). By leveraging differentiable sorting and ranking techniques such as SoftSort and SoftRank, XicorAttention remains end-to-end trainable. Experimental results across seven multivariate time series benchmarks demonstrate that XicorAttention significantly improves forecasting accuracy, achieving up to a 9.1% reduction in Mean Squared Error on datasets with high nonlinearity.

Introduction

In the domain of time series forecasting, Transformers have established themselves as a dominant architecture. Most research efforts have focused on improving computational efficiency through sparse attention or enhancing data representation via patching. However, the fundamental mathematical operation of the attention mechanism—the scaled dot-product—has remained largely unchanged.

The scaled dot-product is essentially a measure of linear relationship, akin to Pearson correlation. This is a significant limitation for real-world data, such as power grid fluctuations or financial market movements, which are characterized by oscillations, abrupt shifts, and complex functional dependencies. If two variables share a strong but nonlinear relationship, standard attention may fail to recognize the connection, leading to suboptimal forecasting performance.

The XicorAttention Mechanism

To address the limitations of linear attention, we introduce XicorAttention. This mechanism swaps the traditional dot-product for Chatterjee’s rank correlation coefficient ($\xi$). Unlike Pearson’s $r$ or Spearman’s $\rho$, Chatterjee’s $\xi$ is designed to detect any functional dependency. If a variable $Y$ is a function of $X$, $\xi$ will approach 1, regardless of the complexity or nonlinearity of that function.

Overcoming Non-Differentiability

The primary challenge in implementing rank-based correlation in deep learning is that sorting and ranking are discrete operations with zero gradients. To maintain end-to-end trainability, XicorAttention employs two continuous approximations:

  1. SoftSort: Utilizes a continuous relaxation to create a "soft" permutation matrix, allowing gradients to flow through the sorting process.
  2. FastSoftRank: Projects data onto a permutahedron to provide a differentiable approximation of ranks.
Architecture of the XicorAttention mechanism, highlighting the integration of SoftSort and SoftRank to compute Chatterjee’s rank correlation coefficient.

As illustrated in the architecture diagram, the Query ($Q$) and Key ($K$) vectors are transformed via linear layers. The SoftSort module generates a permutation matrix based on the query, which is then used to compute the sorted keys and subsequently the rank-based coefficient $\xi_n(q, k)$. A "straight-through estimator" is used to ensure exact ranks are used during the forward pass while maintaining soft gradients for backpropagation.

Experimental Evaluation

XicorAttention was integrated into three state-of-the-art models—PatchTST, iTransformer, and TimeXer—and evaluated across seven popular multivariate time series benchmarks, including ETT (Electricity Transformer Temperature), Exchange (currency rates), Weather, Electricity, and Traffic.

Key Results

  • Forecasting Accuracy: XicorAttention consistently outperformed standard attention across most datasets. Notably, on the Exchange dataset, it improved Mean Squared Error (MSE) by approximately 9.1% compared to the original PatchTST.
  • Nonlinear Robustness: The model proved highly effective at capturing oscillatory patterns and was inherently robust to outliers, as it does not rely on assumptions about data distribution.
  • Architecture Synergy: The most significant gains were observed when XicorAttention was paired with PatchTST, suggesting that capturing nonlinear dependencies within a single variable's history is particularly impactful for forecasting.

Discussion and Limitations

While XicorAttention offers superior accuracy for complex signals, it introduces two primary trade-offs. First, the computational cost of sorting ($O(n \log n)$ or $O(n^2)$ approximations) is higher than simple matrix multiplication, leading to increased memory usage and slower training times. Second, Chatterjee’s $\xi$ can have lower statistical power than linear measures when the underlying signal is perfectly linear or very smooth, as seen in some Traffic and Weather benchmarks where performance was comparable to or slightly below the baseline.

Conclusion

XicorAttention represents a fundamental shift in how Transformers process time series data by moving beyond linear correlation. By making rank-based correlation differentiable, we enable models to capture the "nonlinear pulse" of real-world data. Future work will focus on optimizing the computational efficiency of these sorting algorithms and refining the $\xi$ coefficient to better handle smooth signals.

Summary updated

BibTeX
@misc{kimura2025xicorattention,
  author = {Daichi Kimura and Tomonori Izumitani and Hisashi Kashima},
  title = {{XicorAttention: Time Series Transformer Using Attention with Nonlinear Correlation}},
  year = {2025},
  eprint = {2506.02694},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.02694},
}
Figure