Meta-learning from Heterogeneous Tensors for Few-shot Tensor Completion

Tomoharu Iwata, Atsutoshi KumagaiAISTATS 2025 · OpenReview

Abstract

We propose neural network-based models for tensor completion in few observation settings. The proposed model can meta-learn inductive bias from multiple heterogeneous tensors without shared modes. Although many tensor completion methods have been proposed, the existing methods cannot leverage knowledge across heterogeneous tensors, and their performance is low when only a small number of elements are observed. The proposed model encodes each element of a given tensor by considering information about other elements while reflecting the tensor structure via a self-attention mechanism. The missing values are predicted by tensor-specific linear projection from the encoded vectors. The proposed model is shared across different tensors, and it is meta-learned such that the expected tensor completion performance is improved using multiple tensors. By experiments using synthetic and real-world tensors, we demonstrate that the proposed method achieves better performance than the existing meta-learning and tensor completion methods.

Introduction

Tensors, or multi-dimensional arrays, are fundamental for representing complex data in various domains, including signal processing, chemometrics, and traffic analysis. A recurring challenge in these fields is tensor completion: estimating missing values from a partially observed tensor.

Traditional methods, such as Canonical Polyadic Decomposition (CPD) or Tucker decomposition, are designed for "single-task" scenarios. They rely solely on the internal information of the target tensor. Consequently, their performance degrades sharply in "few-shot" settings—where the missing-value ratio is extremely high (e.g., 99% missing). Furthermore, existing transfer learning approaches for tensors typically require "shared modes," meaning the tensors must share specific dimensions like the same set of users or timestamps. This paper addresses the more difficult problem of learning from heterogeneous tensors that share no common modes or sizes, effectively transferring structural inductive biases across entirely different domains.

Neural Tensor Attention Networks (NTAN)

The core contribution of this work is the Neural Tensor Attention Network (NTAN). Unlike standard sequence-based models, NTAN is designed to respect the multi-dimensional structure of tensor data.

Architecture of the proposed Neural Tensor Attention Network (NTAN).

The architecture follows a three-stage process:

  1. Embedding Layer: Observed values and their corresponding binary masks are projected into a high-dimensional latent space.
  2. Multi-head m-mode Tensor Attention (mMTA): This is the engine of the model. Instead of treating the tensor as a flat vector, mMTA performs attention sequentially along each mode. For a 3rd-order tensor, it applies attention across rows, then columns, then tubes. This allows the model to capture complex dependencies while remaining invariant to the specific size of each dimension.
  3. Adaptive Prediction Layer: To handle the diversity of heterogeneous tensors, the model employs an adaptive final layer. It formulates the prediction as a regularized linear least-squares problem, allowing the model to solve for optimal projection weights in closed form based on the observed elements of the target tensor. This ensures rapid and precise adaptation during inference.

Meta-Learning Framework

NTAN is trained using an episodic meta-learning strategy. During the meta-training phase, the model is exposed to a wide variety of tensors (the "meta-training set"). In each episode, it practices completing sub-tensors sampled from this set. The goal is to optimize the shared parameters of the embedding and attention layers so that the model develops a generalized "inductive bias"—learning, for example, what a low-rank or smooth structure looks like across different data types.

Experimental Evaluation

The researchers evaluated NTAN against a broad suite of baselines, including traditional decomposition methods (CPD, TD, TRD) and recent neural network-based completion models (SEM, FEA).

Performance comparison (RMSE) on real-world datasets across different observation ratios (μ).

As shown in the results table, NTAN consistently outperforms competing methods, particularly in extreme scarcity settings. For instance, on the "Enose" dataset with only 1% of values observed (μ=0.01), NTAN achieves an RMSE of 0.068, whereas traditional CPD fails with an error of 0.909.

Key findings from the experiments include:

  • Cross-Domain Transfer: NTAN successfully transfers knowledge from unrelated domains. A model trained on chemical datasets can effectively complete traffic data tensors, proving it learns structural patterns rather than specific feature identities.
  • Scaling with Data: The model's performance improves as the number of meta-training tensors increases, demonstrating its ability to aggregate knowledge from a heterogeneous pool of data.
  • Flexibility: NTAN adapts its behavior based on the training data. If trained on low-rank synthetic data, it learns to perform low-rank completion; if trained on nonlinear data, it captures those complexities.

Conclusion and Future Work

NTAN represents a significant step forward in few-shot tensor completion by enabling meta-learning across heterogeneous datasets. By leveraging multi-mode attention and closed-form adaptation, it provides a robust and efficient solution for data-scarce environments.

Future research directions include addressing the quadratic computational complexity of the attention mechanism for very large tensors and exploring performance under non-random missingness patterns, where data might be missing in structured blocks due to sensor failures or systematic gaps.

This summary is based on the paper "Meta-learning from Heterogeneous Tensors for Few-shot Tensor Completion" presented at AISTATS 2025.

Summary updated

Abstract

We propose neural network-based models for tensor completion in few observation settings. The proposed model can meta-learn inductive bias from multiple heterogeneous tensors without shared modes. Although many tensor completion methods have been proposed, the existing methods cannot leverage knowledge across heterogeneous tensors, and their performance is low when only a small number of elements are observed. The proposed model encodes each element of a given tensor by considering information about other elements while reflecting the tensor structure via a self-attention mechanism. The missing values are predicted by tensor-specific linear projection from the encoded vectors. The proposed model is shared across different tensors, and it is meta-learned such that the expected tensor completion performance is improved using multiple tensors. By experiments using synthetic and real-world tensors, we demonstrate that the proposed method achieves better performance than the existing meta-learning and tensor completion methods.
BibTeX
@inproceedings{iwata2025meta,
  author = {Tomoharu Iwata and Atsutoshi Kumagai},
  title = {{Meta-learning from Heterogeneous Tensors for Few-shot Tensor Completion}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
  url = {https://openreview.net/forum?id=EddglKlyb0},
}

Loading related content...

Figure