Weighted Tensor Decomposition for Learning Latent Variables with Partial Data

Omer Gottesman, Weiwei Pan, Finale Doshi‐Velez

Abstract

Tensor decomposition methods are popular tools for learning latent variables given only lower-order moments of the data. However, the standard assumption is that we have sufficient data to estimate these moments to high accuracy. In this work, we consider the case in which certain dimensions of the data are not always observed—common in applied settings, where not all measurements may be taken for all observations—resulting in moment estimates of varying quality. We derive a weighted tensor decomposition approach that is computationally as efficient as the non-weighted approach, and demonstrate that it outperforms methods that do not appropriately leverage these less-observed dimensions.

Introduction

Uncovering hidden patterns or "latent variables" in complex datasets is a core task in machine learning. Whether identifying underlying health states in clinical records or primary topics in document corpora, researchers often turn to tensor decomposition. This method analyzes statistical moments (e.g., variance, skewness) to recover model parameters. However, a significant gap exists between theory and practice: standard algorithms assume "complete" datasets where every feature is measured for every sample.

In real-world applications, such as medicine or biology, data is frequently incomplete. For instance, while every ICU patient has their heart rate monitored, only a subset may undergo specialized genetic testing. This leads to a dilemma: should one use only the "full" samples, discarding potentially valuable rare measurements, or include all data at the risk of introducing significant noise?

The Challenge of Partial Data

The researchers identify two common but flawed strategies for handling missingness:

  1. The Full Dimensionality Method: Including all data, even rare measurements. This often introduces excessive noise, degrading the quality of estimates for even the most common features.
  2. The Partial Dimensionality Method: Discarding rare measurements entirely. This ignores valuable correlations that could otherwise enhance the model's accuracy.
Data matrix visualization showing (a) general missingness and (b) a split between full ($N_f$) and partial ($N_p$) samples.

As illustrated above, the data matrix often consists of a set of samples where all $D$ dimensions are observed ($N_f$) and a larger set where only a subset of dimensions are observed ($N_p$).

Methods: Weighted Tensor Power Method (WTPM)

To address this, the authors propose the Weighted Tensor Power Method (WTPM). The core innovation is a weighting scheme that adjusts the contribution of each data dimension based on its observation frequency.

The Rescaling "Aha!" Moment

A key theoretical contribution of this work is the realization that weighting high-dimensional tensors is mathematically equivalent to a simple pre-processing step: rescaling the raw input data. By multiplying each dimension $d$ by a weight $w_d$, standard, highly optimized tensor decomposition algorithms can be used without modification.

Optimal Weighting Strategy

Through an analysis of reconstruction error, the authors derived an intuitive optimal weighting rule: the weight for a dimension should be proportional to the probability of it being observed ($w_d \propto p_d$). The workflow is straightforward:

  1. Scale the input data by the weights.
  2. Execute standard tensor decomposition.
  3. Scale the resulting parameters back to their original units.

This approach provides a "computational free lunch," as it adds zero overhead to the execution time while significantly improving robustness.

Experimental Validation

WTPM was tested across three distinct scenarios:

  • Synthetic Data: Gaussian Mixture Models (GMM) and Gamma-Poisson (GP) models were used to simulate various missingness patterns.
  • Semi-synthetic Data: Real-world structures from the 20-Newsgroups and Autism Comorbidities datasets were used to simulate missing data and test topic recovery.
  • Real-world Data: The MIMIC-III Sepsis ICU dataset, featuring 47 clinical dimensions for over 18,000 patients, served as a benchmark for performance on messy, real-world clinical records.

Key Results and Insights

The experiments revealed several critical insights into the behavior of latent variable learning with partial data.

Error Reduction and Dimensionality

The method consistently outperformed naive baselines. In high-dimensional settings where some features were rare, WTPM significantly reduced reconstruction error compared to the "Full" method, which was typically overwhelmed by noise.

Performance plots showing (a) the effect of dimensionality on error and (b) the impact of increasing partial samples.

As shown in the figure above, increasing the number of samples $N$ generally reduces error, but the benefit of including partial data depends on the ratio of signal to noise. WTPM effectively navigates this trade-off, interpolating between the extremes of ignoring partial data and being overwhelmed by it.

Robustness and Real-world Impact

On the MIMIC-III sepsis data, WTPM achieved higher log-likelihood on held-out test data than the partial dimensionality method. This demonstrates its ability to extract meaningful signals from incomplete clinical records. Furthermore, the method proved robust even when missingness was correlated (e.g., entire panels of tests being skipped together), despite the theoretical derivation assuming data is missing at random.

Conclusion and Future Work

WTPM offers a simple, mathematically rigorous, and computationally efficient solution for learning latent variables from partial data. While highly effective for complete dimensions, the authors note that reconstructing parameters for very rare dimensions remains sensitive to noise. Future work may involve refining weights to account for the specific structure of latent topics or extending the method to other models like Hidden Markov Models (HMMs).

Summary updated

Abstract

Tensor decomposition methods are popular tools for learning latent variables given only lower-order moments of the data. However, the standard assumption is that we have sufficient data to estimate these moments to high accuracy. In this work, we consider the case in which certain dimensions of the data are not always observed---common in applied settings, where not all measurements may be taken for all observations---resulting in moment estimates of varying quality. We derive a weighted tensor decomposition approach that is computationally as efficient as the non-weighted approach, and demonstrate that it outperforms methods that do not appropriately leverage these less-observed dimensions.
BibTeX
@inproceedings{gottesman2018weighted,
  author = {Omer Gottesman and Weiwei Pan and Finale Doshi‐Velez},
  title = {{Weighted Tensor Decomposition for Learning Latent Variables with Partial Data}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2018},
  url = {https://openreview.net/forum?id=cY3fyvTN0So},
}

Loading related content...

Figure