Weighted Tensor Decomposition for Learning Latent Variables with Partial Data

Omer Gottesman, Weiwei Pan, Finale Doshi-Velez

Abstract

Tensor decomposition methods are popular tools for learning latent variables given only lower-order moments of the data. However, the standard assumption is that we have sufficient data to estimate these moments accurately across all dimensions. In real-world scenarios, data is often "patchy," with some features observed much more frequently than others. This paper introduces the Weighted Tensor Power Method (WTPM), which weights moment estimates based on observation frequency. WTPM maintains the computational efficiency of standard methods while significantly improving parameter recovery in the presence of partial data, as demonstrated on synthetic and real-world clinical datasets.

Introduction: The Challenge of Patchy Data

Tensor decomposition is a powerful unsupervised learning tool used to recover hidden structures, such as topics in text or clusters in medical records. These methods typically rely on decomposing high-order moments (e.g., mean, covariance, or third-order moments) of the observed data. A significant hurdle in applying these methods to real-world datasets is the prevalence of missing values.

Visualization of full vs. partial data availability across dimensions.

As shown in the figure above, data is often "patchy." For instance, in clinical settings, common vitals like heart rate are recorded frequently, while specialized lab tests are rare. Standard approaches face a dilemma: discard dimensions with missing data (losing valuable correlations) or treat missing values as zero (introducing significant noise). The goal is to leverage all available data without allowing infrequent, noisy measurements to degrade the recovery of well-observed dimensions.

The Weighted Tensor Power Method (WTPM)

The authors propose the Weighted Tensor Power Method (WTPM) to address this imbalance. The core innovation is assigning a weight $w_d$ to each dimension $d$ based on its observation frequency.

The Weighting Mechanism

To maintain the mathematical properties required for tensor decomposition—specifically, that the tensors remain a sum of rank-1 components—the weight of a tensor element is defined as the product of the weights of its constituent dimensions: $w_{d_1 d_2 d_3} = w_{d_1} w_{d_2} w_{d_3}$.

Implementation via Data Rescaling

A key practical insight is that this weighting is equivalent to rescaling the input data: $x^*_{dn} = x_{dn} w_d$. This allows WTPM to be implemented as a simple pre-processing step, after which any standard tensor decomposition algorithm can be applied. This ensures that WTPM has the same computational complexity as traditional methods.

Optimal Weight Selection

Theoretical analysis of the Gamma-Poisson model suggests that the optimal weight for a dimension is proportional to its observation probability ($w_d \propto p_d$). This heuristic has proven effective even in models where the specific derivation does not strictly apply, such as Gaussian mixtures.

Experimental Results and Insights

The authors evaluated WTPM using synthetic models (Gaussian Mixture and Gamma-Poisson), semi-synthetic text and medical data, and real-world clinical data from the MIMIC-III database.

The Cross-over Effect

A critical finding is the "cross-over" effect between full and partial dimensionality methods.

Comparison of full vs. partial dimensionality methods showing the cross-over effect as partial data volume increases.

As illustrated in the plot, when the amount of partial data ($N_p$) is low, the "Full" method (blue) performs better because any additional information is beneficial. However, as $N_p$ increases, the noise from poorly-estimated rare dimensions becomes a bottleneck for the "Full" method. Conversely, the "Partial" method (green) improves as it focuses only on high-quality data. WTPM effectively bridges this gap, matching or exceeding the performance of the best-performing baseline across all regimes.

Real-World Application: ICU Data

On the MIMIC-III sepsis dataset, WTPM achieved higher log-likelihood on held-out test data compared to the partial dimensionality approach. It demonstrated robustness to violations of the "Missing Completely at Random" (MCAR) assumption, which is common in clinical settings where missingness is often correlated with patient state.

Conclusion

WTPM provides a robust and computationally efficient framework for learning latent variables from incomplete datasets. By intelligently weighting dimensions based on their reliability, it avoids the pitfalls of both discarding data and over-relying on noisy estimates. Future work may explore more sophisticated weight selection strategies that account for the underlying structure of the latent variables and extend the framework to models like Latent Dirichlet Allocation (LDA) and Hidden Markov Models (HMMs).

Summary updated

Abstract

Tensor decomposition methods are popular tools for learning latent variables given only lowerorder moments of the data. However, the standard assumption is that we have sufficient data to estimate ...
BibTeX
@inproceedings{gottesman2018weighted,
  author = {Omer Gottesman and Weiwei Pan and Finale Doshi-Velez},
  title = {{Weighted Tensor Decomposition for Learning Latent Variables with Partial Data}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2018},
  url = {https://openreview.net/forum?id=cY3fyvTN0So},
}

Loading related content...

Figure