Applications of Online Nonnegative Matrix Factorization to Image and Time-Series Data

Hanbaek Lyu, Georg Menz, Deanna Needell, Christopher Strohmeier

Abstract

Online Nonnegative Matrix Factorization (ONMF) provides a scalable and efficient framework for dictionary learning in streaming data environments. Unlike traditional offline NMF, which requires access to the entire dataset simultaneously, ONMF processes data incrementally, making it suitable for modern high-volume data streams. This paper explores the versatility of ONMF across diverse data modalities, including historical climate time-series and video sequences. We demonstrate that by maintaining historical summary statistics, ONMF can effectively perform tasks such as missing data recovery (in-painting), temporal change detection, and image colorization. Our results highlight the model's ability to capture interpretable physical correlations and adapt to evolving data patterns in real-time.

Introduction

Nonnegative Matrix Factorization (NMF) is a cornerstone technique in dimensionality reduction and topic modeling. By decomposing complex data into a small set of interpretable, additive parts, it allows for the discovery of latent structures within high-dimensional datasets. However, the standard NMF algorithm is an "offline" process, necessitating that the full dataset be stored in memory. In the era of "Big Data," where information from sensors, video feeds, and healthcare systems arrives in continuous streams, this requirement becomes a significant bottleneck.

This paper addresses the need for an efficient, "online" version of NMF that can update its dictionary incrementally as new samples arrive. By processing data in a streaming fashion, Online Nonnegative Matrix Factorization (ONMF) avoids the need to re-process historical data, offering a computationally viable path for real-time analysis and reconstruction.

Methodology

The core objective of the ONMF framework is to approximate a data matrix $X$ as the product of two smaller, non-negative matrices: a dictionary $W$, representing the fundamental features or "atoms," and a code matrix $H$, representing the activation levels of those features for each data point.

Conceptual diagram of Nonnegative Matrix Factorization (NMF) where a data matrix $X$ is approximated by the product of a dictionary $W$ and a code matrix $H$.

To achieve online processing, the algorithm utilizes aggregation matrices ($A_t$ and $B_t$). These matrices serve as summary statistics that encapsulate all relevant historical information. Consequently, the dictionary $W$ can be updated at each time step $t$ using only the current data sample and these summary statistics, ensuring memory efficiency.

Furthermore, the authors employ specialized techniques for different data types:

  • Temporal Dictionary Learning: A sliding window approach is used for time-series data to learn patterns of evolution over specific intervals.
  • Joint Dictionary Learning: Data from multiple correlated sources (e.g., temperatures from different cities) are stacked to learn a shared dictionary that captures inter-dependencies.
  • Patch-based Processing: High-resolution images are decomposed into small, overlapping patches to maintain computational tractability.

Experimental Setup

The approach was evaluated across four distinct datasets:

  1. Historical Temperature Data: Monthly averages for Los Angeles (LA), San Diego (SD), San Francisco (SF), and New York City (NYC) from 1944 to 2020.
  2. Video Data: A 75-frame grayscale video of a moving candle.
  3. Synthetic Video: A composite of the candle video and white noise designed to test phase transition detection.
  4. Color Images: Leonid Afremov’s "Rain’s Rustle" for compression tests and various landscape images for color restoration tasks.

Analysis and Key Results

Data Recovery and In-painting

One of the most significant applications demonstrated is the recovery of missing data. By learning a joint dictionary for multiple cities, the model captures the inherent correlations between their temperature profiles. When data for a specific city is missing (represented as 0°F in the dataset), the ONMF model leverages the observed data from correlated cities and the learned dictionary to "in-paint" the missing values with high accuracy.

Reconstruction of historical temperature data across four cities. The model successfully "in-paints" missing data points (blue drops to zero) using learned correlations (red lines).

Temporal Awareness and Change Detection

In video analysis, ONMF demonstrated temporal awareness. Unlike offline methods that treat all frames as a static collection, ONMF's dictionary atoms evolve. In the candle video experiments, specific atoms "activated" only as the candle moved into corresponding spatial positions. Additionally, the model successfully detected "phase transitions" in synthetic videos, clearly identifying the boundary where structured video ended and white noise began.

Image Compression and Colorization

For static images, the patch-based dictionary approach proved effective for both compression and colorization. The authors successfully reconstructed complex, colorful paintings using a dictionary of only 100 small patches. In colorization tasks, a "landscape dictionary" was used to match textures in grayscale images to learned color atoms, effectively restoring natural-looking hues to images of grass and sand.

Discussion and Conclusion

While ONMF shows great promise, certain limitations remain. The use of small patches can occasionally lead to "blockiness" in image reconstructions, and the current color restoration method relies on a linear assumption that may not always hold for artistic photography. Future work may involve integrating ONMF more closely with Convolutional Neural Networks (CNNs) to improve patch classification and handle more complex non-linear data relationships.

In conclusion, Online Nonnegative Matrix Factorization offers a robust and interpretable solution for real-time data analysis. Its ability to learn from streaming data while maintaining historical context makes it an invaluable tool for modern data science applications ranging from climate monitoring to computer vision.

Summary updated

Abstract

Online nonnegative matrix factorization (ONMF) is a matrix factorization technique in the online setting where data are acquired in a streaming fashion and the matrix factors are updated each time. This enables factor analysis to be performed concurrently with the arrival of new data samples. In this article, we demonstrate how one can use online nonnegative matrix factorization algorithms to learn joint dictionary atoms from an ensemble of correlated data sets. We propose a temporal dictionary learning scheme for time-series data sets, based on ONMF algorithms. We demonstrate our dictionary learning technique in the application contexts of historical temperature data, video frames, and color images.
BibTeX
@inproceedings{lyu2020applications,
  author = {Hanbaek Lyu and Georg Menz and Deanna Needell and Christopher Strohmeier},
  title = {{Applications of Online Nonnegative Matrix Factorization to Image and Time-Series Data}},
  booktitle = {ITA},
  year = {2020},
  url = {https://openreview.net/forum?id=f1nAtffCFES},
}
Figure