StreaMulT: Streaming Multimodal Transformer for Heterogeneous and Arbitrary Long Sequential Data

Victor Pellegrain, Myriam Tami, Michel Batteux, Céline HudelotarXiv 2021 · arXiv

Introduction

In the era of Industry 4.0, the real-time monitoring of complex industrial systems—such as jet engines or automated factory lines—demands the integration of diverse data sources. These systems generate high-frequency sensor readings, discrete textual maintenance logs, and visual data like thermal imaging. Processing this data presents three significant challenges for machine learning: heterogeneity of data types, temporal unalignment between related events, and the potentially infinite length of data streams.

Traditional Transformer architectures, while effective at capturing long-range dependencies, are limited by their quadratic computational and memory complexity ($O(n^2)$). This makes them unsuitable for continuous streaming data. StreaMulT (Streaming Multimodal Transformer) is introduced as a novel architecture designed to overcome these bottlenecks, enabling the processing of unaligned and arbitrarily long multimodal streams in real-time.

The StreaMulT Architecture

The StreaMulT framework is built upon three core pillars that allow it to maintain constant computational overhead while preserving historical context.

The StreaMulT architecture utilizes block processing and cross-modal transformers to fuse heterogeneous data streams.

1. Block Processing and Segmentation

To handle infinite data streams, StreaMulT partitions input modalities into non-overlapping segments. By processing data in discrete blocks, the model ensures that inference complexity remains $O(1)$ per segment. This approach prevents the memory explosion typically associated with long-sequence Transformers, allowing the system to run indefinitely.

2. Streaming Crossmodal Transformer (SCT)

The SCT serves as the primary fusion engine. Unlike standard fusion techniques that require perfectly aligned data, the SCT uses cross-modal attention to allow one modality (e.g., video) to "attend" to another (e.g., text). This mechanism is essential for identifying correlations across different time scales, such as linking a current sensor glitch to a maintenance event mentioned weeks prior.

3. Augmented Memory Bank and Training

To ensure continuity between segments, StreaMulT utilizes an augmented memory bank inspired by the Emformer architecture. This bank stores a compressed summary of previous segments, allowing the current block to look back at historical context without re-processing it. During training, a flexible scheme parallelizes operations across subsequences of segments, enabling the model to learn long-range dependencies efficiently without exceeding memory limits.

Experimental Evaluation

The researchers evaluated StreaMulT on the CMU-MOSEI dataset, the gold standard for Multimodal Sentiment Analysis. The dataset includes visual (facial expressions), acoustic (voice features), and textual (transcripts) modalities. The model was tested on both "aligned" versions, where words match specific video frames, and "unaligned" raw streams.

To simulate the "arbitrarily long" data characteristic of industrial environments, the authors concatenated clips from the same speaker. While vanilla Multimodal Transformers (MulT) failed to process these extended sequences due to memory exhaustion, StreaMulT handled them with ease, maintaining stable predictive accuracy throughout.

Key Results

StreaMulT-BART established a new state-of-the-art (SOTA) on the CMU-MOSEI unaligned benchmark, achieving a binary accuracy and F1-score of 86.95%. The results demonstrate that StreaMulT performs as well as, or better than, models designed for static data, proving that streaming functionality does not require a sacrifice in peak performance.

Key Insights: Visualizing Memory and Embeddings

A critical finding of the study was the significant impact of input feature quality. By replacing traditional GloVe embeddings with modern contextual embeddings like BERT or BART, the model's performance improved dramatically. This suggests that the quality of individual unimodal feature extractors is as vital as the fusion mechanism itself.

Attention heatmap demonstrating the model's ability to retrieve historical textual cues from the memory bank to inform current predictions.

The efficacy of the memory bank is further validated through attention weight visualizations. As shown in the heatmap, the model actively retrieves historical textual cues from the memory bank to inform its current predictions. This "reach back" capability is what allows the model to maintain context across long temporal gaps.

Limitations and Future Work

While StreaMulT shows great promise, the authors identify several areas for future development. Currently, the model has been primarily validated on sentiment analysis data; applying it to true industrial datasets is a necessary next step to confirm its utility in factory settings. Additionally, the current model uses fixed temporal bounds for segments. Future iterations may incorporate "learned" segmentation, where the model dynamically determines the optimal boundaries for data chunks. Finally, scaling the architecture to handle dozens of simultaneous sensor streams remains an open research question.

Conclusion

StreaMulT represents a significant step forward in multimodal machine learning, providing a scalable solution for processing infinite, heterogeneous data streams. By combining block processing with a persistent memory bank, it offers the long-term memory required for complex system monitoring while maintaining the real-time performance necessary for Industry 4.0 applications.

Summary updated

BibTeX
@misc{pellegrain2021streamult,
  author = {Victor Pellegrain and Myriam Tami and Michel Batteux and Céline Hudelot},
  title = {{StreaMulT: Streaming Multimodal Transformer for Heterogeneous and Arbitrary Long Sequential Data}},
  year = {2021},
  eprint = {2110.08021},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.08021},
}
Figure