Abstract
Modout is a novel stochastic regularization technique designed to automatically learn optimal fusion strategies for multimodal data within neural networks. By treating modality-specific units as clusters and learning connection probabilities between them, Modout avoids the pitfalls of manual architecture design. Evaluated on the Montalbano gesture recognition dataset, Modout outperforms standard Dropout and ModDrop, achieving performance on par with state-of-the-art hand-tuned models while offering significantly greater flexibility.
Introduction
In multimodal deep learning, integrating diverse data streams—such as RGB video, depth, and audio—is a significant challenge. The standard approach typically involves concatenating features into a single fully-connected layer. However, this "early fusion" often fails to respect the inherent structure of the data. While stochastic regularization methods like Dropout prevent overfitting, they treat all neurons uniformly, ignoring their modality of origin. Previous methods like ModDrop randomly drop entire modalities at the input but do not address how or when to combine signals in deeper layers. Modout addresses this by learning the fusion strategy directly from the data.
The Modout Method
Modout introduces a modality-aware masking mechanism that operates on the connections between modality groups in adjacent layers. Unlike standard Dropout, which drops units, or DropConnect, which drops individual weights, Modout drops entire clusters of connections based on learned probabilities.
Key features of the method include:
- Learned Fusion Probabilities: The probability of connecting Modality A in one layer to Modality B in the next is a parameter learned via backpropagation.
- Guaranteed Self-Propagation: Diagonal elements of the connectivity matrix are fixed to 1, ensuring that a modality's signal always passes to its corresponding units in the next layer.
- Efficiency: The method adds a negligible number of parameters, proportional only to the square of the number of modalities.
As shown in the figures above, Modout can transition from a dense "full fusion" state (Figure 4) to a sparse, optimized "fused" state (Figure 6) where only relevant cross-modality interactions are maintained.
Experimental Evaluation
The researchers evaluated Modout on the Montalbano Gesture Recognition Dataset (13,000+ Italian gestures) and a simulated Multimodal MNIST dataset.
On the Montalbano classification task, Modout combined with standard Dropout achieved 93.8% test accuracy, surpassing standard Dropout (92.5%) and ModDrop (92.4%). For gesture localization, it achieved a Jaccard Index of 0.888, rivaling complex architectures that utilize temporal convolutions and LSTMs.
Key Insights and Discussion
Modout effectively performs an automated architecture search. Experiments revealed that highly correlated modalities (like quadrants of an MNIST digit) favor early fusion, while less correlated modalities (like audio and skeletal pose) benefit from late fusion. Modout successfully discovered these patterns without manual intervention.
Furthermore, the authors found that "binarizing" the learned probabilities after training—setting them to 0 or 1 to create a fixed, pruned architecture—followed by fine-tuning yielded the best overall performance. This suggests Modout is not only a regularizer but also a powerful tool for architecture discovery.
Conclusion and Future Work
Modout provides a flexible, automated way to handle modality fusion in deep networks. While currently limited by its frame-independent treatment of data, future work aims to incorporate temporal correlations (e.g., via Markov Random Fields) and validate the method across a broader range of multimodal tasks and network architectures.
Summary updated