TL;DR
This paper introduces "multi-multi-instance learning" (MMIL), a novel framework for learning from data organized in nested bags (e.g., documents as bags of sentences, which are bags of words). They propose a new neural network architecture with "bag-layers" that can effectively model this hierarchical structure and, crucially, enable more interpretable predictions by revealing latent structures at multiple levels.
What Problem Does This Paper Address?
Traditional machine learning often struggles with data that has inherent hierarchical or nested structures. While Multi-Instance Learning (MIL) addresses data organized as bags of instances (e.g., an image as a bag of patches), many real-world scenarios involve deeper nesting. For instance, a document can be seen as a collection of sentences, where each sentence is composed of words. Existing MIL methods often flatten these structures or make simplifying assumptions that limit their expressiveness and interpretability. This paper tackles the challenge of learning from such nested data structures and aims to provide a more transparent understanding of the learned models and their predictions.
Methods
The core contribution is the Multi-Multi-Instance Learning (MMIL) framework and a corresponding neural network architecture.
- MMIL Framework: This extends MIL by allowing examples to be nested bags of instances (bags of bags). It handles arbitrary nesting levels and relaxes the binary label assumption for instances and sub-bags, allowing for categorical labels.
- Bag-Layer: A novel neural network layer designed to aggregate representations of instances or sub-bags within a bag, regardless of their size. This layer is inspired by weight-sharing mechanisms in CNNs and graph neural networks, and is similar to the invariant pooling in DeepSets.
- MMIL Network Architecture: For a two-level bag-of-bags structure, the architecture typically uses two stacked bag-layers: a bottom layer to aggregate instances into sub-bags, and a top layer to aggregate sub-bags into the final top-bag representation. These bag-layers can be combined with standard neural network layers to form deep architectures.
- Interpretability Framework: A key aspect is the ability to interpret MMIL models. This involves learning synthetic cluster identifiers for instance and sub-bag representations, and then training decision trees to approximate the MMIL network's decision function. This allows for extracting symbolic rules that explain both the overall model and individual predictions.
Datasets
The authors evaluate their approach on a variety of datasets:
- Semi-Synthetic MNIST: A dataset created from MNIST digits to illustrate the interpretation of models in a controlled environment.
- IMDB Dataset: A standard benchmark for binary sentiment classification, adapted to the MMIL setting by treating reviews as bags of sentences, and sentences as bags of trigrams.
- Citation Networks: Standard datasets (CiteSeer, Cora, PubMed) used for node classification and graph classification tasks.
- Social Network Datasets: COLLAB, IMDB-BINARY, IMDB-MULTI, REDDIT-BINARY, REDDIT-MULTI5K, REDDIT-MULTI12K for graph classification.
- Point Clouds (ModelNet40): Used to demonstrate the benefits of MMIL for handling data with varying orientations and spatial resolutions.
- Plant Distribution Data: A novel dataset of geo-localized plant species in Germany, used to evaluate prediction methods for identifying false negatives.
Key Insights and Ideas
- Hierarchical Data is Ubiquitous: The paper highlights that many real-world data types (text, images, graphs, biological data) naturally exhibit nested structures, making MMIL a broadly applicable framework.
- Bag-Layers Offer Flexibility: The proposed bag-layer is a powerful building block that can aggregate variable-sized inputs, providing a flexible alternative to fixed-size pooling in traditional CNNs or graph networks.
- Interpretability Beyond MIL: MMIL offers a richer interpretability than standard MIL. By inferring labels at multiple levels (instance, sub-bag, top-bag), it allows for a more granular understanding of how predictions are made.
- Decomposition as an Alternative to Message Passing: The authors propose viewing MMIL networks as a "decomposition schema" rather than a "message passing schema," offering a different perspective on how information is processed in hierarchical models, particularly in relation to graph neural networks.
Key Results
- Competitive Accuracy: MMIL models achieved competitive results across various tasks, including graph classification (outperforming some state-of-the-art models), sentiment analysis, and point cloud classification.
- Effective Interpretation: The proposed interpretability framework successfully recovered latent logic functions and symbolic rules in semi-synthetic experiments, demonstrating its ability to explain model behavior.
- Advantage in Complex Data: MMIL showed advantages over MIL and other methods, particularly on datasets with complex hierarchical structures or varying spatial resolutions (e.g., point clouds). For instance, on the P100 point cloud dataset, MMIL outperformed DeepSets, especially at lower spatial resolutions.
Limitations and Future Work
- Focus on Classification: The current work primarily focuses on classifying whole bags-of-bags. The authors note that extending this to tasks where individual instances or sub-bags need to be classified is less clearly defined due to the lack of explicit assumptions about label spaces and functional relationships between levels.
- Computational Cost: While not explicitly detailed, deep neural networks with multiple bag-layers can be computationally intensive.
- Semantic Interpretation of Clusters: While the framework provides a way to cluster representations, assigning meaningful semantic labels to these clusters still requires domain expertise and is not fully automated.
- Future Work: The authors suggest exploring more complex MMIL problems, potentially involving different aggregation functions, deeper nesting, and further investigation into the relationship between MMIL and graph neural networks. The interpretability framework also offers avenues for further development in generating more intuitive explanations.
Summary updated