Abstract
This paper introduces MAGNeto, a deep learning framework designed for Extractive Tags Summarization (ETS). ETS is the task of identifying a subset of truly representative tags from a noisy and redundant set of image metadata. MAGNeto employs a two-stream architecture that fuses visual and textual information using Transformer encoders, a novel gating mechanism, and a specialized "BCE-Dice" loss function. Evaluated on the NUS-WIDE benchmark, MAGNeto achieves a 94.1% F1 score, effectively addressing challenges such as class imbalance and modality convergence.
Introduction: The Extractive Tags Summarization Task
Digital asset management and image retrieval systems rely heavily on tags. However, real-world metadata is often "noisy," containing redundant, overly broad, or incorrect tags. The authors define the Extractive Tags Summarization (ETS) task: given an image and its original tag set, the model must extract only the most relevant tags. This task is particularly challenging due to significant class imbalance—where most tags are irrelevant—and the need for effective multi-modal fusion between visual and textual data.
Methodology
MAGNeto utilizes a two-stream neural network to process multi-modal inputs and prune irrelevant metadata:
- Tag-Stream: Processes input tags using a Tag Embedder followed by a Transformer encoder to capture semantic relationships between words.
- Image-Stream: Extracts visual features using a ResNet-based CNN. A Multi-Head Attention layer is employed where tags serve as "queries" to attend to specific spatial regions of the image ("keys/values"), ensuring visual features are contextually aligned with the textual metadata.
Key Innovations
- Gating Mechanism: Instead of simple concatenation, a Gating Layer learns a weighting vector ($\alpha$) to dynamically balance the contribution of visual versus textual streams for each tag. This allows the model to prioritize visual evidence when tags are ambiguous or textual context when visual features are unclear.
- BCE-Dice Loss: To combat the sparse nature of important tags, the authors treat ETS as a 1D segmentation problem. They combine standard Binary Cross-Entropy (BCE) with Dice Loss. While BCE handles per-tag classification, Dice Loss optimizes for the global overlap between predicted and ground-truth sets, significantly improving performance on imbalanced data.
- Auxiliary Losses: A common challenge in multi-modal learning is that the tag stream often converges much faster than the image stream. To prevent the model from over-relying on the textual data, auxiliary loss functions are applied to each stream independently during training, forcing both to learn robust representations.
- TAD Augmentation and Pre-training: A technique called Tag Adding & Dropping (TAD) randomly injects or removes tags during training to simulate real-world noise. Furthermore, unsupervised pre-training on an outlier detection task provides a superior initialization point for the final supervised learning stage.
Experimental Results
The model was evaluated on the NUS-WIDE public benchmark (~26,500 images) and a private Large-Scale (LS) dataset (~700,000 images, 24k+ concepts).
On NUS-WIDE, MAGNeto achieved a peak 94.1% F1 score when utilizing auxiliary losses, outperforming baselines that relied on only one modality or lacked the gating mechanism. Ablation studies confirmed that the gating mechanism and the BCE-Dice loss were the primary drivers of performance. On the significantly more complex LS dataset, the model maintained a 50% F1 score, demonstrating its ability to scale to massive concept spaces despite inherent data noise.
Conclusion and Future Work
MAGNeto provides an efficient solution for cleaning image metadata through multi-modal fusion. While highly effective, the authors suggest that future work could focus on more sophisticated mapping between tag and image feature spaces and addressing global concept frequency imbalances, such as the prevalence of common tags like "sky" compared to rarer concepts.
Summary updated