MAGNeto: An Efficient Deep Learning Method for the Extractive Tags Summarization Problem

Hieu Trong Phung, Anh Tuan Vu, Tung Dinh Nguyen, Lam Thanh Do, Giang Nam Ngo, Trung Thanh Tran, Ngoc C. LêarXiv 2020 · arXiv

Abstract

This paper introduces MAGNeto, a deep learning framework designed for Extractive Tags Summarization (ETS). ETS is the task of identifying a subset of truly representative tags from a noisy and redundant set of image metadata. MAGNeto employs a two-stream architecture that fuses visual and textual information using Transformer encoders, a novel gating mechanism, and a specialized "BCE-Dice" loss function. Evaluated on the NUS-WIDE benchmark, MAGNeto achieves a 94.1% F1 score, effectively addressing challenges such as class imbalance and modality convergence.

Introduction: The Extractive Tags Summarization Task

Digital asset management and image retrieval systems rely heavily on tags. However, real-world metadata is often "noisy," containing redundant, overly broad, or incorrect tags. The authors define the Extractive Tags Summarization (ETS) task: given an image and its original tag set, the model must extract only the most relevant tags. This task is particularly challenging due to significant class imbalance—where most tags are irrelevant—and the need for effective multi-modal fusion between visual and textual data.

Methodology

MAGNeto utilizes a two-stream neural network to process multi-modal inputs and prune irrelevant metadata:

  1. Tag-Stream: Processes input tags using a Tag Embedder followed by a Transformer encoder to capture semantic relationships between words.
  2. Image-Stream: Extracts visual features using a ResNet-based CNN. A Multi-Head Attention layer is employed where tags serve as "queries" to attend to specific spatial regions of the image ("keys/values"), ensuring visual features are contextually aligned with the textual metadata.
The detailed architecture of MAGNeto, showing the two-stream fusion process and gating mechanism.

Key Innovations

  • Gating Mechanism: Instead of simple concatenation, a Gating Layer learns a weighting vector ($\alpha$) to dynamically balance the contribution of visual versus textual streams for each tag. This allows the model to prioritize visual evidence when tags are ambiguous or textual context when visual features are unclear.
  • BCE-Dice Loss: To combat the sparse nature of important tags, the authors treat ETS as a 1D segmentation problem. They combine standard Binary Cross-Entropy (BCE) with Dice Loss. While BCE handles per-tag classification, Dice Loss optimizes for the global overlap between predicted and ground-truth sets, significantly improving performance on imbalanced data.
  • Auxiliary Losses: A common challenge in multi-modal learning is that the tag stream often converges much faster than the image stream. To prevent the model from over-relying on the textual data, auxiliary loss functions are applied to each stream independently during training, forcing both to learn robust representations.
  • TAD Augmentation and Pre-training: A technique called Tag Adding & Dropping (TAD) randomly injects or removes tags during training to simulate real-world noise. Furthermore, unsupervised pre-training on an outlier detection task provides a superior initialization point for the final supervised learning stage.

Experimental Results

The model was evaluated on the NUS-WIDE public benchmark (~26,500 images) and a private Large-Scale (LS) dataset (~700,000 images, 24k+ concepts).

Performance comparison (F1 score) of MAGNeto against various baseline models over 500 training epochs.

On NUS-WIDE, MAGNeto achieved a peak 94.1% F1 score when utilizing auxiliary losses, outperforming baselines that relied on only one modality or lacked the gating mechanism. Ablation studies confirmed that the gating mechanism and the BCE-Dice loss were the primary drivers of performance. On the significantly more complex LS dataset, the model maintained a 50% F1 score, demonstrating its ability to scale to massive concept spaces despite inherent data noise.

Conclusion and Future Work

MAGNeto provides an efficient solution for cleaning image metadata through multi-modal fusion. While highly effective, the authors suggest that future work could focus on more sophisticated mapping between tag and image feature spaces and addressing global concept frequency imbalances, such as the prevalence of common tags like "sky" compared to rarer concepts.

Summary updated

Abstract

In this work, we study a new image annotation task named Extractive Tags Summarization (ETS). The goal is to extract important tags from the context lying in an image and its corresponding tags. We adjust some state-of-the-art deep learning models to utilize both visual and textual information. Our proposed solution consists of different widely used blocks like convolutional and self-attention layers, together with a novel idea of combining auxiliary loss functions and the gating mechanism to glue and elevate these fundamental components and form a unified architecture. Besides, we introduce a loss function that aims to reduce the imbalance of the training data and a simple but effective data augmentation technique dedicated to alleviates the effect of outliers on the final results. Last but not least, we explore an unsupervised pre-training strategy to further boost the performance of the model by making use of the abundant amount of available unlabeled data. Our model shows the good results as 90% $F_\text{1}$ score on the public NUS-WIDE benchmark, and 50% $F_\text{1}$ score on a noisy large-scale real-world private dataset. Source code for reproducing the experiments is publicly available at: https://github.com/pixta-dev/labteam
BibTeX
@misc{phung2020magneto,
  author = {Hieu Trong Phung and Anh Tuan Vu and Tung Dinh Nguyen and Lam Thanh Do and Giang Nam Ngo and Trung Thanh Tran and Ngoc C. Lê},
  title = {{MAGNeto: An Efficient Deep Learning Method for the Extractive Tags Summarization Problem}},
  year = {2020},
  eprint = {2011.04349},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.04349},
}
Figure