Multimodal Item Categorization Fully Based on Transformer

Lei Chen, Houwei Chou, Yandi Xia, Hirokazu MiyakeACL 2021

TL;DR

This paper introduces an all-Transformer approach for multimodal item categorization, combining text and image data. They demonstrate that a Transformer-based image model can match or exceed ResNet performance while training significantly faster, and that a cross-modal attention fusion mechanism is crucial for achieving state-of-the-art results.

What Problem Does This Paper Address?

In e-commerce, accurately categorizing millions of products is essential for efficient search and discovery. This task, known as Item Categorization (IC), typically relies on product titles (text) and images. While text-based methods are common, product images, especially in domains like fashion, often contain richer and more precise information. Existing multimodal IC research often combines Transformer-based text models (like BERT) with CNN-based image models (like ResNet). This creates a bottleneck, as image model training is much slower than text model training, and it complicates system maintenance. This paper aims to address this by proposing a unified, all-Transformer architecture for both text and image processing, and to systematically compare different fusion strategies for multimodal data.

Methods

The core of the proposed system is the Transformer architecture, applied to both text and image modalities.

  • Text Encoding: A pre-trained Japanese BERT model is used to encode product titles into a sequence of text representations.
  • Image Encoding: A Vision Transformer (ViT) model is employed for image encoding. ViT treats images as sequences of patches, similar to how Transformers process text, allowing for direct application of the Transformer architecture to vision tasks.
  • Multimodal Fusion: The paper explores three fusion strategies:
  • Late Fusion: Decisions from individual text and image models are combined (e.g., by weighted averaging of probabilities).
  • Early Fusion (Shallow): The global representations (e.g., the [CLS] token output) from the text and image encoders are concatenated and fed into a simple MLP.
  • Early Fusion (Cross-Modal Attention): This is the most sophisticated method, where a cross-modal attention layer is used to allow text and image representations to interact. Specifically, it uses text representations as queries and image representations as keys/values (and vice-versa) to compute attention weights, enabling a richer fusion. A gate mechanism is also introduced to control the influence of the visual channel.

Datasets

The experiments were conducted on a dataset of approximately 500,000 products from a large Japanese e-commerce platform. This dataset focuses on three major product categories: Beverages (B), Appliances (A), and Men's Fashion (M). The task was to predict leaf-level product categories using Japanese titles and images.

Key Insights and Ideas

  • Transformer for Everything: The paper champions the idea of a unified Transformer-based architecture for both text and image processing in multimodal tasks, simplifying the system and potentially improving efficiency.
  • ViT's Training Speed Advantage: A significant finding is that ViT models can be trained approximately four times faster than traditional CNN-based models like ResNet, which is a critical factor for deploying systems in industry where training time is a bottleneck.
  • Cross-Modal Attention is Key: The cross-modal attention fusion method consistently outperformed both unimodal models and simpler fusion strategies (late fusion, shallow early fusion), highlighting the importance of allowing modalities to interact deeply.
  • Image-Text Alignment Challenges: The authors suggest that shallow early fusion might perform poorly because product images and titles may not always be perfectly aligned or may even contradict each other (e.g., an image of a wine bottle box showing only the box).

Key Results

  • Performance: The ViT image model achieved performance comparable to or better than the ResNet-based BiT model on two out of three product genres.
  • Speed: ViT training was observed to be about four times faster than BiT training.
  • Fusion Gains: The cross-modal attention fusion method consistently improved performance over both the text-only BERT model and the image-only ViT model across all three genres. For example, on the Men's Fashion genre, the cross-modal fusion achieved a macro-F1 score of 0.815, compared to 0.802 for text-only and 0.733 for image-only.
  • Shallow Fusion Underperformance: The shallow early fusion method resulted in poorer performance than unimodal models in some cases.

Limitations and Future Work

  • Limitations: The paper notes that shallow early fusion can be sensitive to misalignment between image and text. The performance of the fused model was worse than the text model in the appliance genre.
  • Future Work: The authors suggest several directions:
  • Joint Modeling: Exploring models that jointly process text and images within a single Transformer, such as FashionBERT.
  • Self-Training: Utilizing self-training techniques to overcome limitations imposed by the size of labeled image data for the image model.

Summary updated

BibTeX
@inproceedings{chen2021multimodal,
  author = {Lei Chen and Houwei Chou and Yandi Xia and Hirokazu Miyake},
  title = {{Multimodal Item Categorization Fully Based on Transformer}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure