An Extensible Multimodal Multi-task Object Dataset with Materials

Trevor Standley, Ruohan Gao, Dawn Chen, Jiajun Wu, Silvio Savarese

Abstract: We present EMMa, an Extensible, Multimodal dataset of Amazon product listings that contains rich Material annotations. It contains more than 2.8 million objects, each with image(s), listing text, mass, price, product ratings, and position in Amazon's product-category taxonomy. We also design a comprehensive taxonomy of 182 physical materials (e.g., Plastic $\rightarrow$ Thermoplastic $\rightarrow$ Acrylic). Objects are annotated with one or more materials from this taxonomy. With the numerous attributes available for each object, we develop a Smart Labeling framework to quickly add new binary labels to all objects with very little manual labeling effort, making the dataset extensible. Each object attribute in our dataset can be included in either the model inputs or outputs, leading to combinatorial possibilities in task configurations. EMMa offers a new benchmark for multi-task learning in computer vision and NLP, and allows practitioners to efficiently add new tasks and object attributes at scale.

Introduction

The "data bottleneck" remains a primary hurdle in machine learning, as manually labeling millions of instances is prohibitively expensive. Furthermore, existing datasets for physical materials are often small or lack an object-centric focus. EMMa (Extensible, Multimodal dataset with Materials) addresses these gaps by providing a massive, real-world dataset where single product listings possess diverse, overlapping attributes like price, mass, material composition, and category.

The EMMa Dataset

EMMa is derived from over 2.8 million Amazon product listings (specifically 2,883,698 objects). This scale is significantly larger and more diverse than previous material datasets like OpenSurfaces or MINC. The dataset includes 7.4 million images, product text, mass, price, ratings, and a hierarchy of 11,592 categories.

Material Taxonomy

A core contribution of EMMa is its hand-curated hierarchy of 182 physical materials. This taxonomy allows for hierarchical reasoning; for instance, a model can confidently label an item as "Metal" or "Gold" even if it is uncertain about a specific sub-type like "18k Gold."

The hierarchical material taxonomy used in EMMa, featuring 182 distinct material types across categories like Metal, Wood, and Plastic.

Smart Labeling and Extensibility

EMMa is designed as a "living" dataset. The standout feature is the Smart Labeling framework, which allows researchers to extend the dataset with new binary properties (e.g., "is it sharp?" or "is it electronic?") with minimal manual effort.

Overview of EMMa's multimodal attributes and the Smart Labeling workflow for dataset extensibility.

As shown in the workflow, a user can "seed" a new label via word search, label a small subset (~1k objects) using an active learning UI, and then propagate these labels across the entire 2.8M object pool using a logistic regression model. In testing, this framework achieved F1 scores and accuracies in the 90s for five new properties with only a few hours of labeling effort per property.

Methods and Modeling

The authors utilized a multimodal late-fusion approach, extracting 768-length embeddings from a fine-tuned ConvNeXT-Base model (images) and an all-mpnet-base-v2 Transformer (text). Task-specific decoders were then trained on these standardized embeddings.

To handle missing data, such as listings without a recorded weight, the researchers employed "Synthetic Labeling." Generation 1 models predicted missing attributes, and Generation 2 models were subsequently trained on this completed dataset, leading to significant performance boosts.

Key Results and Insights

  • Text as a Physical Sensor: Text models proved surprisingly adept at capturing physical properties like mass, inferring it from the context of product descriptions even when not explicitly stated.
  • Multi-task Synergy: Training on diverse tasks simultaneously (e.g., predicting price and mass) forced the model to learn more robust, generalizable object embeddings compared to single-task training.
  • Mass Prediction: The image-based mass prediction model achieved a 0.709 Minimum Ratio Error (MnRE), outperforming the previous state-of-the-art image2mass model (0.672) on the same test set.
  • Multimodal Superiority: "Generation 2" models utilizing all available modalities (images, text, and metadata) consistently outperformed single-modality models across all tasks.

Conclusion

EMMa provides a massive, extensible platform for multi-task learning. By combining a rich material taxonomy with a Smart Labeling framework, it allows the research community to define and add new physical properties at scale, moving beyond static benchmarks toward a more dynamic, community-driven data ecosystem.

Summary updated

Abstract

We present EMMa, an Extensible, Multimodal dataset of Amazon product listings that contains rich Material annotations. It contains more than 2.8 million objects, each with image(s), listing text, mass, price, product ratings, and position in Amazon's product-category taxonomy. We also design a comprehensive taxonomy of 182 physical materials (e.g., Plastic $\rightarrow$ Thermoplastic $\rightarrow$ Acrylic). Objects are annotated with one or more materials from this taxonomy. With the numerous attributes available for each object, we develop a Smart Labeling framework to quickly add new binary labels to all objects with very little manual labeling effort, making the dataset extensible. Each object attribute in our dataset can be included in either the model inputs or outputs, leading to combinatorial possibilities in task configurations. For example, we can train a model to predict the object category from the listing text, or the mass and price from the product listing image. EMMa offers a new benchmark for multi-task learning in computer vision and NLP, and allows practitioners to efficiently add new tasks and object attributes at scale.
BibTeX
@inproceedings{standley2023extensible,
  author = {Trevor Standley and Ruohan Gao and Dawn Chen and Jiajun Wu and Silvio Savarese},
  title = {{An Extensible Multimodal Multi-task Object Dataset with Materials}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=n70oyIlS4g},
}
Figure