An Extensible Multi-modal Multi-task Object Dataset with Materials

Trevor Scott Standley, Ruohan Gao, Dawn Chen, Jiajun Wu, Silvio Savarese

TL;DR

EMMa is a massive new multimodal dataset featuring over 2.8 million Amazon product objects, each annotated with a rich taxonomy of 182 materials and multiple attributes like mass, price, and ratings. Its standout feature is a "Smart Labeling" framework that allows researchers to add new binary object properties (e.g., "is it fragile?") to the entire dataset with only a few hours of manual effort.

Overview of the EMMa dataset and its Smart Labeling framework.

What Problem Does This Paper Address?

The "data bottleneck" remains the primary hurdle for machine learning practitioners. Manually labeling millions of instances is prohibitively expensive, and communicating subtle corner cases to human annotators (e.g., "Should a hole puncher be labeled as 'sharp'?") is notoriously difficult.

Furthermore, while computer vision has many datasets, few are truly object-centric and multimodal while also covering materials. Most material datasets focus on pixel-level surface textures (like "wood" or "glass") rather than the composition of the object itself (like "copper wire inside a plastic sheath"). EMMa aims to bridge this gap by providing a large-scale, extensible benchmark for multi-task learning across vision and language.

Methods: Building EMMa and Smart Labeling

The authors employ several clever strategies to build and expand EMMa:

  1. Late-Fusion Multimodal Modeling: They fine-tune a ConvNeXT model for images and an all-mpnet-base-v2 Transformer for text. Instead of end-to-end training for every task, they extract 768-dimensional embeddings from these models and feed them into task-specific decoders. This makes the system modular and computationally tractable.
  2. Smart Labeling Framework: This is the paper's core innovation. To add a new label, a user follows a four-step pipeline:
  3. Word Search: Seed the training set using keywords from product descriptions.
  4. Active Learning: The system presents the user with "uncertain" objects (those with predicted probabilities near 0.5) to refine the decision boundary.
  5. Correction/Review: The user fixes false positives/negatives to hone precision and recall.
  6. Synthetic Labeling (Gen 2): Since many Amazon listings have missing fields (e.g., no weight or price listed), the authors trained "Everything" models that use all available modalities to predict missing values, effectively "filling in" the dataset to provide denser supervision.

Dataset Composition and Material Taxonomy

The researchers synthesized EMMa by merging and filtering three primary sources: the Amazon Selling Partner API (ASPA), image2mass, and UCSD Amazon product reviews. After aggressive filtering for duplicates and quality, the final EMMa dataset contains 2,883,698 instances.

The comprehensive taxonomy of 182 physical materials used in EMMa.

It is significantly larger and more diverse in material types than existing benchmarks like OpenSurfaces (36 categories) or MINC (23 categories). The dataset successfully maps objects to a hierarchy of 182 materials, providing a much more granular understanding of object composition than previous datasets.

Key Insights and Results

  • Text as a Physical Sensor: One of the most surprising findings is that text-only models are remarkably good at predicting physical properties like mass, even when the weight isn't explicitly mentioned in the text. The model likely infers mass from the object's category and description.
  • The Power of Soft Labels: By using the Smart Labeling tool to generate probabilistic labels rather than hard binary ones, the authors found that models are more robust. The uncertainty in the label prevents the model from over-fitting to noisy or ambiguous data.
  • Combinatorial Task Possibilities: Because EMMa treats every attribute (image, text, price, mass, etc.) as potentially either an input or an output, it allows for a massive variety of task configurations, making it a "Swiss Army Knife" for multi-task learning research.
  • Mass Prediction: Their multimodal model achieved a Minimum Ratio Error (MnRE) of 0.709, outperforming the previous state-of-the-art image2mass model (0.672).
  • Smart Labeling Efficiency: Using the Smart Labeling tool, the authors added five new properties (Sharp, Transparent, Deformable, Articulating, Electronic). They achieved F1 scores in the 90s for all five properties with minimal human labeling. For example, the "Sharp" property reached 92.0% F1 and 99.7% accuracy.

Limitations and Future Work

  • Synthetic Reliance: A significant portion of the dataset's labels are "synthetic" (predicted by models). While the authors performed sanity checks, these are still approximations of ground truth.
  • Domain Specificity: EMMa is strictly composed of Amazon product listings. While massive, it may not generalize perfectly to "in-the-wild" robotics scenarios or natural outdoor scenes where lighting and context differ from studio product photography.
  • Future Directions: The authors suggest that EMMa could be used for image generation (using text/attributes as input) or captioning (using images/attributes to generate text), though they did not explore these tasks in the current paper.

Summary updated

Abstract

We present EMMa, an Extensible, Multimodal dataset of Amazon product listings that contains rich Material annotations. It contains more than 2.8 million objects, each with image(s), listing text, mass, price, product ratings, and position in Amazon’s product-category taxonomy. We also design a comprehensive taxonomy of 182 physical materials (e.g., Plastic → Thermoplastic → Acrylic). Objects areannotated with one or more materials from this taxonomy. With the numerous attributes available for each object, we develop a Smart Labeling framework to quickly add new binary labels to all objects with very little manual labeling effort, making the dataset extensible. Each object attribute in our dataset can be included in either the model inputs or outputs, leading to combinatorial possibilities in task configurations. For example, we can train a model to predict the object category from the listing text, or the mass and price from the product listing image. EMMa offers a new benchmark for multi-task learning in computer vision and NLP, and allows practitioners to efficiently add new tasks and object attributes at scale.
BibTeX
@inproceedings{standley2023extensible,
  author = {Trevor Scott Standley and Ruohan Gao and Dawn Chen and Jiajun Wu and Silvio Savarese},
  title = {{An Extensible Multi-modal Multi-task Object Dataset with Materials}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=n70oyIlS4g},
}
Figure