Abstract
Consumer-to-consumer (C2C) marketplaces represent a significant source of unstructured data, yet their inherent noise—ranging from inconsistent descriptions to varying image quality—poses a challenge for traditional classification models. This study evaluates the efficacy of OpenFlamingo, an open-source multimodal vision-language model, in organizing a massive dataset of over 1.2 million automotive part listings from Craigslist and OfferUp. By utilizing joint text-image embeddings and k-means clustering, we demonstrate that multimodal models can successfully categorize heterogeneous data "in the wild," while also identifying specific architectural limitations regarding multi-image processing and textual boilerplate.
Introduction
Online marketplaces like Craigslist and OfferUp are goldmines of consumer data, but they are notoriously difficult to parse. Listings often feature a chaotic mix of technical specifications, informal language, and varying photographic perspectives. Traditional Natural Language Processing (NLP) or Computer Vision (CV) models typically analyze these signals in isolation, often missing the broader context of a listing.
This research investigates whether a multimodal approach—one that processes text and images simultaneously—can effectively categorize high-volume, heterogeneous automotive data. Specifically, the study evaluates how well OpenFlamingo, a state-of-the-art vision-language model, handles real-world data without explicit domain-specific fine-tuning.
Methodology
The research pipeline was structured into four distinct phases:
- Data Acquisition: The team scraped a comprehensive dataset of over 1.2 million posts from OfferUp and Craigslist, representing approximately 550GB of raw data.
- Preprocessing & Tokenization: To provide the model with necessary context, each post was prefixed with the string "This is a post." Specific tokens, such as
<image>, were utilized to denote the placement of visual data within the sequence, ensuring the model could align text with its corresponding imagery. - Embedding Extraction: The researchers utilized OpenFlamingo, which incorporates a frozen CLIP ViT-L/14 vision encoder and a frozen language model. This architecture was used to generate high-dimensional vectors (embeddings) representing the synthesized meaning of the text and images.
- Clustering & Visualization: K-means clustering was applied with a target of $k=20$ clusters. To interpret these high-dimensional results, the team used Uniform Manifold Approximation and Projection (UMAP) to project the clusters into a 2D space for qualitative analysis.
Dataset Characteristics
The study utilized a massive, newly assembled dataset of C2C automotive listings:
- OfferUp: 650,654 posts.
- Craigslist: 637,679 posts.
- Total: 1,288,333 posts.
Unlike standard benchmarks such as COCO or LAION, this dataset is "in the wild." It contains significant noise, including varying image counts per post, informal slang, and inconsistent formatting, making it a rigorous test for zero-shot embedding capabilities.
Key Insights and Discussion
Multimodal Synergy
The model demonstrated a clear ability to synthesize signals. Rather than grouping items solely by visual similarity or keyword matching, it successfully identified semantically coherent clusters such as "Tires and Trucks" or "Internal Engine Components." This suggests that the joint embedding space effectively captures the relationship between how a part looks and how it is described.
The "Boilerplate" Trap
An interesting phenomenon emerged within the Craigslist data: the "boilerplate" trap. One specific cluster was defined not by the automotive parts themselves, but by the structural patterns of the text—specifically, recurring contact information and closing statements (e.g., "If you need other parts, call me"). This indicates that the language component of the model can sometimes prioritize structural text patterns over the primary subject matter.
Frozen Model Utility
The study highlights the utility of "frozen" models. Despite not being fine-tuned on automotive data, the frozen weights of the CLIP and language model components were sufficient to extract meaningful features from a niche domain. This suggests that large-scale multimodal models possess significant transfer learning capabilities for specialized industrial or commercial applications.
Results
The researchers successfully partitioned the 1.2 million posts into 20 distinct clusters. Qualitative analysis of the ten posts closest to each cluster's centroid confirmed high semantic purity in the majority of groups.
- Craigslist Results: Clear clusters emerged for outer body parts, seating, tires, and internal mechanical components.
- OfferUp Results: Similar success was found with clusters dedicated to lighting (head/tail lights) and "colorful body parts."
- Outliers: Approximately 15% of the clusters (3 out of 20) lacked a clear internal pattern, appearing as "catch-all" groups for miscellaneous or highly noisy data.
Limitations and Future Work
The primary bottleneck identified was OpenFlamingo’s handling of multiple images. C2C listings frequently include 5–10 photos to show different angles or defects of a part. Because the current architecture is optimized for processing a single image per sequence in this context, the model occasionally struggled to aggregate information from image galleries, leading to some randomness in categorization.
Future research should focus on modifying the model architecture to better aggregate multiple visual elements within a single embedding. Improving this multi-image integration would likely result in tighter, more consistent clusters, enhancing the utility of these models for automated marketplace moderation and advanced search functionality.
Summary updated