Abstract
In this paper, we propose a multimodal search engine that combines visual and textual cues to retrieve items from a multimedia database aesthetically similar to the query. The goal of our engine is to enable intuitive retrieval of fashion merchandise such as clothes or furniture. Existing search engines treat textual input only as an additional source of information about the query image and do not correspond to the real-life scenario where the user looks for 'the same shirt but of denim'. Our novel method, dubbed DeepStyle, mitigates those shortcomings by using a joint neural network architecture to model contextual dependencies between features of different modalities. We prove the robustness of this approach on two different challenging datasets of fashion items and furniture where our DeepStyle engine outperforms baseline methods by 18-21% on the tested datasets. Our search engine is commercially deployed and available through a Web-based application.
Introduction
Traditional search engines often operate in silos, treating text and images as independent data streams. This leads to a disconnect when users attempt to perform complex, context-aware searches, such as finding "the same shirt but made of denim." In such cases, standard engines typically treat the text as mere metadata rather than a functional modifier of the visual query.
Furthermore, there is a significant gap between visual similarity (pixels that look alike) and stylistic similarity (items that belong together aesthetically). DeepStyle addresses these challenges by bridging the gap between multimodal inputs and capturing the "style" of an item through its context of use, rather than just its intrinsic visual properties.
Methodology
The DeepStyle framework utilizes two primary neural architectures to model joint multimodal spaces:
- DeepStyle (Joint Embedding): This model integrates image features extracted via a pre-trained ResNet-50 with textual features processed through word2vec. By training on a classification task, the network learns a shared representation where semantic and visual regularities are preserved.
- DeepStyle-Siamese: This advanced iteration employs a Siamese architecture with two branches. It uses a contrastive loss function to ensure that items appearing in the same context—such as a single outfit or a curated room scene—are mapped closer together in the embedding space, while unrelated items are pushed apart.
To evaluate performance, the authors introduced the Average Intra-List Similarity (AILS) metric, which quantifies the empirical probability that retrieved items would appear together in human-curated sets.
Datasets
The research leverages two distinct datasets to test the model's robustness:
- Polyvore: A large fashion dataset containing over 82,000 items. The authors enhanced this dataset through web scraping to provide richer textual descriptions and cleaner outfit groupings.
- IKEA Dataset: A novel contribution consisting of 298 room scenes and 2,193 product photos. This dataset is unique in its combination of interior scene context with individual product metadata, filling a gap in publicly available interior design research data.
Key Results
DeepStyle demonstrates significant improvements over state-of-the-art baselines like Visual Search Embedding (VSE). Specifically, the DeepStyle-Siamese model achieved a 21% performance gain on the IKEA dataset and a 32% gain on the Polyvore dataset in terms of AILS.
Qualitatively, the model excels at interpreting textual modifiers that change the intent of a visual query. As shown in the comparison below, when a user provides an image of a blazer and adds the text "+ gold," DeepStyle correctly retrieves a gold-colored blazer, whereas the VSE baseline fails by returning unrelated orange pants.
Conclusion
DeepStyle represents a shift from simple visual matching to context-aware stylistic retrieval. By treating style as a latent variable derived from item co-occurrence, the model provides a more intuitive search experience that aligns with real-world user behavior. The system is currently commercially deployed, proving its viability for large-scale applications in fashion and interior design. Future work will explore extending this modality-agnostic architecture to include audiovisual data for even more immersive search experiences.
Summary updated