Abstract
FArMARe introduces a novel "text-to-apartment" recommendation task that allows users to find their ideal home using free-form natural language queries instead of rigid filters. By employing a multi-task learning framework that combines cross-modal contrastive ranking with a furniture-aware classification objective, the researchers significantly improved the accuracy of matching complex 3D apartment scenes to detailed user descriptions.
Introduction
Finding a new apartment is notoriously tedious. While current web platforms allow users to filter by price, location, or number of rooms, they fail to capture nuanced preferences—like a "wide dining room with a wooden table" or a "minimalist bedroom with a king-size bed."
As we move toward "digital twins" and metaverse-based property tours, there is a growing need for systems that can search 3D environments based on these specific interests. However, existing 3D scene retrieval methods usually rely on image-based queries or sketches, which are less intuitive for users than text. Furthermore, standard vision-language models often struggle with the sheer density of information in an apartment (multiple rooms, dozens of furniture pieces, and varied styles).
Methodology
The authors propose FArMARe (Furniture-AwaRe Multi-task methodology for Apartment REcommendation). The architecture consists of two main branches:
- Visual Branch: The system samples multiple viewpoints from a 3D apartment scene. These images are processed by a Vision Transformer (ViT) pre-trained via CLIP. To handle the sequence of images, they use a 1D convolutional network (Conv1d) to extract features before mapping them into a joint embedding space.
- Textual Branch: Since apartment descriptions are often very long (averaging 319 words), the authors split them into individual sentences. These are processed by a CLIP-based Transformer and a bidirectional GRU to create a final semantic representation.
Multi-task Learning
Standard models only use a Contrastive Loss (triplet loss) to pull matching text and images together in the embedding space. FArMARe adds a second Furniture-Aware Classification task. Using tags generated by a fine-tuned YOLOv8 detector (e.g., "wardrobe," "dining table"), the model is forced to explicitly identify furniture items within the images. This "furniture-awareness" ensures the visual features capture the specific details users actually care about.
The FArMARe Dataset
The researchers found that existing indoor datasets lacked the detailed textual descriptions needed for this task. Consequently, they introduced a new benchmark:
- The FArMARe Dataset: Comprising 6,081 indoor apartments derived from the 3d-FRONT dataset.
- Annotations: Each apartment is annotated with detailed descriptions of furniture categories, styles, themes, and materials.
- Comparison: Unlike previous datasets like Houses3k (exterior only) or Matterport3d (no text), this dataset provides high-quality indoor visual data paired with rich, descriptive language.
Key Results and Insights
The authors compared FArMARe against several baselines (Average-pool networks and standard Convolutional networks):
- State-of-the-Art Performance: FArMARe achieved 24.8% R@1 and 49.1% R@10 for the text-to-apartment task using CLIP features.
- Multi-task Benefit: Adding the furniture-aware objective provided a clear lift over the single-task version (CNV), improving R@1 by +1.1% and R@5 by +2.0%.
- Joint Pre-training is Essential: Using backbones (like CLIP) that were jointly trained on vision and language provides a massive performance boost (+30% R@10) compared to using separate models like ResNet and BERT.
- Granularity Matters: Breaking long descriptions into sentence-level features is more effective than treating the entire text as a single "bag of tokens." This helps the model handle the high word count without losing local context.
Conclusion and Future Work
While FArMARe sets a new benchmark for apartment recommendation, several challenges remain. The current system relies on an object detector to provide tags for the multi-task objective; if the detector fails or only picks one "predominant" object in a crowded room, the model receives incomplete information. Furthermore, the model does not yet explicitly model the spatial relationship between items (e.g., "the chair next to the window").
Future directions include moving toward multi-label classification to identify all objects in a frame and incorporating spatial attention mechanisms to improve the "explainability" and precision of recommendations.
Summary updated