Abstract
Traditional referring expression resolution focuses on identifying well-defined objects. However, in many real-world scenarios, identifiable objects may be absent. This paper introduces SomethingFinder, a system designed to locate "undefined" regions using low-level visual cues like color and geometry. We present the Something-W dataset, consisting of 25,000 expressions, and a model architecture that prioritizes early-stage CNN features. Our approach demonstrates that local spatial context and low-level attributes are sufficient for effective object-agnostic localization.
Introduction: Beyond Object-Centric Vision
In computer vision, referring expression resolution typically involves identifying a specific object, such as "the man in the red hat." This object-centric approach relies on the presence of nameable landmarks. However, humans can easily describe regions in "messy" or novel environments using basic visual attributes—colors, textures, and geometric features like lines and corners. SomethingFinder addresses this object-agnostic reference challenge: finding a location when the description lacks high-level conceptual labels.
Methodology: Prioritizing Low-Level Cues
The SomethingFinder architecture modifies the GroundNet framework to emphasize geometric and color-based details over high-level object recognition.
- Visualize Module: Standard models often use "deep" CNN features that excel at classification but lose spatial precision. SomethingFinder utilizes early-stage features (Conv1 or Conv2 from ResNet50) to retain raw geometric information, such as edges and corners.
- LookAround Module: This module predicts target blocks by evaluating the visual features of a region and its immediate neighbors (up, down, left, right). It matches these features against spatial relationships described in the text, such as "to the left of a gray vertical line."
- Curriculum Learning: The model is trained using a "simple-to-complex" strategy, starting with expressions referencing few neighbors before progressing to more intricate descriptions.
The Something-W Dataset
To benchmark this task, we introduced Something-W, built upon the Wireframe dataset. It contains 25,000 automatically generated expressions across 15,000 training images.
The dataset uses a vocabulary of 21 colors and geometric labels (vertical, horizontal, or diagonal lines; corners). Expressions are categorized by complexity ($\eta$), which denotes the number of neighboring regions used as anchors.
Key Results and Insights
The model was evaluated using Top-K accuracy on a grid of 4x4 blocks.
- Performance: The best configuration (AggEnc with
conv2features) achieved a Top-1 accuracy of 11.0% and a Top-10 accuracy of 53.6%. While seemingly low, this is a significant result given the difficulty of pinpointing small, non-object regions in complex scenes. - The Value of Low-Level Data: In this task, lines and colors are the primary signals rather than noise. Using early CNN layers—a deliberate "downgrade" in abstraction—proved essential for performance.
- Curriculum Learning Impact: This strategy improved Top-1 accuracy to 11.8% and halved the required training time.
Conclusion and Future Work
SomethingFinder demonstrates the feasibility of object-agnostic localization. While the current dataset uses synthetic language templates, future work will focus on more diverse, natural human speech and multi-scale region proposals. This research paves the way for robots and AI systems to navigate and interact with environments where traditional object landmarks are unavailable.
Summary updated