TL;DR
This paper introduces Refer360°, a novel dataset and task for language-guided navigation in 360° environments. It addresses the challenge of grounding language instructions in dynamic, partial views, moving beyond static object-centric targets to arbitrary locations in complex scenes.
What Problem Does This Paper Address?
Current language grounding and navigation tasks often rely on simplified scenarios: either static 2D images or 360° environments where the agent has a complete, holistic view. This doesn't reflect real-world human perception, which is characterized by a limited, dynamic field of view (FoV). Humans also often need to navigate to arbitrary points in space, not just pre-defined objects. Existing datasets either use fixed FoVs, holistic views, or restrict targets to specific objects. This paper tackles the problem of enabling AI systems to understand and follow sequential instructions in realistic 360° environments with partial, dynamic FoVs, and targets that can be anywhere.
Methods
The core contribution is the Refer360° dataset, which features:
- 360° Scenes: Utilizes 2,000 panoramic scenes from the SUN360 dataset.
- Partial and Dynamic FoV: Annotators and simulated agents experience the scene through a limited FoV that can be continuously changed (pan, tilt) to follow instructions. This mimics human visual perception.
- Arbitrary Target Locations: Targets are randomly selected points in the 3D space, not restricted to annotated objects. This leads to more complex and linguistically rich instructions.
- Sequential Instructions: Instructions are given as a sequence of steps, where each step requires adjusting the FoV and grounding intermediate references. Annotators provided at least three instructions per target.
- Annotation Process: A multi-stage process on Amazon Mechanical Turk involving both a "finding task" (where annotators locate a target and adjust their FoV) and a "description task" (where annotators provide sequential instructions to guide another annotator).
Datasets
- Refer360° Dataset: A newly introduced benchmark consisting of 17,137 instruction sequences for 2,000 360° scenes.
- SUN360 Dataset: Used as the source for the 360° panoramic images.
- Comparison Datasets: The paper compares Refer360° with existing datasets like Touchdown-SDR, Google-Ref, and Refer-UNC, highlighting Refer360°'s unique characteristics (partial/dynamic FoV, random targets).
Key Insights and Ideas
- Random Targets Drive Complexity: Restricting targets to random points, rather than pre-defined objects, forces annotators to use more complex language, relying on spatial relationships and descriptions of intermediate steps. This makes the task more challenging and realistic.
- Sequentiality is Crucial: The partial and dynamic FoV necessitates a sequential understanding of instructions. An agent cannot simply process all instructions at once; it must adapt its view and reasoning at each step.
- Linguistic Richness: The dataset captures a wide range of linguistic phenomena, including coreference, sequencing, counting, allocentric/egocentric spatial mentions, and even 3D understanding, making it valuable for studying language grounding.
- Partial FoV is a Major Hurdle: Models trained on holistic views (like LingUNet) perform significantly worse on Refer360°, underscoring the challenge posed by partial and dynamic visual input.
Key Results
- Instruction Ablation Study: Annotators' accuracy in finding the target dropped significantly when only given the last instruction (37% accuracy) or the last two (63% accuracy), compared to the full instruction sequence (88% accuracy). This confirms the necessity of all instructions.
- LingUNet Performance: The state-of-the-art LingUNet model, designed for holistic views, achieved only 13.0% accuracy on Refer360°, a stark contrast to its performance on Touchdown-SDR (23.5% replicated). This highlights the dataset's difficulty and the limitations of current approaches.
- Dataset Statistics: Refer360° images contain fewer objects on average than Touchdown-SDR but a wider variety of object types, contributing to greater lexical diversity in instructions.
Limitations and Future Work
- Model Performance: The paper acknowledges that current state-of-the-art models struggle significantly with Refer360°, indicating a need for new architectures and approaches that can handle sequential, partial-view reasoning.
- Future Directions: The authors suggest that Refer360° can be used to model tasks involving single instructions, multiple instructions, or interactive language. It also supports research into different granularities of vision (object, region, pixel) and action spaces (static, discrete, continuous). The dataset's rich linguistic phenomena also open avenues for studying nuanced language understanding.
Summary updated