Abstract
Object Referring Analysis (ORA) requires models to identify specific objects based on complex natural language descriptions, a task where standard zero-shot vision-language models often fail due to a lack of spatial logic and a tendency for Large Language Models (LLMs) to hallucinate. We present FLORA (Formal Language for Object Referring and Analysis), a training-free framework that leverages LLMs as logical translators. By constraining LLM output to a structured formal language and employing a Bayesian probabilistic framework to integrate multiple off-the-shelf vision interpreters, FLORA achieves significant performance gains. Experimental results on benchmarks like RefCOCO show up to a 45% relative improvement in accuracy over state-of-the-art grounding detectors without any fine-tuning.
Introduction
Object Referring Analysis (ORA) encompasses tasks such as identifying a specific object within a bounding box (Comprehension) or a mask (Segmentation) based on natural language descriptions. Unlike standard object detection, ORA requires understanding complex relationships, such as "the blue cracked cup to the left of the laptop."
Current state-of-the-art models typically rely on massive annotated datasets. Zero-shot approaches often struggle because Vision-Language Models (VLMs) frequently treat descriptions as "bags-of-words," ignoring spatial logic. While Large Language Models (LLMs) excel at reasoning, they are prone to "hallucinations"—generating details not present in the image—which can mislead ORA systems.
The FLORA Framework
FLORA (Formal Language for Object Referring and Analysis) addresses these challenges through a three-pillar approach:
1. Formal Language Model (FLM)
To prevent LLM hallucinations and rambling, FLORA defines a strict syntax using hashtags (e.g., #vase, #on left, #swirly orange). This forces the LLM to categorize its reasoning into four distinct semantic buckets: Object Type, Location, Visual Pattern, and Object Relation.
2. Context-guided Prompting (CoP-FLM)
The system utilizes specialized system-level and instance-level prompts to guide models like Llama-3. This process translates unstructured human descriptions into the structured FLM format, effectively using the LLM as a logical regulator.
3. Bayesian Probabilistic Inference
Once the description is structured, FLORA employs a Bayesian framework to score detected objects. It treats each component (type, location, pattern, relation) as an independent variable. "Interpreters"—off-the-shelf models such as Grounding DINO for object types and CLIP for visual patterns—calculate the probability of an object matching each parsed component. The final selection is based on the highest combined probability.
Experimental Evaluation
FLORA was evaluated across several benchmark datasets, demonstrating its versatility and robustness:
- RefCOCO/+/g: On the RefCOCO validation set, FLORA improved accuracy from 50.4% to 73.7%—a relative improvement of nearly 45%. It consistently outperformed other zero-shot methods like ReCLIP and SS-CLIP.
- Segmentation: When integrated with Segment Anything (SAM), FLORA achieved an oIoU of 55.7 on RefCOCO, significantly higher than the Grounding DINO + SAM baseline (35.3).
- Who’s Waldo: In person-centric linking tasks, FLORA achieved 62.1% accuracy, nearing the supervised state-of-the-art (63.5%) without any task-specific training.
- PhraseCut: Tested on large-scale referring expression segmentation with diverse attributes.
Conclusion
FLORA demonstrates that the interaction between LLMs and vision models can be significantly improved without additional training. By decomposing complex descriptions into a formal language and applying probabilistic reasoning, FLORA achieves state-of-the-art zero-shot performance. Future work may extend this framework to real-time applications like robotics and specialized domains like medical imaging.
Summary updated