Introduction
Modern photo management systems excel at keyword-based searches like "dog" or "beach." However, they frequently falter when faced with structured image retrieval—tasks defined by complex semantic relationships and logical constraints. For instance, a user might seek a specific moment where "the bride is to the left of the groom while holding flowers." Such queries expose the "semantic gap" in current vision models, which struggle to reason about high-level spatial and logical relationships.
System Architecture: Neuro-Symbolic Synthesis
PhotoScout addresses these challenges through a neuro-symbolic program synthesis approach. Rather than relying solely on embedding similarity, the system constructs a formal program to execute the search.
- Natural Language to Sketch: An LLM (GPT-3.5 Turbo) translates natural language queries into "program sketches" within a custom Domain-Specific Language (DSL) based on first-order logic.
- Grounding and Interaction: To resolve ambiguities (e.g., identifying specific individuals), users provide visual examples or tag objects. These "ground" the abstract logic in visual data.
- Program Synthesis: PhotoScout performs an enumerative search to fill "holes" in the program sketch, identifying the simplest logical structure that satisfies all user-provided positive and negative examples.
- Execution: The final program leverages neural models (Amazon Rekognition) for object detection and attribute recognition, then applies the synthesized logic across the image library.
Evaluation and Results
The system was evaluated using three distinct Flickr-sourced datasets: Transportation, Festival, and Wedding. These datasets represent personal collections where structured queries are most prevalent.
In a user study with 25 participants, PhotoScout was compared against a CLIP-based baseline (ClipWrapper). Key findings include:
- Improved Accuracy: PhotoScout achieved a 34% higher F1 score than the baseline.
- Efficiency and Confidence: While synthesis introduces a brief computational delay (up to 4.8 seconds), users reported higher confidence in results and required fewer manual refinements.
- Explainability: The symbolic nature of the programs allows the system to provide natural language explanations for its selections, fostering user trust.
Discussion and Future Work
While PhotoScout significantly advances structured retrieval, it remains dependent on the accuracy of underlying object detectors. Failures in detection (e.g., obscured faces) propagate to the logical layer. Future research will explore "human-in-the-loop" corrections, allowing users to refine detection results to further improve synthesis outcomes.
Summary updated