Scene Designer: a Unified Model for Scene Search and Synthesis from Sketch

Leo Sampaio Ferraz Ribeiro, Tu Bui, John P. Collomosse, Moacir Ponti

Abstract

Scene Designer is a novel method for searching and generating images using free-hand sketches of scene compositions; i.e. drawings that describe both the appearance and relative positions of objects. Our core contribution is a single unified model to learn both a cross-modal search embedding for matching sketched compositions to images, and an object embedding for layout synthesis. We show that a graph neural network (GNN) followed by Transformer under our novel contrastive learning setting is required to allow learning correlations between object type, appearance and arrangement, driving a mask generation module that synthesises coherent scene layouts, whilst also delivering state of the art sketch based visual search of scenes.
BibTeX
@inproceedings{ribeiro2021scene,
  author = {Leo Sampaio Ferraz Ribeiro and Tu Bui and John P. Collomosse and Moacir Ponti},
  title = {{Scene Designer: a Unified Model for Scene Search and Synthesis from Sketch}},
  booktitle = {ICCVW},
  year = {2021},
  doi = {10.1109/iccvw54120.2021.00275},
}
Figure