-
Unsupervised Scene Sketch to Photo Synthesis
-
Jiayun Wang and Sangryul Jeon and Stella X. Yu and Xi Zhang and Himanshu Arora and Yu Lou
-
European Conference on Computer Vision Workshop on Advances in Image Manipulation (AIM), Tel Aviv, Isarel, 23 October 2022
-
Paper
|
Slides
-
Abstract
-
Sketches make an intuitive and powerful visual expression as they are fast executed freehand drawings. We present a method for synthesizing realistic photos from scene sketches. Without the need for sketch and photo pairs, our framework directly learns from readily avail- able large-scale photo datasets in an unsupervised manner. To this end, we introduce a standardization module that provides pseudo sketch-photo pairs during training by converting photos and sketches to a standardized domain, i.e. the edge map. The reduced domain gap between sketch and photo also allows us to disentangle them into two components: holistic scene structures and low-level visual styles such as color and texture. Taking this advantage, we synthesize a photo-realistic image by combining the structure of a sketch and the visual style of a reference photo. Extensive experimental results on perceptual similarity metrics and human perceptual studies show the proposed method could generate realistic photos with high fidelity from scene sketches and outperform state-of-the-art photo synthesis baselines. We also demonstrate that our framework facilitates a controllable manipulation of photo synthesis by editing strokes of corresponding sketches, delivering more fine-grained details than previous approaches that rely on region-level editing.
-
Keywords
-
unsupervised learning, sketch to photo synthesis
|