Sketch-to-Art: Synthesizing Stylized Art Images from Sketches

Bingchen Liu, Kunpeng Song, Yizhe Zhu, Ahmed Elgammal

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We propose a new approach for synthesizing fully detailed art-stylized images from sketches. Given a sketch, with no semantic tagging, and a reference image of a specific style, the model can synthesize meaningful details with colors and textures. Based on the GAN framework, the model consists of three novel modules designed explicitly for better artistic style capturing and generation. To enforce the content faithfulness, we introduce the dual-masked mechanism which directly shapes the feature maps according to sketch. To capture more artistic style aspects, we design feature-map transformation for a better style consistency to the reference image. Finally, an inverse process of instance-normalization disentangles the style and content information and further improves the synthesis quality. Experiments demonstrate a significant qualitative and quantitative boost over baseline models based on previous state-of-the-art techniques, modified for the proposed task (17% better Frechet Inception distance and 18% better style classification score). Moreover, the lightweight design of the proposed modules enables the high-quality synthesis at 512 × 512 resolution.

Original languageEnglish (US)
Title of host publicationComputer Vision – ACCV 2020 - 15th Asian Conference on Computer Vision, 2020, Revised Selected Papers
EditorsHiroshi Ishikawa, Cheng-Lin Liu, Tomas Pajdla, Jianbo Shi
PublisherSpringer Science and Business Media Deutschland GmbH
Pages207-222
Number of pages16
ISBN (Print)9783030695439
DOIs
StatePublished - 2021
Event15th Asian Conference on Computer Vision, ACCV 2020 - Virtual, Online
Duration: Nov 30 2020Dec 4 2020

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12627 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference15th Asian Conference on Computer Vision, ACCV 2020
CityVirtual, Online
Period11/30/2012/4/20

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint

Dive into the research topics of 'Sketch-to-Art: Synthesizing Stylized Art Images from Sketches'. Together they form a unique fingerprint.

Cite this