본 연구에서는Stable Diffusion 프레임워크를 활용하여 게임 스타일의 스케치, 특히 도시 장면을 생성하는 방법을 소개한다. 확산 기반의 모델인Stable Diffusion은 쉬운 접근성과 뛰어난 성능으로 많은 연구자와 일반인들에게 선호되며, 텍스트-스케치, 이미지-스케치의 생성이 가능하다. Stable Diffusion의 몇 가지 문제는 이미지의 국소성 보존 문제 및 미세 조정인데, 이를ControlNet과DreamBooth를 사용하여 해결한다. 결과적으로, 본 연구를 통해 게임 제작에 사용될 수 있는 텍스트-스케치, 이미지-스케치 생성이 가능하며, 더 나아가 아티스트를 돕는 툴로도 활용될 수 있다.
Games are a vital part of our culture, leading the generation of tools such as Adobe Photoshop and Unity for game developers. Although sketches are a fundamental form that can be stylized in various ways, there is a scarcity of tools capable of generating images into sketch. To bridge this gap in the artistic sketch domain of gaming and the field of deep generative models, we propose a multimodal sketch generation framework with Stable Diffusion, focusing especially on urban scenes. Stable Diffusion, a model within the diffusion-based category, has gained notable attention in the open-source community and is user-friendly. Thus, we have chosen to utilize Stable Diffusion in our approach. This model processes input prompts and images through the CLIP encoder and effectively generates images. However, Stable Diffusion faces challenges such as a losing locality of input image and difficulties in fine-tuning. To overcome these issues, we incorporate ControlNet and DreamBooth into our framework. We conclude with a demonstration of promising results, urban landscape sketch, in both text-to-sketch and image-to-sketch generation.
1. Introduction
2. Method
3. Conclusion
Reference