본 논문에서는 인물 사진에서 자동으로 캐릭터를 생성하는 diffusion 기반 모델을 제안한다. 우리의 네트워크는 세 단계 diffusion process, UNet, denoising 과정을 거쳐 최종 캐릭터를 생성한다. diffusion process에서는 세부스타일까지 손실 없이 학습하게 하기 위해 스타일 벡터에 노이즈를 점진적으로 추가한 노이즈 벡터 집합을 만든다. 스타일 이미지를 제외한 모든 입력 값은 CLIP 인코더로 벡터로 만든 뒤, 앞서 생성한 노이즈 스타일 벡터와 UNet에서 학습하게 된다. 우리는 세부 조건을 조정하기 위해 CLIP 인코더를 사용한다. 그 후 UNet을 통한 벡터의 노이즈를 제거해 최종적인 캐릭터 이미지를 얻는다.
With the recent advent of Metaverse, the character industry that reflects the characteristics of users' faces is drawing attention. there is a hassle that users have to select face components such as eyes, nose, and mouth one by one. In this paper, we propose a diffusion-based model that automatically generates characters from content human photographs. Our model generates user artistic characters by reflecting content information such as face angle, direction, and shape of a content human photo. In particular, our model automatically analyzes detailed information such as glasses and whiskers from content photo images and reflects them in artistic characters generated. Our network generates the final character through a three-step: diffusion process, UNet, and denoising processes. We use image encoders and CLIP encoders for the connection between style and input data. In the diffusion process, a collection of noise vectors is gradually added to a style vector to enable lossless learning of the detailed styles. All input values except for the style images are vectorized with CLIP encoders and then learned with noise style vectors in the UNet. Subsequently, noise is removed from the vectors through the UNet to obtain the artistic character image. We demonstrate our performance by comparing the results of other models with our results. Our method reflects content information without loss and generates natural high-definition characters.
1. Introduction
2. Body
3. Conclusion
Reference