Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing ECCV 2024

Vadim Titov AIRI icon
Madina Khalmatova UNSW icon
Alexandra Ivanova HSE icon AIRI icon Skol icon
Dmitry Vetrov CU icon
Aibek Alanov HSE icon AIRI icon
HSE icon
HSE University
AIRI icon
AIRI
Skoltech icon
Skolkovo Institute of Science and Technology
UNSW icon
UNSW Sydney
CU icon
Constructor University, Bremen
Guide-and-Rescale teaser
The proposed Guide-and-Rescale method allows to manipulate images for a wide range of different editings. It achieves a good balance between quality of manipulation and preservation of the original image.

Abstract

Despite recent advances in large-scale text-to-image generative models, manipulating real images with these models remains a challenging problem. The main limitations of existing editing methods are that they either fail to perform with consistent quality on a wide range of image edits or require time-consuming hyperparameter tuning or fine-tuning of the diffusion model to preserve the image-specific appearance of the input image. We propose a novel approach that is built upon a modified diffusion sampling process via the guidance mechanism. In this work, we explore the self-guidance technique to preserve the overall structure of the input image and its local regions appearance that should not be edited. In particular, we explicitly introduce layout-preserving energy functions that are aimed to save local and global structures of the source image. Additionally, we propose a noise rescaling mechanism that allows to preserve noise distribution by balancing the norms of classifier-free guidance and our proposed guiders during generation. Such a guiding approach does not require fine-tuning the diffusion model and exact inversion process. As a result, the proposed method provides a fast and high-quality editing mechanism. In our experiments, we show through human evaluation and quantitative analysis that the proposed method allows to produce desired editing which is more preferable by humans and also achieves a better trade-off between editing quality and preservation of the original image.

Overview of Guide-and-Rescale

First, our method uses a classic ddim inversion of the source real image. Then the method performs real image editing via the classical denoising process. For every denoising step the noise term is modified by guider that utilizes latents $z_t$ from the current generation process and time-aligned ddim latents $z^*_t$.

Guide-and-Rescale Results

Visual comparison of our method with baselines over different types of editing. Our approach shows more consistent results than existing methods and achieves a better trade-off between editing quality and preservation of the structure of the original image.

BibTeX

@article{titov2024guideandrescale
    title={Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing},
    author={Vadim Titov and Madina Khalmatova and Alexandra Ivanova and Dmitry Vetrov and Aibek Alanov},
    journal={arXiv preprint arXiv:2409.01322},
    year={2024}
}