SDEdit论文笔记

SDEdit

SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (ICLR 2022)

contribution

• 引入新的图像合成和编辑方法Stochastic Differential Editing (SDEdit)，通过随机微分方程（SDE）逆向求解生成图像。
• 可以自然地在真实性和忠诚（输入相关性）之间实现平衡。
• SDEdit在多项任务（包括基于笔画的图像合成和编辑）上优于最先进的基于GAN的方法。

Image Synthesis with Stochastic Differential Equations

扩散模型的前向过程是对原始样本添加多步噪声扰动，本文将多步扰动推广到无限次扰动，即使用随机微分方程模拟噪声扰动数据变化的过程。同时随机微分方程本身有逆向解法，因此自然实现了从noise到image的图像生成。

Perturbing data with multiple noise scales is key to the success of previous methods. We propose to generalize this idea further to an infinite number of noise scales, such that perturbed data distributions evolve according to an SDE as the noise intensifies

• Perturbing data with SDEs 正向过程

  $x(t) = α(t)x(0) + σ(t)z, z ∼ N (0, I)$

σ(t) : [0, 1] → [0, ∞) describes the magnitude of the noise z
α(t) : [0, 1] → [0, 1] denotes the magnitude of the data x(0)

VE-SDE:

reverse SDE

Reverse VE-SDE:

其中$\nabla_{x}logp_t(x)$可用score function进行估计，训练score network($sθ(x(t),t)$)拟合$\nabla_{x}logp_t(x)$，常用网络Unet

SDE solution: 选取0，1之间离散化的时间序列，不断迭代,生成x(0)

Loss Function

The overall training objective is a weighted sum over t of each individual learning objective $L_t$

Guided image synthesis and Editing with SDEdit

task

• Realism-faithfulness trade-off

  Our goal is to produce full resolution images with two desiderata:

  Realism: The image should appear realistic measured by KID
  Faithfulness: The image should be similar to the guide $x^{(g)}$ measured by $L_2$ distance.

• Method

为了平衡Realism-faithfulness，需要选择恰当的时间$t_0$进行Reverse SDE

• Detail

经验显示t在[0.3-0.6]范围内效果较好，同时也可以根据具体任务微调。

• Experiments

Stroke-based image generation SDEdit 大大的提高了对guide信息的忠诚度，同时生成图片满足真实性