Paint by Example: Exemplar-based Image Editing with Diffusion Models (CVPR 2023) by Yang et al.

● Summary: Simple method for Image editing with a diffusion model only using CLIP [CLS] token embedding

 

● Approach highlight

• Image editing without labels using only the detection model
• Crop the original image and augment the image for CLIP embedding
• Only use [CLS] token to prevent the model from just doing copy-and-paste
• Classifier free sampling for image identity (scale factor)

 

● Main Results

 

 

● Discussion

• Could it have been possible to use the entire CLIP embedding instead of just the [CLS] token, but in a reduced form?