about
VQGAN-CLIP: Open Domain Image Generation/Editing with Natural Language Guidance (arxiv.org)
17 points by minimaxir on Apr 20, 2022 | hide | past | pdf | 1 comment on HN

In plain words: An image generator is steered by a system that scores how well a picture matches a written description, letting any prompt create or edit images with no training. It produced better-looking images than earlier specially trained systems for text-to-image and editing.

Abstract · VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance

Generating and editing images from open domain text prompts is a challenging task that heretofore has required expensive and specially trained models. We demonstrate a novel methodology for both tasks which is capable of producing images of high visual quality from text prompts of significant semantic complexity without any training by using a multimodal encoder to guide image generations. We demonstrate on a variety of tasks how using CLIP [37] to guide VQGAN [11] produces higher visual quality outputs than prior, less flexible approaches like DALL-E [38], GLIDE [33] and Open-Edit [24], despite not being trained for the tasks presented. Our code is available in a public repository.

Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, Edward Raff
arXiv:2204.08583 · cs.CV · submitted Apr 18, 2022 · updated Sep 4, 2022
abstract · pdf · html · Accepted for publication at ECCV 2022 Code available at https://github.com/EleutherAI/vqgan-clip/tree/main/notebooks

add comment on HN

Well done! Beautiful image quality, and under 11GB vram is excellent.

https://github.com/EleutherAI/vqgan-clip/tree/main/notebooks

I feel better about humanity when you guys roll things out. Kudos to you and Coreweave!