about
ImageBind: One Embedding Space to Bind Them All (arxiv.org)
1 point by jonbaer on May 11, 2023 | hide | past | pdf | discuss on HN

In plain words: It lines up images, text, audio, depth, heat, and motion data in one shared space, using only image pairs instead of pairing every type with every other. On recognizing things in them with no training examples, it beat specialist models trained on labeled data.

Abstract · ImageBind: One Embedding Space To Bind Them All

We present ImageBind, an approach to learn a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together. ImageBind can leverage recent large scale vision-language models, and extends their zero-shot capabilities to new modalities just by using their natural pairing with images. It enables novel emergent applications 'out-of-the-box' including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. The emergent capabilities improve with the strength of the image encoder and we set a new state-of-the-art on emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models. Finally, we show strong few-shot recognition results outperforming prior work, and that ImageBind serves as a new way to evaluate vision models for visual and non-visual tasks.

Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra
arXiv:2305.05665 · cs.CV, cs.AI, cs.LG, cs.MM · submitted May 9, 2023 · updated May 31, 2023
abstract · pdf · html · CVPR 2023 (Highlighted Paper). Website: https://imagebind.metademolab.com/ Code/Models: https://github.com/facebookresearch/ImageBind

add comment on HN