about
Chameleon: Mixed-Modal Early-Fusion Foundation Models (arxiv.org)
4 points by ericflo on May 17, 2024 | hide | past | pdf | discuss on HN

In plain words: Chameleon turns images and text into one shared stream of tokens, so a single model can read and create them in any order to build documents mixing pictures and words. Human judges rated its mixed image-text outputs at or above Gemini Pro and GPT-4V.

Abstract

We present Chameleon, a family of early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence. We outline a stable training approach from inception, an alignment recipe, and an architectural parameterization tailored for the early-fusion, token-based, mixed-modal setting. The models are evaluated on a comprehensive range of tasks, including visual question answering, image captioning, text generation, image generation, and long-form mixed modal generation. Chameleon demonstrates broad and general capabilities, including state-of-the-art performance in image captioning tasks, outperforms Llama-2 in text-only tasks while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and performs non-trivial image generation, all in a single model. It also matches or exceeds the performance of much larger models, including Gemini Pro and GPT-4V, according to human judgments on a new long-form mixed-modal generation evaluation, where either the prompt or outputs contain mixed sequences of both images and text. Chameleon marks a significant step forward in a unified modeling of full multimodal documents.

Chameleon Team
arXiv:2405.09818 · cs.CL · submitted May 16, 2024 · updated Mar 21, 2025
abstract · pdf · html

add comment on HN
Also discussed: May 2024 (304 points, 40 comments) · May 2024 (1 point, 0 comments) · May 2024 (6 points, 0 comments)