about
Cross-Modal Alignment Without Paired Data (arxiv.org)
2 points by nicholascarolan 35 days ago | hide | past | pdf | discuss on HN

In plain words: Two separately trained models can be matched without any paired examples by finding one rotation that lines up the shapes of their internal meaning spaces. This works across many datasets and, with only a handful of pairs, beats usual alignment methods.

Abstract · Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data

Multimodal representations enable zero-shot classification and retrieval, but aligning independently trained models usually requires large amounts of paired data. Yet, the Platonic Representation Hypothesis suggests that models trained on different modalities may converge spontaneously toward a shared representation geometry. But then, do we even need paired examples for cross-modal alignment? Remarkably, we show that paired examples are unnecessary for coarse cross-modal alignment. Our simple Wasserstein Procrustes method with a coarse geometric initialization aligns two disjoint embedding sets by estimating a single orthogonal map without seeing any pairs. Across datasets, modalities, and unimodal models, we show that we can consistently align independently trained representations without pairs, and standard geometric alignment metrics accurately predict when this is possible. Nevertheless, we can naturally benefit from paired examples. In the very few-pair regime, our method substantially outperforms existing ones, while staying competitive with pair-based methods with more added examples. Finally, we demonstrate that the resulting alignments can enable text-to-image generation without paired examples. These results show that independently trained models often share enough geometry to establish cross-modal correspondence with little or no paired data.

Dominik Schnaus, Thomas Dagès, Daniel Cremers, Xi Wang, Phillip Isola
arXiv:2610.09411 · cs.LG, cs.AI, cs.CV · submitted Oct 7, 2026 · updated Oct 8, 2026
abstract · pdf · html · Project: https://dominik-schnaus.github.io/unpaired-rosetta/, Code: https://github.com/dominik-schnaus/unpaired-rosetta

add comment on HN