In plain words: A big language model writes many practice tasks in 93 languages, and a smaller open model learns to compare texts from those alone. With no labeled data it does well on tough benchmarks, and adding labeled examples beats the best on both major ones.
Abstract
In this paper, we introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. Unlike existing methods that often depend on multi-stage intermediate pre-training with billions of weakly-supervised text pairs, followed by fine-tuning with a few labeled datasets, our method does not require building complex training pipelines or relying on manually collected datasets that are often constrained by task diversity and language coverage. We leverage proprietary LLMs to generate diverse synthetic data for hundreds of thousands of text embedding tasks across 93 languages. We then fine-tune open-source decoder-only LLMs on the synthetic data using standard contrastive loss. Experiments demonstrate that our method achieves strong performance on highly competitive text embedding benchmarks without using any labeled data. Furthermore, when fine-tuned with a mixture of synthetic and labeled data, our model sets new state-of-the-art results on the BEIR and MTEB benchmarks.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
arXiv:2401.00368 · cs.CL, cs.IR · submitted Dec 31, 2023 · updated May 31, 2024
abstract · pdf · html · Accepted by ACL 2024
E5/BGE large are an order of magnitude smaller than Mistral-7B. So is this just "bigger model wins" in disguise?
I need to read the whole paper carefully, but this jumped out at me.