LLaMA: Open and Efficient Foundation Language Models (2023)

In plain words: They built language models from 7 to 65 billion internal settings, trained only on text anyone can find online, and released them all. The 13-billion one beat a far larger well-known model on most tests, while the largest matched the best.

Abstract · LLaMA: Open and Efficient Foundation Language Models

We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, et al.
arXiv:2302.13971 · cs.CL · submitted Feb 27, 2023
abstract · pdf · html