In plain words: DeepSeek-Coder is a set of free-to-use code models trained from scratch on 2 trillion tokens of whole-project code, learning by filling in missing pieces to write and complete code in long files. It beat open and closed rivals like Codex and GPT-3.5 on coding tests.
Abstract · DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
The rapid development of large language models has revolutionized code intelligence in software development. However, the predominance of closed-source models has restricted extensive research and development. To address this, we introduce the DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens. These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling. Our extensive evaluations demonstrate that DeepSeek-Coder not only achieves state-of-the-art performance among open-source code models across multiple benchmarks but also surpasses existing closed-source models like Codex and GPT-3.5. Furthermore, DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, et al.
arXiv:2401.14196 · cs.SE, cs.CL, cs.LG · submitted Jan 25, 2024 · updated Jan 26, 2024
abstract · pdf · html
Is it just the fact that python and js like languages are more popular in OSS so they get a bigger share in in the datasets being generated for training these models? As a rust developer I would love to contribute towards creating more rust data out there (other than just writing rust code and open sourcing it)..