about
Python has 189X the dataset size compared to Rust (arxiv.org)
2 points by skp1995 on Jan 26, 2024 | hide | past | pdf | 4 comments on HN

In plain words: DeepSeek-Coder is a set of free-to-use code models trained from scratch on 2 trillion tokens of whole-project code, learning by filling in missing pieces to write and complete code in long files. It beat open and closed rivals like Codex and GPT-3.5 on coding tests.

Abstract · DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence

The rapid development of large language models has revolutionized code intelligence in software development. However, the predominance of closed-source models has restricted extensive research and development. To address this, we introduce the DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens. These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling. Our extensive evaluations demonstrate that DeepSeek-Coder not only achieves state-of-the-art performance among open-source code models across multiple benchmarks but also surpasses existing closed-source models like Codex and GPT-3.5. Furthermore, DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, et al.
arXiv:2401.14196 · cs.SE, cs.CL, cs.LG · submitted Jan 25, 2024 · updated Jan 26, 2024
abstract · pdf · html

add comment on HN

Not a dig at the team, deepseek coder is one of the best coding models out there. What surprised me was the lack of training data for rust 0.08% compared to python which is around 15.12%

Is it just the fact that python and js like languages are more popular in OSS so they get a bigger share in in the datasets being generated for training these models? As a rust developer I would love to contribute towards creating more rust data out there (other than just writing rust code and open sourcing it)..

I think Rust is harder too.

On top of all the essential complexity of coding, Rust adds the accidental complexity of solving a logic puzzle to satisfy the borrow checker. If there is one thing LLMs can’t do it is solve logic puzzles, at least not if you want the right answer.

Thats true, maybe one of the reasons I like rust. But logical puzzle solving implies constraints and that's something LLMs can solve (in a constrained environment with the compiler telling you what's right and what's wrong..) I might be over ambitious about this but to me rust code would seem more solvable compared to say python or raw javascript (which have no such compiler based guidance)
Well that kind of constraint solving is NP-complete and we know a lot about algorithms that can do that and they don't look like an LLM in a loop with a small number of iterations.

I think people don't give garbage collection the credit it deserves for advances in software reuse. If you are creating libraries in, say, C, you face tough problems with memory allocation and deallocation. If the library wants to allocate memory do you just use malloc() or let the application pass a pointer to an allocator? When the library doesn't need a piece of storage can it assume the application doesn't need it anymore?

With garbage collection you can smoosh together N systems into 1 and always know memory management is sound because memory management is a self-contained system and isn't weaved intimately into libraries and applications.

I did a free trial of the Jetbrains assistant and I was blown away by its talent for science fiction chat (only knew one person who was better) but its programming ability was not so hot. It was remarkably good at "linear" tasks where you don't need to understand the relationships between things but it did not succeed when there was any depth in the problem.

For instance it could infer the SQL scheme for my database just by looking at the stubs that JooQ generated and it would make a good try at writing SQL queries. I really wanted help writing this kind of query

https://www.jooq.org/doc/latest/manual/sql-building/sql-stat...

and it could not get it right at all because of the reference circularity. Similarly, it did a good job of explaining code one line at a time but it could not say something like "this is a topological sort" or "this makes no sense at all".