My crude reaction to this piece is that it feels like they are describing a technique for finding a local maximum for any coding problem. It is an unsurprising result but it seems like it would be hard to avoid getting stuck in a low valley without human intervention. I also believe that for any deterministic program, the best human written program will converge on the optimal solution given the real world matetial and temporal constraints with or without LLMs. I am not sure if LLMs will be in wide use in 10 years, but they will certainly help us humans write better code if only to avoid getting stuck in the traps they introduce.

Hmmm, I mean there is a more general question: in the space of algorithms, how continuous are the paths between different solutions?

For instance, if you start with bubble sort, I don't think it's at all clear this algorithm can take small finite steps to turn into improved to merge sort. This is the low valley you are talking about.

This also aligns nicely with the results in the paper. GPT-4 improves, so it's perhaps good enough to make larger steps or transformations of an algorithm, whereas weaker models don't.

The real interesting result would be if they improved an algorithm beyond what people have been able to do. Otherwise it's just interesting and a peek into the future..

For the local maximum issue, there's plenty of optimization algorithms that deal with this issue which I think are perfectly viable for this case scenario.

Vaguely unrelated - the tech youtubers in LLM space are becoming really fast. Heard about this paper there before hn.

Normally they're a good day or two behind

could you please provide us with some recommendations?

1. This search space is too large to hit anything by firing shots in the dark.

2. Results will either be completely redundant or ethically questionable.

3. The resources used for optimization should not exceed what performance may be gained.

There is no route around full comprehension of this problem before you can solve it. It is a philosophical event horizon.

People routinely solve quadratic assignment problems (perhaps not provably optimal, but finding good solutions within a small bound of optimality) where the search space is greater than 10^3000 [0], using relatively simple heuristics like simulated annealing.

[0] e.g. any of the QAPLIB instances in Goh et al. Proceedings of the 2022 Genetic and Evolutionary Computation Conference

> 1. This search space is too large to hit anything by firing shots in the dark.

2. Results will either be completely redundant or ethically questionable.

3. The resources used for optimization should not exceed what performance may be gained.

There is no route around full comprehension of this problem before you can solve it. It is a philosophical event horizon.

My sweet summer child

http://www.incompleteideas.net/IncIdeas/BitterLesson.html

That links to a spiritual quest described in the language of information theory. It is looking for what makes us different from machines instead of what makes us similar. It is an ego talking about predictions that did not pan out.

Rookie mistake.

That's not a spiritual quest, that's an empirical observation of what worked and what failed. Expert systems without the guidance of statistical learning has gone nowhere. The 90s Japan is calling and they want you back to wage war against the AI winter and make their fifth generation computing project great again.