Seems promising. I guess the one catch is that all this extra computation required for self assembly might not be received back in the form of a better system compared to using that computation to better train a human designed neural network.

Designing neural net architectures seems like exactly the sort of problem that neural nets ought to beat humans at handily. I'd be surprised if some kind of self-optimizing architecture wasn't standard in a decade or two.

> ... Designing neural net architectures seems like exactly the sort of problem that neural nets ought to beat humans at handily. I'd be surprised if some kind of self-optimizing architecture wasn't standard in a decade or two.

wouldn't evolutionary algorithms be better suited for this ? evolution is how we got here after all ? perhaps different constraints on the 'state space' can 'guide' the whole process...

There are several evolutionary algorithms that are used to design network topography as well as weights. NEAT is a popular one: https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_t...

There are a few problems with this approach though. For one, there are often many local maxima that a genetic algorithm can easily converge on.

Secondly, the state space is often enormous, and testing/discarding millions of generations of poorly constructed networks is a very inefficient way of sampling it.

These problems are often enough to prohibit GA use.

I suppose that eventually, we'll have the metric being actual "money" that the neural nets try to earn (by mining crpyto, or solving captchas, etc.), such they'd have to pay for their own infrastructure costs. It would then be up to each of them to decide if they'd like to spawn "children", and then decide on an appropriate revenue-share model with them (maybe even pyramid schemes).

i went over these resources, and what strikes me as kind of odd, is the fact that the folks involved in this research / exploration tried to evolve both the architecture and the weights.

i was naively thinking that architecture should be evolved, while weights should be computed just as it is being done now i.e. via back-propogation etc.

> There are a few problems with this approach though. For one, there are often many local maxima that a genetic algorithm can easily converge on.

> Secondly, the state space is often enormous, and testing/discarding millions of generations of poorly constructed networks is a very inefficient way of sampling it.

This is true for the living as well.

If evolution is a response to the environment, then you might say the environment also evolves with the organism, and that a neutral net is merely trying to respond to its logical environment, in a logical way, to maybe repeat (or regenerate) the environment that created it?

> Designing neural net architectures seems like exactly the sort of problem that neural nets ought to beat humans at handily.

Is it? The “easy” tasks for neural nets to beat humans at are things that are harder for humans to do than for humans to provide volumes of high-quality training data by way of which neural nets can be trained.

It’s not clear to ne that designing neural nets really fits that.

The current crop of LLM are better aware of all the state of the art in NN architecture than yours-truly. They know things such as the number of layers used for X, Y and Z, the class of cost function which is popular with a specific architecture, dropout-vs-no-dropout, etc.

That kind of knowledge still needs a human engineer to be leveraged, but even now it seems it's more because nobody has built a harness for the LLM to be able to put together neural network architectures. And, there are already automatic constraint checkers for NN architectures, so even if an LLM hallucinates something wrong, constraint checks could be used to automatically call its attention to some design flaws.

Maybe, but I think humans are better at building a NN architecture.

A NN could design a better wing for a bird, but a human can design a jet engine. While the NN could eventually get to the jet engine, all the computation might be better used to train a human design architecture on more data/more epochs/ more complex optimization

Abstract:

> Biological nervous systems are created in a fundamentally different way than current artificial neural networks. Despite its impressive results in a variety of different domains, deep learning often requires considerable engineering effort to design high-performing neural architectures. By contrast, biological nervous systems are grown through a dynamic self-organizing process. In this paper, we take initial steps toward neural networks that grow through a developmental process that mirrors key properties of embryonic development in biological organisms. The growth process is guided by another neural network, which we call a Neural Developmental Program (N DP) and which operates through local communication alone. We investigate the role of neural growth on different machine learning benchmarks and different optimization methods (evolutionary training, online RL, offline RL, and supervised learning). Additionally, we highlight future research directions and opportunities enabled by having self-organization driving the growth of neural networks.

Except our brain wasn’t built by meta-neural networks.

Its development is governed by a DNA algorithm.

Our DNA program is changing slowly by evolution, but at any given time, the design we have is one design intended to work well across all survival problems.

Each of us doesn’t have a brain architecture that our body custom designed for some different set of problems or data.

Not critiquing the tech. Critiquing some of its rationalization.

Our brains indeed are not quite like neural networks, but evolution isn't uniform for everyone, including in terms of brain development. There's also sexual dimorphism and epigenetics to consider. A lot more haphazard than custom designing, that's for sure.

But the expression of the brain as it grows is very dynamic, influenced by many factors outside the genetic, including itself, self-referentially.

You could pretty easily raise one twin as an imbecile and the other as competent, purely by the means of changing their sensory inputs during the phase of brain development. If you had no scruples, of course.

Except our brain wasn’t built by meta-neural networks.

Its development is governed by a DNA algorithm.

Our DNA program is changing slowly by evolution, but at any given time, the design we have is one design intended to work well across all survival problems.

Each of us doesn’t have a brain architecture that our body custom designed for some different set of problems or data.