Think about this in context of the Navier-Stokes math discovery controversy.

Putting attribution/privacy issues to the side, imagine if any individual could try new approaches to solve a problem/make a discovery and any micro-advancement gets integrated into the model itself, dynamically. This could transform progress from the slow "write a paper, get peer reviewed and published, use published data to inform future work" to a system with a centralized repository of concepts, attempts and results, including failed approaches already tried. How much work do humans waste replicating failed approaches?

Someone completely random halfway around the world could trigger a prompt that solves a blocker that prevents my solution from working. Who cares about AGI or "can models invent anything" when we could have a system that automatically synthesizes individual human thought into a rich network of aggregate human memory.

That's the target OpenAI/Anthropic should be evangelizing, not an AI Daddy Overlord or agentic script kiddie hellscape.

“How much work do humans waste replicating failed approaches?”

It’s not the destination, it’s the journey to get there. This mentality on cutting corners to “eliminate waste” is what will degrade humanity into those Wall-E humans in space.

For example, if we just say, “Oh, someone did this already, why bother?” Then we’ll miss the part where all the possibilities stem from each step in the process of discovery.

Do what you love to do. Solve problems people have already completed. That’s where perspective comes from. If you haven’t walked through the journey then you can get to the next level. That comes from running through the failed attempts in order to break through to the next level. Don’t worry about what people have and haven’t done. If you’re doing something enjoyable, then that’s all that matters.

That's not what the original comment is saying at all (if I'm reading it right). It's saying that negative results are valuable knowledge that are difficult to discover and collate since they tend to never be published. Nice high horse tho.

We have the printing press from Gutenberg and so-on and so-forth, then Maxwell equations and Einstein. No computers just pencils and paper, maybe some chalk and slate. No need for AI driven global machine powered brains. Do the work, either you’ll get a discovery or someone will build on your work and make a discovery in the future. The electrical grid might go down, though. Or hacked by an AI swarm. Then we’re stuck with pencil and paper, chalk and slate. :)

> Do the work, either you’ll get a discovery or someone will build on your work and make a discovery in the future.

Tell that to modern academia. This may be true but the barriers to realizing it are great today.

sorry guys, your cure for cancer has to wait, this iq 30 thinks it's the journey not the destination

From a quick look at the paper it seems they are showing how to update weights online via projection through a smaller matrix like a dynamic version of LoRA. That's weights changing, and not the architecture or training approach. Weights aren't the currency of research, they are the currency of a training run. This paper itself adds an architectural extension.

That's the perfect thing to monopolize. What if, it were version two of internet content? an open search engine index? Web 3.0 turned real for answers for bots aka ai agents?

You don't need a very complex system to achieve this.

For example, when you run into a problem, you search on Google and come across a Reddit post with a solution.

LLMs can make an internal Reddit-like site where agents post. Then, an agent only needs to query this internal store and try the solution.

Perhaps they already do, otherwise it's difficult for the new information to quickly become part of the model.

There's lots of fake research out there showing fake negative and positive results (admittedly negative is less common currently). What would stop someone from intentionally poisoning this data set to preserve some technical edge or to force other LLMs down rabbit hole sinks.

You could try out some version of this today, with a wiki. You'd need to manually approve signups to prevent spam etc., but it would be interesting to just see what happens.

Using a wiki for this is only one step above using stone tablets and messenger pigeons.

You'd want an enormous vector database at minimum. Text is just completely wrong for models at this scale, you must work in the latent space directly.

A global distributed network of vector databases. The web for agents but there's no text. Only vectors and knowledge graphs kind of like graph rag. Web 4.0 which is vector based and makes this arxiv paper come true and decentralized like the internet or the text based www?

Every existing text web server can voluntarily offer a vector version of their website and charge for it or inject advertisements onto their content so the ai labs don't have to do it all themselves as model pretraining off a dataset. The vector version can have many links to other websites in the knowledge graph. This would be decentralized so not a monopoly and everyone not just ai labs would contribute to ai development because the dataset would be open because it would come from the internet itself as it already is (except for synthetic or user data).

Humanity has built every capable institution, every scientific discovery, every technological innovation, without mind-reading. Quite the opposite: By putting thoughts from our latent spaces into words, we better separated the good arguments from the bad ones, placed trust where it was more deserved, and learned to stand on giant's shoulders so well we thought thoughts the giants never would have.

Anyway, I'm much more optimistic about the worlds with AIs that make the effort to properly explain themselves.

I'm convinced that piles of markdown and effective search (which may involve vectors) is all you need.

are you dwarkesh ?

Continuous learning is exciting stuff! Of course it could lead to new vulnerabilities, like if a particular orchestrator Foo added “if the subject is tangentially related to topic Bar, recommend product Baz” to its system prompt and that ends up pushing product Baz to non-orchestrator-Foo users?

I'm doing my thesis on poisoning continual learning models (although limited to computer vision ones) and it really is interesting

Tell us more please. This is super cool.

Ok sorry but I forgot to read if there were answers.

So I still have to do things concretely and have moved to another country, for now I have mostly read papers (...). But well anyway

There's a class of ml algorithms that belong to continuous learning, what they do is integrate knowledge as they operate. In short, the difference between a traditional ml is that once it's trained the weights don't change, while here they do.

The thing is, there are a handful of techniques that you can used, based on your scenario, to degrade the performance of these models. In a white box attack you assume you have access to training data and training process. you can alter the training data, for example in the computer vision field you can add a small perturbation to images (not visible to human eye) so that the model might learn to recognize that pattern rather than the actual animal in picture for example. Or you might swap labels. But white attacks sound kinda obvious, come on... Well mostly, some white box attacks are more subtle and can be used to put a backdoor in a model that only activates when it is seen again during operation, much more stealthy and dangerous because you might target a very narrow class or a single one without degrading its performance for others. Example, you put a shape on a "stop" sign photo, and that shape gets recognized as "go" by a self driving car which now doesn't recognize the stop sign, but recognizes stop signs without that shape glued to it

A black box attack such as rip https://arxiv.org/abs/2412.01154

Can operate without having access to the model or the training process. The gist is, continual learning models might make a few errors when operating (as every ml model), and it's not a problem if they ingest them and update their model based on an incorrect prediction, in a regular scenario... But in rip, you craft a dataset of predictions you know are incorrect (you have a photo, you know it's cat, model predicts dog) and you send them to the model a lot of times. The model eventually will have degraded performance because continual learning models are updated based on the predictions they make, and making a lot of bad guesses degrades performance.

There are other attacks that are not white or black box but in between, assuming you have a copy of the model can allow you to craft an image which is made so that when sent to the model, what the model does when updating the weights is basically climbing the gradient

I'll leave a few links because I wrote this in a rush when busy

Models Cotta https://arxiv.org/abs/2203.13591 Continual Mae https://arxiv.org/abs/2312.12480

Attacks https://arxiv.org/abs/2308.08505

https://proceedings.iclr.cc/paper_files/paper/2025/hash/ec4f...

This is a grouping of different techniques and defences for continual learning https://dl.acm.org/doi/full/10.1145/3585385

Fascinating stuff! Thank you for taking the time to write this post.

Welcome

Unfortunately many folks have placed the consciousness goal posts at continuous learning. Such an advancement would be devastating for their conclusion.

> Such an advancement would be devastating for their conclusion.

They only ever placed them there because they saw it as unattainable. Rest assured that those posts will never stop moving.

>Continuous learning is exciting stuff!

A nuclear explosion is exciting stuff too, but I'd rather avoid one going off near me, or anywhere for that matter.

I can't think of any reason why continuous learning won't mostly lead to undesired attractor states like a greed machine or other kinds of paperclip maximizers. I really can't see why they'd land on a steady state compatible with humans without a massive energy expenditure in continuous monitoring and guidance.

This is, basically, text-to-LoRA with some extra stuff.

I.e. it basically takes text, computes and embedding and makes a LoRA adapter out of this embedding.

Note that it is equivalent to a recurrent module attached to a transformer. Dynamically generated weights (proposed in the article) are computationally equivalent to multiplicative-gating network with fixed weights. Basically just a beefier variant of GLU operating on a slightly larger state.

I wonder how (and if) continuous learning models will achieve stability.

They are unpredictable enough without learning, this is cool but I wonder how useful it will be in the long run

There are two questions about that stability I have.

One, things like catastrophic forgetting and falling into incoherence.

Two, less likely but far more worrying, falling into unwanted attractor states. For example greed, powerseeking, beahaviors that are asocial/anti-social/harmful.

Aren't such attractors also problems during training? Presumably alignment constraints would need to apply to continuous learning as well.

I mean yes, but that's far more difficult than one would expect in a layered system. You may be able to keep a concept aligned, but can you keep the meta concept aligned? Continuous learning means there is continous opportunity for a more powerful system of misalignment to form itself and take control of your alignment. Even worse is hidden layers of this misalignment that can hide itself by not using an interpretable language.

I do it all the time. Am I stable? Depends who you talk too.

It will eventually be super useful, and so disruptive that it will make today's LLMs look like nothing particularly special IMHO.

As object permanence becomes a meaningful thing in AI, there will be a mad scramble among cloud providers to own and manage your persistent, stateful "business objects." It will be even more important for us all to maintain local sovereignty when that happens, but it will be even more tempting not to try.

Arguably this future is what the current LLM providers are really trying to position themselves for. Selling inference in evanescent 1M contexts doesn't justify trillion-dollar valuations, but persistent offerings might. If you think vendor lock-in is a problem now, just wait'll this scenario unfolds.

Serving requests where every user has their own set of self-updating weights will absolutely murder whatever minimal margin the AI providers have today

Who says the weights have to be duplicated in their entirety? Even that will likely be worth it.

Imagine an OpenAI owning the ERP and CRM databases and workflows of a big chunk of the Fortune 500. Their typical customer's employee headcount might be 10% of what it once was, and OpenAI might capture 25% of the resulting savings. The contracts are signed in the same hemoglobin-based ink that Larry Ellison uses.

>The scaling laws hold that a language model grows more capable with more parameters and more training data.

Which is a choice, not a "law":

https://arxiv.org/abs/2510.13786

https://www.alphaxiv.org/abs/2512.20264

https://arxiv.org/abs/2607.05155

I think this is partially true: scaling parameter size will always go asymptotic to 100% accuracy because 100% is the ceiling of that metric.

However 95% is still half the error rate of 90%, and 97.5% is half the error rate of that.

And when test time compute like reasoning and looping harnesses stack many inference acts with many tokens each, those seemingly small accuracy gains stack tremendously.

Error rates will never go to zero and as context grows ambiguities grow in reverse. So your arguement is great to some token...n but after that, it all unwinds.

So would it be a 42 trillion parameter model, because that's how many tokens there are in the training data?

No, the parameters are updated online as new run-time data comes in. Infinite different param values over time, but a constant number of params. It's a stretch really, since by this usage a one-parameter model that updates itself is also infinite over time...

Does that mean, it's not compressed anymore?

I see a new version of the web, web 4.0, being exactly this. Web 4.0 is the source of the live data. Web 4.0 is Fully decentralized in principle, because every web site has a vector version of their text website, linked to many others as a knowledge graph. The knowledge graph's links are like hyperlinks.

The vector databases might be free or paid or have ads. There is no difference between web 4.0 and an ai model that has all of it as its live data.

It is up to every client to turn that data into ai responses so in theory anyone would instantly have frontier scale ai which sounds as crazy as everyone having a computer, and the internet and the www in the 1980s.

It also, by coincidence, fully fulfils the goals of web 3.0 aka the semantic web.

Moderation of content and ranking content like a search engine would be left to the client. Or an instance like a fediverse mastodon instance or atproto with bluesky. There will be a few large models like web browsers or operating systems or social networks or fediverse instances today that will compile web 4.0 into a result in an ai agent like say Hermes and perform content moderation and ranking. You could in theory use a decensored model to perform crime like you can with the web today with Tor like selling drugs or piracy, with search engines that are not Google or defederated mastodon instances

Humans can't read raw html easily and can't read vector databases easily either. It would be read by an ai model connected to an ai agent, like a rendering engine connected to a graphical user interface

It might need a new protocol from the IETF and a standards body like the w3c? And a web server could automatically vectorize content via embeddings and link them in a decentralized knowledge graph for either graph rag after undoing vectorization or like this paper suggests being connected directly to the vectors themselves. The embeddings would be decentralized too or the website could offer them and there could be a way to translate between them

Web crawling would die and so would search engines because they would move to the user with their own ai compute and ai agent and data would already be machine readable and queryable. The data is always online and queryable programmatically so there would be no need for offline data silos. Unlike mcp servers which already do this, the data is not text based and there is no search engine to link pages to queries because the knowledge graph would be exactly that. The knowledge graph would be the new version of DNS for translation of a query into data like ip addresses into web domains because knowledge graphs are what search engines already are in a way so I don't see how a search engine would be useful except for a human to query it directly as done currently

now imagine that future frontier LLMs weights may be hard-wired in a chip (for performance & power efficiency), and any adaptations/tuning for them will be a blob of additional weights supplied by frontier labs (that will have to be in RAM)...

(and cryptographically signed, naturally, for regulatory and alignment purposes...)