One thing that is clouding this discussion is that most people are mixing up a lot of different characteristics that animals like humans have as if they were all the same thing.

So for many people they don't really distinguish between things like "reasoning", "self-aware", "conscious", "alive", "sentient", "intelligent", "has world model". They also don't distinguish between different types or varying levels of cognitive abilities.

It seems clear that high functioning LLMs must have some type of world model. But that doesn't mean it's necessarily exactly the same type of highly grounded model that a human would have, especially if it was trained on only text. It might be less rich or different but still quite useful.

Another example, LLMs clearly don't have the same type of fast adaptation in a realtime 3d environment that animals have. (That's not to say that they can't mimic it in some rough ways).

But if you don't really break all of this stuff down carefully in your head then it can be hard to accept that LLMs are doing anything interesting. Because in that worldview, it's all the same thing, so they have to give the LLM all of the other characteristics at the same time.

It's not the first time this has been a problem to accurately specify either. Throughout history animals proved nigh impossible to classify in these terms and were considered so many things, from something that's intertwined with nature as a hive mind of sorts, to complete automatons by Descartes, and later to beings that are mostly completely identical to us by Darwin.

I suppose it's also an interesting juxtaposition, animals can do everything but talk, while LLMs can do nothing but talk.

So all we have to do is plug a USB cable from a device running an LLM in to the back of a monkey.

For the most part our daily life doesn't depend on contrasting minimal pairs of these attributes. Humans, if they have one, tend to have all.

AI entities, may exhibit one, or none or sometimes some and that maps poorly to our prior experience with reasoning entities.

How would self-aware, non-conscious agent respond?

How would a reasoning but non-intelligent agent respond?

Well said.

Also, the paper’s use of “world model” (instead of simply “model”) doesn’t help clarify the situation.

I do not see how referring to a world model as simply a "model" could possibly provide clarity.

LLM literally stands for Large Language Model, it has a 'model' in it by definition.

Would we think of it differently if we called it LLC - large language correlation?

Is-a vs has-a relationship.

Fair. Yeah, the word 'model' is highly overloaded. To me, 'domain model' would work better.

I think the term "world model" is a highly overloaded term. What does it really mean in your opinion?

A reasonably precise although not universal definition could be to take the data points from the study in relation to the use of those terms in the study.

I think the main idea though is that there is some structure in the memory that resembles a structure in the real world to some meaningful degree. One aspect of this is composition, the other is how closely it maps to the real world.

If it were literally just a lookup table where the indices did not correspond to physical proximity then it would definitely not be valid to say that this was really a world model in that context.

Some understanding of the laws of nature, as we know it.

If an LLM is trained on just text, then it's internal model can only be based on such text.

I think the 'conceit' of LLM as AI advocates is that this model is just as (or at least nearly as) our own internal model - that the 'simplest' model to match the text is indeed our 'real' model that matches reality as a whole. A bit of a stretch surely!

I’m no expert, but the word “just” is tricky. You could say mammalian brain is trained on “just” nerve impulses.

you could, but you’d be wrong. (there are many other signaling modalities!)

LLMs are a redemonstration of the lesson of sun worship: apes are easily fooled.

If the output is indistinguishably 'world modelling', so must its mechanism be.

There is no 'world' model in a hashmap from questions to answers about a world. There is only the condition that, given those questions, any old ape would accept those answers.

Science is required to determine whether a system has properties, such as a rational ecology; ie. a genuine mechanism of environmental coordination.

No engineer, no statistician, and certainly no VC HN Hype Man can determine-so. The nature of reality is no up for Big-O equivalences.

Systems have properties in virtue of their nature, not in virute of what I will pay $20/mo for

Sorry, that's just surface-level tripe. You could make the same dismissive comment about any new technology. You're not saying anything interesting.

The claim is that you cannot determine whether a system has a 'world model' through inspection of it's abstract token inputs/outputs.

But that's not all this paper is doing. So why bring it up?

My comment is a reply to another commenter, not to the paper

Nice try LLMy

There was an interesting thought from a Microsoft researcher in an episode of This American Life. He had been given early access to GPT4 and found that it had gained an understanding of gravity and balancing. 3.5 would fail to describe a safe order for stacking 3 eggs, a bottle, a book and a nail. But 4 would give robust answers with logical justification added to each construction step for the tower.

The researcher remarked (paraphrasing) “The models are built to predict the next token. At some point the only way to improve token prediction is to start modeling the world and building an understanding of how things work.” Since I heard this I see it everywhere with LLMs. Load up a 30B parameter model on your computer and toy around with it. When it fails I now see that it’s reached a concept it doesn’t understand and is in more-or-less Markov Chain mode. Give the same task to GPT4 and it proves it’s modeled the concept appropriately because it can pull the right words out of the ether.

> Give the same task to GPT4 and it proves it’s modeled the concept appropriately because it can pull the right words out of the ether.

I take your point, but we should be careful when using phrased like “modeled the concept appropriately” here. That implies correctness.

There are many ‘concept models’ that could work well enough. Unless we can inspect the concept model directly (inside the LLM, somehow!) we are left to reason about how the concept is represented via statistics over a set of interactions. So, if, over the course of many interactions, it seems that a model is saying reasonable things we can have more confidence that it has achieved a concept model that is “good enough”. But such an assessment is always in relationship to the test protocol. Claims to generality should be done carefully.

In some ways, my standard here is probably not that different than a rigorous human educational assessment. I’m not talking about standardized tests, I’m talking about adversarial challenges like one would hope to see during a Ph.D. defense.

Yeah I’m setting the bar at the level of a child’s understanding. Because smaller models are well below that, just picking tokens at random. That’s how misunderstanding manifests with LLMs. They don’t say “I don’t know”, they just go off the rails.

The quality bar for different use cases might be such that an understanding of a system on par with a 6 year old is isn’t useful. But right now we clearly have two clusters and I’m drawing the boundary where it’s most useful.

> I take your point, but we should be careful when using phrased like “modeled the concept appropriately” here. That implies correctness.

All models are wrong, but some are useful.

Perhaps we should say "modeled the concept usefully".

> In some ways, my standard here is probably not that different than a rigorous human educational assessment. I’m not talking about standardized tests, I’m talking about adversarial challenges like one would hope to see during a Ph.D. defense.

Usefulness is very contextual obviously. Newton's laws aren't entirely correct but are still useful.

This is the same reasoning that Ilya Sutskever uses. That next token prediction objective causes the model to learn world models. Because being able to predict the next token requires a lot of the knowledge/model of the world.

The Twitter thread is worth looking at. Very Fascinating

https://twitter.com/wesg52/status/1709551516577902782?t=3b2F...

Seconded, it's a good summary with graphs and commentary. Also nitter to avoid the login wall: https://nitter.net/wesg52/status/1709551516577902782

Thank you for your service.

I don't understand what the colors represent and I can't read the thread. Can you summarize?

You should be able to read the thread here https://nitter.net/wesg52/status/1709551516577902782

The colors in the first tweet just represent different territories(continents)

I probably saw this too late to get a reply but: Yes, I understand that part. I don’t understand how that connects to the underlying data. Where did the x and y coordinates for each point come from? Where did the territory value of each point come from for that matter?

I think we have a new member to add to the group: Lies, damn lies, and statistics… and now neural activations. How exciting!

This reminds me of all those fMRI studies that look at brain blood flow before the dead salmon experiment came out.

Haha, I have no idea how I was not aware of the dead salmon experiment prior. This article [1] has a nice overview for anybody else in my shoes. Thanks for the reference.

[1] - https://blogs.scientificamerican.com/scicurious-brain/ignobe...

First of all that's an excellent article. But also:

> Some people like to use the salmon study as proof that fMRI is woo, but this isn't the case, it's actually a study to show the importance of correcting your stats.

I.e. be aware of the "look elsewhere effect".

Perhaps the most important public discussion, at least among technical people, is making sure we really understand the experimental setup.

(I admit I could be projecting my lack of understanding; I’m not rock-solid on the experimental setup yet.)

Still to me, it matters if the linearity been described can be observed directly _in_ the model. My understanding is that it is not; rather, it is tested with an additional model. The term probing can be misunderstood; it isn’t -just- an internal probe. There are ‘derivative’ models as well.

This raises questions about what we mean when we we are talking about internal representations! In my view, a derivative of an internal representation is not the same thing.

(I welcome people who can educate, correct me, or even reframe the topics. The above thoughts are subject to change.)

It's important to note that these probe models are very simple - they're "trained" by just doing a linear regression between the hidden activations and the desired output. This means that the probes can barely do any computation themselves, so if they work at all this is a strong indication that the signal they predict was already in the hidden activations.

For even more proof, see "Figure 5: Space and time neurons in Llama-2 models" for single neurons in LLMs that already encode this information, without even having to use a probe model to extract it.

Thanks. After some additional research, I tend to agree.

One thing I learned of note is that while a _positive_ result (i.e. a probe showing a relationship) is proof that a label is encoded somehow; a _negative_ result does _not_ prove that the network under test does _not_ encode the information _somehow_. (The information could be encoded in more complex ways that the probe does not discover).

Much simpler models like word2vec also showed linear embeddings of geography in this way.

source?

It was something like

    Paris - France + Japan = Tokyo

I know that's not a source but you could also just google "word2vec", as I recall most of the explainer blog posts had similar examples.

That gets "analogies" rather than "spatial awareness" per-se.

You put quotes around spatial awareness but the commenter being referenced said geography.

So it's good at SAT type questions?

It's not "good" at anything, it's a set of word embeddings. You have to apply them in some context using a feedforward model of some sort for them to be used in some task.

A lot of HNers were so adamant that the LLMs understand absolutely nothing and that these models are just predicting the next most likely word.

I think with this paper it becomes clear that the adamant denial was just human bias talking. LLMs crossed a certain line here.

You would think people would be amazed or in awe or react in fear at technological break through a and they often are. The weird part for LLMs was the delusion. Many people were in denial that it was any sort of breakthrough.

My only experience so far is with chatgpt, and I am not an expert in AI, or even just in LLMs.

With those disclaimers, my interactions inform my opinion that the LLM behind chatgpt has no internal world model. It shows no understanding of basic facts and makes very silly mistakes very easily. I have my bias, like anyone else, but in the case of AI in particular, I should say that I don't think there's anything especially magical or sacred about conscience and the human brain (or the brain of other animals, for that matter), and I'm sure it must be possible to arrive at different forms of intelligence starting with the "inanimate" building blocks of hardware and software, but what I've seen so far in LLMs doesn't make me support your statement that they've crossed a certain line. In fact, I'm very much let down by the experience and I'm afraid once the hype goes down, we may be in for another AI winter.

As I said at the start of the comment, I'm well aware I'm not a subject-matter expert. I'm open to being wrong. I just wanted to point out that not all of us unimpressed are in denial. I'd love to be in awe at an AI breakthrough, and I kind of feel I will be in awe sometime during my lifetime, but LLMs are not (yet?) that for me.

Edit: s/are not in denial/are in denial/

Current LLMs likely have multiple world models with varying qualities. Good prompts are required to activate suitable models for each task.

1. Did you use GPT-3.5 (free version) or GPT-4 (paid one) to form the judgment? Their performances on harder tasks differ significantly as shown in https://openai.com/research/gpt-4.

2. Have you tried adding "Please think step by step" to some harder requests? This simple phrase gets most current LLMs to perform significantly better. It's a bit like asking students to show their work, which forces them to think more clearly.

Current LLMs, without additional mechanisms, tend to be perform like a drunk or sleepy human, i.e. using mostly intuition or System-1 (as defined in "Thinking Fast and Slow"). The prompt such as "think step by step" asks it to think more in the System-2 style. (There are other techniques which get them to perform even better still.)

I think of current LLMs as a very-well-read, but often sleepy intern who needs strict instructions, feedback, and sometimes extra training if you want them to perform well.

As stated in another response, I'm using 3, so I'm probably missing some good stuff.

I have tried adding things like "please think step by step" (quite literally that question actually), and also "please make sure you check the facts before answering so you don't include non-existing arguments" (when asking about a cli tool that takes arguments), but I didn't notice a significant improvement.

I like what you say in your last line, though I think a big challenge is that a very-well-read intern got to be very-well-read for at least two reasons: 1) they like to read and can do that a lot (LLMs can do this, not saying they like, but saying training them is analogous to someone reading a lot), 2) they went through a curated list of reading material. My experiences make me think part 2 is a weak part of chatgpt.

I still think LLMs are not the way towards an interesting AI. I don't know why we're insisting so much on natural language. I mean, I can understand this for simpler tasks like a support chatbot, but I wish there was (maybe there is?) good research on building an AI that is not based on human language, since it's one of the worst mediums to communicate with rigor.

I don't think baseline ChatGPT si capable to "please check the facts before answering so you don't include non-existing arguments".

People are already asking you if you have tried <insert version here> but this pretty much echos my experience and I have used both free and paid versions of ChatGPT (3.5 and 4 respectively) as well as the GPT4 API directly. I had a similar experience between all three of them, although the GPT4 based versions were certainly better in terms of output quality. In my opinion it seems pretty clear that this is a fundamental limit of the architecture and all the RLHF and increase in model sizes wont change that.

It's so fascinating going through previous GPT-X threads. You can almost everybody making the same kind of extrapolation mistakes you're making. People just don't seem all that interested in revising their model when it's obviously wrong.

I really don't understand replies like this.

This is just the beginning. It is a tiny, tiny view into this new technology.

Nobody with an ounce of intelligence looked at the first iPhone and thought "ah well it's just another phone". It's the potential of this new technology that is so impressive.

How can you not be impressed by this? I am literally incapable of empathizing with your perspective. All I can think is that the world is going to pass you by.

I'm not sure I follow how you get from me not being impressed by the LLMs I've interacted with (you may be incapable of empathizing with that perspective, but that doesn't make me less unimpressed at them) to some hypothetical people thinking the first iPhone was just another phone, and therefore not having an ounce of intelligence.

I'm not worried about the world passing me by, I think LLMs and similar technology will increase, not decrease, the need for people who can troubleshoot and fix live production software. I mean I do have my passive income if I'm forced into early retirement, but I'm not fearing that to happen soon.

Mind you, I did say I'd love to be in awe at an AI breakthrough; I'm not a luddite. I just don't see anything in LLMs to impress me, and on the other side, I do see a lot of hype behind it.

I felt similarly to you with cryptocurrency but then not much improved in the 10 years since. Maybe LLM's will be different.

I personally didn't have that "ah ha" moment with crypto/blockchain, but I understand your point. to me AI is different - it's current iteration is useful, today. just this year alone it has improved at an insane pace, so I'm excited to see where we are in a few years.

True but in those 10 Years AI has been moving at a rocket pace. Draw the trend line.

Have you used GPT-4?

No, I've only used GPT-3 so far. Should I be excited about GPT-4? :)

Examples of things I'd file as silly mistakes:

- mentioning non-existing settings when asking it about a particular software.

- responding to roughly "give me a way to list all rds clusters that are running a version that will reach EOL within the next 12 months" (which I know is almost impossible to do without scraping, as the EOLs are not returned by the API, but it was worth trying) with a query that compares the version with < 12.0. It did acknowledge the error when I asked "won't that just compare the version number" and, interestingly, it then did add to the response that the API does not provide EOL for versions, but the first response was IMHO bad in a very basic way.

- providing wrong steps in an answer to "give me steps to do X", wrong in a way that would cause the process to go very bad, and then when I point out the problematic step, it responds with a seemingly random alternate step that is also wrong. I guess this part is the one that is less bad, because it loosely reminds me of people who are bad at reasoning and just give back random answers to a logic or math question.

I must say it is surprisingly good (though it will still make up facts at times) when answering questions about elisp though.

Paraphrasing Hofstadter, I'd say if I'm asked about the intelligence of chatgpt using gpt-3 I'd say it's slightly better than a thermostat or a mosquito, but not much more. Furthermore, my (admittedly not very valuable to others) gut feeling is this is not the path to get to what I'd consider an intelligent being, mostly because in my interactions, it makes lots of bad mistakes, and none of the good ones (here I'm using good and bad as qualifiers for mistakes based on my experiences as a parent, and as a mentor to junior peers, where a good mistake is something that lets you know they're on the right path even if they haven't arrived at the right answer/place yet). Try asking it about analogies, for example ("what's a good analogy to explain X to someone new to it"), the results I've gotten are underwhelming, when not just plain wrong.

Edit: s/fall/file/ Formatting.

GPT-4 should do significantly better, but still worse than expert humans, on the tasks you mentioned. They would also benefit from good prompting, such as those in

- https://lilianweng.github.io/posts/2023-03-15-prompt-enginee...

- https://help.openai.com/en/articles/6654000-best-practices-f...

Not a large percentage of humans would be able to do the tasks listed above without significant experience or training either. Average humans are also not great at reasoning.

Thanks, I'll wait for it to be available (I'm a casual user and I'm not the one setting this up or paying for it, if someone is paying for it, so I'm not in control of which version I use).

I'm in full agreement about humans and reasoning, btw. I just don't think chatgpt (with the version I've used) is anywhere on the same league as the worst (way below-average) humans either. I do think it's quite useful as a writing aid. You do need to review what it gives you, but you'd have to do that even if you were relying on a human writing aid.

Try Bing chat, it has a GPT-4 mode, which you have to look for but it’s there and free.

> No, I've only used GPT-3 so far. Should I be excited about GPT-4?

The difference is extremely dramatic. Any experience with 3 or 3.5 is completely irrelevant for 4.

There is a clear difference, but "extremely dramatic" is obviously hyperbole.

(I use 3.5 and 4, both in the chatgpt interface and via API.)

GPT-4 is a considerable amount more accurate and intelligent than 3.5. There are countless anecdotes on the internet like yours of the foibles of "ChatGPT", which when asked reveal they are only using the free version. For reference, GPT-4 costs 20x in the API as GPT-3.5 Turbo.

I have made similar mistakes to all the ones you mentioned. Do I have an internal world model?

Paraphrasing hofsadter?

You realize Hofstadter is not delusional about LLMs at all. His view point is completely opposite of yours more logical and rational and he doesn't need to use chatGPT 4.

Hofstadter is not a normal person because he is extremely unbiased. chatGPT basically got him to do a 180 on everything he thought and talked about in every book he has written. He flipped, you subscribe to his viewpoints, and you haven't flipped.

Keep in mind he criticized early versions of gpt3 and gpt2. chatGPT changed the game.

https://www.nytimes.com/2023/07/13/opinion/ai-chatgpt-consci...

https://www.lesswrong.com/posts/kAmgdEjq2eYQkB5PP/douglas-ho...

He literally said something along the lines of his core beliefs are collapsing. Paraphrasing. I kid you not. It's abnormal. It's very rare to find a person that can change his core beliefs. Extremely rare. Let's be frank. You are not that person. What will happen is you will reinterpret reality such that it fits your core beliefs and you won't know you're doing this. It's very predictable: Using chatGPT 4 will not revise your opinion on AI because you don't want to face the truth. chatGPT 3.5 is already enough to see how we crossed a certain line here you can't admit it... so your experience will be the same with gpt4.

Here's a quote from one of those links:

"Of course, it reinforces the idea that human creativity and so forth come from the brain's hardware. There is nothing else than the brain's hardware, which is neural nets. But one thing that has completely surprised me is that these LLMs and other systems like them are all feed-forward. It's like the firing of the neurons is going only in one direction. And I would never have thought that deep thinking could come out of a network that only goes in one direction, out of firing neurons in only one direction. And that doesn't make sense to me, but that just shows that I'm naive.

It also makes me feel that maybe the human mind is not so mysterious and complex and impenetrably complex as I imagined it was when I was writing Gödel, Escher, Bach and writing I Am a Strange Loop. I felt at those times, quite a number of years ago, that as I say, we were very far away from reaching anything computational that could possibly rival us. It was getting more fluid, but I didn't think it was going to happen, you know, within a very short time."

The guy literally flipped bro. He literally just admitted what he wrote in his books are wrong. The books may not be wrong but he believes it now. Degrading hundreds of pages of his own writing and his own core beliefs is a display of incredible rationality, scientific thinking and lack of bias.

You seem to be convinced I'm scared of something and in some sort of denial. However, I did say "I don't think there's anything especially magical or sacred about conscience and the human brain (or the brain of other animals, for that matter), and I'm sure it must be possible to arrive at different forms of intelligence starting with the "inanimate" building blocks of hardware and software". I just don't see that in LLMs, or at least not in the ChatGPT version I've interacted with.

I very much respect Hofstadter and the fact that he changed his opinion on GPT (I'm based on what you're saying, I have not read the links you provided yet). I'm not sure that means he changed his core beliefs though, even if he said so. At least, my understanding and interpretation of his work is that we're not the only substrate that can develop "strange loops", and that this is not limited only to organic substrates/living beings. He may have changed his view in GPT, but if he was true to his words, he must have always been open to software being able to develop a being.

I very much think AGI is possible, but:

- I don't think it will happen through LLMs, or at least not through LLMs alone, and

- I don't think it will be human-like. In fact, I expect it will feel very alien to us, and we'll feel very alien to them too. Mostly because we're not brains in jars. We eat, sleep, get laid, feel pain, etc., and all of that feeds back into the loop to make us what we are.

I don't think you're scared but I do think you're in denial.

Read what he says. He literally says it's depressing to watch all his core beliefs get dismantled. I'm not making this up. These words came out of his mouth. Follow the links. This is not some minor realization he went through. His entire world view was changed and he's the one claiming this... Not me.

His books which I read are all about recursion and strange loops and he's shocked that LLMs are simply feed forward networks with zero feedback. He claims it debunks everything he wrote.

He has the most to lose in disbelieving what's happening with LLMs. But he's to rational to do what you or most people do. He has to accept the truth and basically admit to being utterly wrong.

Once you read the links you will not agree with him. That's the most likely behavior for most people. But think on this. He has the most to lose and he thinks the books he wrote is wrong while you think the books he wrote is right.

Given the context the conditions are ripe for him to be the most biased and you to be less so, but he believes the opposite against his own self interest. From this you know he's not biased. Most likely you are.

It has nothing to do with fear or dislike of agi. It's just common human bias no one is sure where this lunacy comes from.

It's the Chinese Room argument all over again. People hear "predicting the next token" and all they can imagine is some sort of a statistical database lookup, ignoring the fact that when you have a huge corpus of data with incredibly complex internal correlations and all that data also happens to correlate with some unknown external thing, it's almost certain that a powerful learner will end up modeling that external thing if and when doing so will cause a quantum leap in prediction performance! A model that includes the external-thing hypothesis will almost certainly be simpler, ceteris paribus, than a model that doesn't.

That is what a model is; a parsimonious explanation of observations that admits extrapolation.

Exactly!

This is a preprint that was just uploaded to arxiv two days ago. Don't be hasty and assume that it settles any matter at all. Many such claims have been made before and many counter-claims also. There is still a lively debate on the subject and it will be some time before there is agreement.

More generally, any scholarly article is a claim, and should never be read as automatically true. That's something to keep in mind.

I thought it was completely obvious that if it can predict things about the world as well as it can it has some form of world model, even if inaccurate. I’m surprised that anyone would argue that a language model doesn’t model things

Yes, I see what you mean. It's frustrating but the word "model" is severely overused in Computer Science and AI and it can cause a lot of confusion.

Briefly, a "world model" is a theory possessed by an autonomous agent that describes the entities that exist in the world and how they interact with each other and with the agent, and that the agent can use to make decisions. This is the sense in which "model" is used when people talk about "model based" approaches to AI (such as planning, for instance, which is "the model based approach to autonomous behaviour").

A "language model" on the other hand is a statistical model of the text in a corpus. A statistical model is really a set of events, and their probabilities. The "events" in a statistical language model are usually word collocations, where a collocation is a word A found near a word B, for some measure of "near". If you've heard about "word embeddings", that's a model of word collocations in text corpora.

So the claim that LLMs "have a model" is about LLMs being autonomous agents with a world-model, while "Large Language Model" is about statistical modelling. There's no reason why a statistical language model should have a world model.

Since language often describes the world, I think a good language model must include a world model

Well, this has been claimed often, namely by some of the people who have developed statistical language models [1] but it's really not obvious how that should work. Where would a language model find the world model? Where would it store it? And why would it even need it?

Obviously I don't know how the human linguistic ability works but it's clear that for us, text, words, language, isn't carrying around with it a representation of the entire world. So where would a language model find it in its training text?

For example if I say "I sat by the sea today", there's no representation of the sea in the word "sea", rather, the representation is inside my head and yours, and the word "sea" is only used to point to that representation [2]. Why would it ever be possible to derive the representation just from the pointer? Like I say, it's really not obvious and I don't think anyone has given a convincing explanation - it's usually words all the way down when people try. In any case, it seems that humans had most of the capabilities conferred by a world model (which we share with other animals) before we had language (which we do not - share with other animals), so it seems likely that a world model must be developed first, before linguistic ability can be built on top of it. But that's a conjecture, of course.

I guess maybe, in principle, it should be possible to treat all of human cognition as some kind of hidden (latent) variable and then train a model to represent it only from its expression in text, but we don't really have any kind of modelling technique with that power. That would be a truly omnipotent model. So powerful, indeed, that it wouldn't need to do that in the first place- it would already be a kind of machine god.

______________

[1] e.g. I believe Tomáš Mikolov, one of the creators of word2vec, has claimed something like that but I'm not sure if I have a source for my own claim; more recently Ilya Sutskever, of OpenAI, has said words to that effect- I have a link somewhere I think, if you are curious.

[2] That's obvious because the word "sea" doesn't mean anything to someone who doesn't know English, but they'll probably have a word for the sea in their language, so they'll have the concept, without needing a specific word for it.

> Where would a language model find the world model? Where would it store it? And why would it even need it?

It is pattern recognition with many layers of abstraction. Obviously it will infer semantic relations at some level. It is a type of machine learning. The entire point of machine learning is to generate a model of the data which can be generalized to new inputs.

It would find the world model in semantic relations in text. It would store it in its vast neural network. It would need a world model in order to perform natural language understanding tasks, which is exactly what it was designed to do. When asked about the world it needs a model of the world in order to generate useful answers.

GPT-4 took months to train on a supercomputer and it generated a neural network of hundreds of gigabytes. What exactly was that supercomputer doing for several months and what exactly would the neural network represent if not a world model? Are you thinking it rote learned answers to every possible question?

> so it seems likely that a world model must be developed first

It is "a" model. It can be completely alien to whatever exists in a human mind but still function and exist as some type of model.

> Why would it ever be possible to derive the representation just from the pointer?

It only needs to have a sufficient representation of the Sea that is useful for the tasks we have trained it to do. It doesn't need to derive "the" one true representation that humans have of the Sea, whatever that is.

>> GPT-4 took months to train on a supercomputer and it generated a neural network of hundreds of gigabytes. What exactly was that supercomputer doing for several months and what exactly would the neural network represent if not a world model?

I believe GPT-4 was trained on a server farm, not a single computer. In any case what it was doing all that time was going over and over the text in its gigantic training corpus, which was of petabyte size, and optimising the objective P(tₖ|tₖ₋ₙ , ..., tₖ₋₁) i.e. the probability of token k given a "sliding window" of the n preceding (or surrounding) tokens.

There is nothing in this objective that needs a world model, and it is really not obvious why optimising this objective should lead to development of a world model, rather than, or in addition to, a model of the training corpus.

It is easy to see how this stuff works. You can train your own language model easily, although of course it would have to be a smaller language model. For example, you can train a Hidden Markov Model on the text of freely available literary works on project Guttenberg, or on wikipedia pages, and without too much compute (an ordinary laptop will do).

In fact, I recommend that as an exercise and as an experiment to gain a better understanding in how language modelling works, for those who are curious about questions regarding their ability to model something beyond text.

A good textbook to begin with statistical language modelling is "Foundations of Statistical Natural Language Processing" by Manning and Schűtze:

https://nlp.stanford.edu/fsnlp/

Or, just as good, "Speech and Language Processing" by Jurafsky and Martin:

https://web.stanford.edu/~jurafsky/slp3/

Or, if you don't have the time for an entire textbook, "Statistical Language Learning" by Eugene Charniak is an excellent, concise introduction to the subject:

https://archive.org/details/statisticallangu0000char

> I believe GPT-4 was trained on a server farm, not a single computer.

Yes a server farm of Nvidia A100 with supercomputing performance.

> It is easy to see how this stuff works. .. you can train a Hidden Markov Model..

No. GPT is not a Markov model.

A HMM is one way to train a language model that optimises the conditional objective I note above. A transformer is another. Working with a HMM will give you an insight into how language modelling works, and it's something you can do easily and cheaply, unlike training a giant transformer architecture.

Those primitive obsolete models are not the subject of the article. I already have a degree in applied mathematics and computer science and know the basics of machine learning.

My questions to you were rhetorical. I wasn't asking you for academic guidance.

I've already posted in another comment that I don't think there's a reason for me to insist, but I'd just like to point out that it's OK to not be an expert in everything but one should not have strong opinions on matters not of one's expertise. It's not a shame to not know everything, but it's unseemly to act like one does.

The exact same would apply to you. Projection. You are extremely arrogant with your opinions that no one else really agrees with and you talk down to people who disagree. Essentially you are insisting that your own personal definition of "world model" is true and the normal meaning of the term is not. Your argument is totally subjective and philosophical.

> development of a world model, rather than, or in addition to, a model of the training corpus

But the training corpus describes the world, so a model of the training corpus is a world model

A model of a training corpus is a model of a training corpus. Unless a system can read the corpus and understand what it says, it won't get any world model by modelling the corpus.

You have a subjective philosophical disagreement and are entitled to your opinion but it's not a technical argument.

It provides "a" world model and it's a useful model for its intended purpose. Not the "one true model" that is your your own personal life experience since birth.

Then humans also have a model our training corpus, which is an interpretation of the world via our senses. If we don’t ever directly experience the true underlying reality, do we truly understand it? In a way no, but it’s close enough that we call it understanding

No one can experience the real world directly. Whether it’s through language, vision (which feels quite “real” but is really just a 2D projection of light that we interpret), sound or anything else, it’s all just some byproduct of the world that we nonetheless can make useful predictions with. LLMs are more limited in their input data, but I don’t think the difference is fundamental; it’s all just different abstract representations of the world. Like how by looking at the sea we can gain some useful understanding of how it works, an LLM can, via descriptions of the sea, pick up enough information about it to answer questions about it or write text about it. It wouldn’t “find” a world model, it learns it based on its training data, it gets stored in the weights (because there’s nowhere else to store it), and it needs it because it’s making predictions about text, which often describes the world. If you don’t have a basic idea of things like cause and effect it’s difficult to write coherent text

> Why would it ever be possible to derive the representation just from the pointer?

Of course it cannot derive a representation from just a pointer. Neither humans or machines can do that. The words are not pointers in isolation. They are connected in a network of semantic relations. The model is in the relationships between pointers. Artificial neural network.

The question then becomes where do the "semantic" relations in the network come from. Unless there is some way to extract the meaning of words with respect to the real-world concepts they refer to, from text, any relation between words that your model, models is not going to be semantic (i.e. it won't represent meaning).

Like I say in another comment, this kind of explanation is "words all the way down", but it really doesn't explain where the process ends (or begins) and where the meaning comes into the words.

To give you an example, here is a small semantic network that I just created by hand, that represents is-a and has-a relations:

  abc32 is-a c356 
  7yt4 is-a c356
  c356 has-a 902a
  c356 has-a 8773

The names of entities in the network, like abc32, replace real English words that I just changed manually (or, well, vim-illy). You can use the is-a and has-a relations in the network to infer e.g. that abc32 has-a 902a, and that 7yt4 and abc32 are things of the same kind, but go ahead and try to understand what the real-world entities I renamed were. You can't- and neither can any model that is trained on text consisting of words whose meaning it understands as much as you understand the meaning of abc32 and 7yt4.

Wikipedia has an article on semantic networks:

https://en.wikipedia.org/wiki/Semantic_network

I haven't read it but from a quick glance it looks informative.

> but go ahead and try to understand what the real-world entities I renamed were.

Obviously not. Insufficient data. Try that substitution exercise again with terabytes of data (Common Crawl dataset) then feed it into a neural network and see if it can find any patterns.

This is not a matter of data quantity, but data quality. Text data doesn't have the information needed to learn a world model from it and so it doesn't matter how big a model you train, it will never learn the right representation (i.e. the right patterns).

> Text data doesn't have the information needed to learn a world model from it

As I mentioned in another reply you have a philosophical disagreement and are missing the point.

It provides "a" model of the world. Not necessarily the "one true model" that is your own personal life experience of the world since birth. There is enough information to build "a" world model in all the text ever written by humans about the world. That is obvious.

So your argument is completely subjective and philosophical, not technical.

GPT now includes imagery in its training set BTW.

I don't do philosophical arguments. But it's obvious this conversation not productive so there's no reason for me to insist.

Here's a definition I found somewhere. Do you agree with it?

A "world model" refers to the internal representation of the external environment that an agent (e.g., a human, animal, or machine) maintains and uses to navigate and make decisions within that environment. In other words, it's a mental or computational model of the world as perceived and understood by the agent.

Obviously LLMs have a world model. It's not your favourite model or preferred model or the best model in town, but it is a model of the world whether you like it or not, whether you agree with it or not.

But you are essentially making a philosophical argument about what defines a "world model". It is a disagreement about a subjective interpretation of the term.

Be careful about jumping to conclusions just to convince yourself you're right and I'm wrong. I do not assume what you think I assume. But it's obvious you're not looking for a curious conversation and reaching any kind of common understanding.

We are talking about the definition of "world model". If you disagree with it then your argument is not technical, it is philosophical in nature. This is like arguing with a religious fundamentalist.

A good language model IS a world model. They can be one in the same. Very likely what's going on with something like chatGPT. The world model is simply encoded as text.

I think LLMs are presenting some uncomfortable philosophical questions for people about how our own brains work and admitting that there is any kind of "intelligence" (even if very basic) in an LLM is an admission that our own brains may work in a similar manner.

For me, one of the most interesting things that have come out of LLMs is the confirmation that humans are very bad at reasoning and, consequently, its' a very bad idea to try and make machines that "think like humans", because that way we'll only make machines with none of the advantages of machines and all the disadvantages of computers.

For instance -I'm not trying to be mean and I'm certainly not blaming you in particular, because I've seen this very often- but the reasoning that because LLMs can generate language, and humans can generate language not only LLMs are somehow like humans but also humans are like LLMs is not sound.

For example, walls have ears, cats have ears, therefore walls are like cats and cats are like walls. That doesn't work because walls' ears are not like cats' ears and even if they were, that still wouldn't make walls cats and cats walls, it would just make them both entities with ears.

>-I'm not trying to be mean and I'm certainly not blaming you in particular, because I've seen this very often- but the reasoning that because LLMs can generate language, and humans can generate language not only LLMs are somehow like humans but also humans are like LLMs is not sound.

Nah. Nobody personifies LLMs like this. What you're laying out here is a fundamental mistake that you'd have to be extremely stupid to make. I think barely anyone is making this mistake to even qualify mentioning it.

Seriously who here things that LLMs are anything like humans? That is not the claim. The claim is that LLMs understand you. Intelligence and understanding are clearly orthogonal to "human-like"

>because that way we'll only make machines with none of the advantages of machines and all the disadvantages of computers.

There is no evidence, basically none whatsoever that general "perfect logical reasoning" is a thing that actually exists in the real world. None.

No animal we've observed does it. Humans certainly don't do it. The only realm this idea actually works is Fiction. and this was not like for a lack of trying. Some of the greatest minds worked on this for decades and some people still don't seem to get it. Logic doesn't scale. They break at real world relationships.

Logic systems are that guy in the stands yelling that he could've made the shot, while he's not even on the field.

That's a common take but it doesn't really hold any water: computers are logic machines and all of Computer Science is based on logic; and it works just fine.

Besides which, you may not hear about them in the news but pretty much all the classical, symbolic- and logic-based approaches of Good, Old-Fashioned AI are still going strong and are doing very well thank you in tasks in which statistical machine learning approaches underperform.

To give a few examples: automated planning and scheduling (used e.g. by NASA in its autonomous guidance systems for its spaceships and rovers), program verification and model checking (the latter has transformed the semiconductor industry and led to several recent Turing awards), SAT-solving and constraint satisfaction (where recent algorithmic advances have made it possible to solve many instances of NP-complete decision problems in polynomial time), adversarial search (AlphaGo and friends aren't going anywhere without Monte Carlo Tree Search), program synthesis (you can generate code with LLMs, but good luck if you want it to work correctly), automated theorem proving, heuristic search, rule learning, etc etc.

To clarify, those are all logic-based approaches that remain the state of the art in classical AI tasks where statistical machine learning has made no progress in the last many decades. You may not read about them in the news and they're not even considered "AI" by many, but that's because they work and work very well, and the "AI Effect" takes hold [1].

Even poor old expert systems are the de facto standard for expressing business logic in the software industry. I guess. Informally, of course.

_________________

[1] https://en.wikipedia.org/wiki/AI_effect

>computers are logic machines and all of Computer Science is based on logic; and it works just fine.

Not what I mean. Logic is part of the real world. Logic is not the real world. The idea that you can use this small subset of the world to model the whole thing is what is incredibly suspect. No one has demonstrated this and there is no real reason to believe it can.

>To clarify, those are all logic-based approaches that remain the state of the art in classical AI tasks where statistical machine learning has made no progress in the last many decades

Logic is good at what logic does. Please don't take this to mean me calling logic useless. It's not that statistical machine learning has not made progress. But you won't beat logic on problems with clear definitions and unambiguous axioms. That is very cool but that is clearly not all of reality.

>> The idea that you can use this small subset of the world to model the whole thing is what is incredibly suspect. No one has demonstrated this and there is no real reason to believe it can.

I agree and I don't think there's any kind of logic that can do that, but there is also no other formal system that can, so far. I'm not sure if you are suggesting there is?

>> But you won't beat logic on problems with clear definitions and unambiguous axioms. That is very cool but that is clearly not all of reality.

Certainly not. Logic is a set of powerful formalisms that we can use to solve certain kinds of problem - it's a form of maths, like geometry or calculus. I don't think anyone expects that geometry or calculus is going to solve every problem in existence and the same goes for logic.

>so far. I'm not sure if you are suggesting there is?

No i wasn't. I guess i wasn't very clear in my first reply.

I was mainly getting at this,

>and, consequently, its' a very bad idea to try and make machines that "think like humans", because that way we'll only make machines with none of the advantages of machines and all the disadvantages of computers.

No one is scaling up and pouring millions of compute into LLMs for general intelligence because they thought it was an excellent idea before the fact(virtually no one did, even some of the most verbal proponents).

They're doing it because it's seems to be working in a way logic failed to. and logic had the headstart, both in research and public consciousness. Nearly all of fictional ai is an envisioning of the hard symbolic logic general intelligence system that dominated early ai research. Logic was not the underdog here.

The point i was really driving at is that you say "because that way we'll only make machines with none of the advantages of machines and all the disadvantages of computers." almost like it's a choice, like Logic and GPT are both on the field and people are going for the worse player. Logic is not even in consideration because ot couldn't make the cut.

Like I say in my earlier comment, that's not right. Logic-based AI is still dominant in many fields. There is a lot of excitement about statistical machine learning (I know, it's an understatement) but that's only because statistical machine learning is finally working and doing things that couldn't be done with logic- not because logic can't do the things that statistical machine learning can't do (it can), and not because statistical machine learning can do the things that logic can do (it can't).

There are two worlds, if you want. For me it's a mistake to try and keep them separated. All the great pioneers of AI were not only this or only that people. e.g. Shannon's MSc thesis gave us boolean logic-based circuits (logic gates) and he also introduced information theory. The people who have made real contributions to AI and to computer science were never one-trick ponies.

An analogy I like to make is that we have both airplanes and helicopters. A flying machine is something so useful to have that we 're going to use any kind we can make. Obviously a helicopter will not compete with a jet for speed, but a jet isn't anywhere as manoeuverable or flexible as a helicopter. So we use both.

>> Logic was not the underdog here.

It wasn't, but there was a bit of a Triassic extinction event, with the last AI winter of the '90s that took the expert systems and basically severed the continuity of logic-based AI research. The story is more complex than that, but logic-based AI was dealt a powerful blow, and progress slowed down. Although again like I say in my other comment, it didn't get completely extinguished. Perhaps, like we recognise birds today as the remaining dinosaurs, we'll recognise the old-new wave of logic-based AI that is hidden by the AI effect.

Would you say that this in itself is due to how incomplete human reasoning is in the first place? That as a result, our ideas of logic and what perfect logic looks like are bound to fail? Or are you saying that the purest mathematical representation of logic cannot scale to a point where they can model and predict real world relationships successfully?

The second. Mathematical logic thrives on precision, clear definitions, and unambiguous axioms, but real-world systems are often marked by vagueness, uncertainty, and dynamic change.

Gödel’s Incompleteness Theorems also demonstrates that in any sufficiently powerful mathematical system, there are true statements that cannot be proven within the system. This implies that no matter how refined a logical system you devise, it will invariably be incomplete or inconsistent when grappling with real-world phenomena.

Gödel didn't say anything about real world phenomena. He was talking about formal languages and mathematics.

Of course. But if you truly cannot model every true statement in any formally devised system, then you are by definition going to have to reject valid rules that your logic cannot verify if you intend your system to perfectly logical.

I believe that's right but only in a deductive setting, and as long as there's a requirement for soundness. Inductive and abductive logical inference are not sound and they are very useful for real-world decision-making. But that is a developing field and there are still many unknowns there.

No offense taken. I tried to be clear in my phrasing that I don't personally subscribe to the "LLMs are just like us" mentality. Just making an observation as to why people have such visceral reactions to any implication that they might be.

This. I think this is the reason why so many people are in denial. Is All of intelligence simply trying to find the best fit curve in an n-dimensional scatter plot of data points?

Seems like the hyperbolic conclusion to make? The paper doesn't make that claim. Simple ml models can take data and find patterns, like separate red and blue, or xy coordinates. Those are a small amount of dimensions and easy to reason about.

This seems like the model found on its own another dimension to segregate data in deeper layers by generalizing and using context when learning. Its cool it can do that, but it still to me seems like drawing a best fit line just with higher dimensions.

I'm sure someone will correct me if they think I am way off.

Indeed. I was offering examples of how you could bootstrap cognitive processes at the beginning of the year and people were just sticking their fingers in their ears. A lot of people exercise their intellectual, economic, or political freedoms by rejecting others' efforts to gain the same things.

>A lot of people exercise their intellectual, economic, or political freedoms by rejecting others' efforts to gain the same things.

Could you please explain what the above has to do with a Large Languange Model?

I'm talking about how many people react to any startling new information or idea. The advent of LLMs is just a relevant example.

No new ideas have been advanced in anything reported here. A state machine change state is all that is going on here.

What I'd like to know is why should anyone care about this preprint? Nothing has been stated on how this software will improve the life of the average human being.

LK-99 at least would have had lead.somewhere if it worked.

Given that LK99 was a ceramyc it was unlikely to cause any major changes.

Who is the subject in the last bit of your last sentence? More pointedly: are you claiming that OpenAI is doing LLM research to gain political freedoms?

No, the second sentence is a general observation on a widespread human behavior pattern. I don't feel like expanding it into an essay right now.

If you actually read the paper, the analysis absolutely does not support the claim that LLMs are capable of maintaining a spatial or chronological internal world model.

The LLMs simply activated when locations or times were mentioned, with no understanding of the relationship between different places or different times beyond the fact that different places and times had different tags. Or in other words, no internal model of the world.

The lack of generally accepted definition for knowledge and intelligence makes these conversations difficult but the above post is important to remember.

The paper has nothing to do with the common sense problem.

While it is a preprint, I do wonder how useful this will be to improve generalization.

Termination analysis on linear polynomials is undecidable over the naturals without many constraints and is undecidable ofer the rationals.

If you think about how sstrees or even rtrees work, this isn't surprising to me that space and time are useful for compression which is required for learnability.

But I think people also see 'linear' and think that means easy. When it can mean serious restrictions on how it can be used.

Hard to explain the connection I have in mind without serious math but here is a paper that may cause some issues with trying to use these internal structures outside of the learnability implications.

https://arxiv.org/abs/2307.14805

> I think with this paper it becomes clear that the adamant denial was just human bias talking

It already becomes clear after playing with ChatGPT-4 for five minutes. But a lot of people still refuse to accept it.

A bigger Chinese Room - is still a Chinese Room. We already went through this with ELIZA 60 years ago

Wait I had the "It's just a stochastic parrot" parroted at me ad nauseum. Was this FUD all along?

No, this paper doesn't change that claim at all.

This paper is about learnability and not generalization of what is learned.

From the papers summary, note that they are referencing learning and not some implications that it gets us closer to a universal quantifier.

"We have provided evidence that LLMs learn linear representations of space and time that are unified across entity types and fairly robust to prompting, and that there exists individual neurons that are highly sensitive to these features. The corollary is that next token prediction alone is sufficient for learning a literal map of the world give sufficient model and data size."

I mean parrots have neurons. I didn't realize the line referred to the actual birds "parrot". Parrots are wicked smart.

Parrots, in the context of stochastic parrots is in reference to repeating or imitation mechanically without understanding.

The 'common sense' problem in AI is the important part, not the as of yet undefined abstract concept of intelligence.

LLMs are still pattern matching and finding with some stochastic aspects.

But artificial nurons are inspired by biological nurons, they are not equivalent in respect to computability. But artificial nural networks are useful and powerful for some problems that traditionally weren't solvable by other means.

Biological nurons have to embrace non-linear stochastic processes, which is a different problem set.

While not expressed in popsci this has been known for decades.

Great so biological neurons can perform feats of intuition, that simulated neurons simply can't. Even theoretically. Gotcha. Thank you very much for the explanation.

Yeah that's not the case lol. ann's can model any function even non linear ones.

Pairwise linear in respect to inputs, for feed forward networks like those that use attention is the restriction related to linear or linearizable.

Those words are important.

But we know that ML cannot model all turning machines in the limit. As to more reasonable models that gets more complex.

Rice's theorem in respect to functions doesn't fit with you claim:

For any non-trivial property of partial functions, no general and effective method can decide whether an algorithm computes a partial function with that property.

The generalization problem being the important part.

But if you want to claim that ANNs solve all of the undecidable problems too, write the paper and become famous.

>But we know that ML cannot model all turning machines in the limit.

No computer is truly turing without infinite memory, including humans.

>Rice's theorem in respect to functions doesn't fit with you claim: For any non-trivial property of partial functions, no general and effective method can decide whether an algorithm computes a partial function with that property. The generalization problem being the important part.

This is...irrelevant. The theorem makes the claim for complete generalization (which humans do not demonstrate) as you say. It doesn't actually matter if the algorithm can't make the claim of property for all processes. Just like it doesn't matter that computers or humans are not really turing complete.

We don't need a computer that is exactly a human. We need a computer that works.

I dunno. The guy that's freebasing copium sounds more convincing. Sorry.

Well as all you have as a response is an ad hominem you aren't the intended audience anyway.

I would love to learn why I am wrong.

You are clearly a smart guy, you are also clearly on the far side of the spectrum. Since it took you a while to notice that I was yanking your chain. That's ok, I'm somewhat of a full spectrum spaz myself.

The problem with smart people is that they have a very hard time to notice when their brain has sidestepped rational thought, and started to go into an emotional latent space. The reason for this is that your emotional life has no problem utilizing complex topics to shield itself from your conscious attention.

Face it. There isn't anything special about what a bio-neuron does. We just choose to ignore a lot of the intricacies of the bio-neuron. Because it is clearly irrelevant in order to sidestep halting. If a task suffers from halts the NN will simply side step the problem. Just like us. We are at a stage of technological development. Where we need to effen stop, and put all our effort into alignment. You realize this is true inside that cholesterol bag you call a brain. Your brain is just doing gymnastics of a preeteen soviet girl level, to stop yourself from getting it. Just get it.

The halting problem is part of asymptotic analysis on this case.

Just like with elementary limits you never reach it but you approach it.

The halting problem on what we typically call computers decidable but not in practical time lines.

The fact that a TM is not physically realizable doesn't change that claim.

Logical conjunctions are an example of something that is difficult in PAC learning and we know it is at least super polynomial but we don't if there are tractable forms like Schaefer's dicotomy allowing for linear time solutions for HORNSat as an example.

The tooling that works for asymptotic analysis on deterministic Turing machines does not transfer to biological nurons, because they simply aren't deterministic Turing machines.

Neurobiologists are fully aware of the limitations of modeling cortical neurons as deterministic systems.

While in pop science that difference may not be popular it is the general consensus of experts.

Your claims that cortical neurons are the same as a deterministic Turing machine is not the best accepted theory today.

In fact recent research says that qbits are a closer model to cortical neurons.

You can't blindly carry over the properties of deterministic Turing machines to qbits.

As non-deterministic Turing machine s are typically assigned to the special case of the type of NTM that defines the complexity class NP, I won't complicate the conversation with trying to explain the implications.

Here are some real challenges in creating intelligent agents, but none of that has anything to do with the halting problem. The reason you latch on to the strawman that I belive neurons are touring machines is because you want to win. I on the other hand, don't care about winning. I want to be right. The best way to be right is to change your mind immediately as you notice the flaws in your thinking.

The halting problem, or some other theoretical and esoteric complexities of computer systems, have nothing to do with the fact that current LLMs are not simply stochastic parrots. This doesn't mean they are conscious. They can't be. Because they aren't even multi-modal yet for starters. But that has nothing to do with halts or qbits. I don't even know where that red herring came from. Lay of the Penrose juice. There is no evidence that mammalian brains are room temperature quantum annealers. Nor is there evidence We need to model complete biological neurons to do learning. What if it's just a question of scale? We don't know that it's not. If it is, we are in big trouble.

Your argument is akin to saying that Gödel's incompleteness theorems, are the reason you can't complete your maths homework. Yes, in the "limit", it's true. But practically, we both know that homework can be solved.

Not quite that simple, or well known.

We know that the class of total Turing computable functions is not learnable in the limit.

But as to what else is AI-complete is mostly open questions.

But there is a difference.

Obviously ML is far better than humans in some domains so it is not a simple dichotomy.

So what is it about a biological neuron that sidesteps the halting problem? Is it because there is a weight and bias set to it. Someone should tell OpenAI to add weights and biases to their LLMs. That will make them make have an emotional like intuition, will it not?

EDIT: Maybe if we train the ANNs to focus their training on attention based techniques. Then they will simply tire on halts and continue on other problems until the model has sufficiently grokked lesser problems to focus on the previous halts.

Why do you think "biological neuron"s sidestep the halting problem?

People, who happen to run on biological neurons, have a sense of boredom that tries other approaches, and is also willing to eventually "give up", which aren't well captured in the standard algorithmic approaches.

The halting problem simply doesn't apply to biological neurons, they don't side step them, unless your context is humans writing down an Algorithm.

The rules of a human writing down an algorithm is the same thing as a Turing machine running an algorithm.

The halting problem applies for any system of computation that is at least as powerful as a TM, including any type of arithmetic or non-arithmetic calculation that is well-defined, AKA deterministic.

Cortical neuron firing is non-deterministic and more closely is modeled as probabilistic but still stochastic.

https://www.biorxiv.org/content/10.1101/2022.12.03.518978v1

Machine learning is constrained by the halting problem.

HALT is the conical example for what is decidable, but other problems exist and sometimes PAC learnability hits practical limits far before the finite time limits of RE.

As an example not invoking HALT:

https://arxiv.org/abs/2208.10255

There are absolutely constraints on BNNs, but as BNNs aren't deterministic Turing machines, it doesn't apply.

The real question is why do people resort to elementary oversimplified models of biological brains?

If you are in the field of studying the brain you will look for deterministic models that fit your needs to make computation more likely to be tractable.

But the false equivalency of ANNs to BNNs is problematic as a distraction from finding tractable solutions for computation.

Isn't that exactly how attention based neural networks work?

Attention is probably most easily conceptualized as run time reweighting.

A powerful tool but it doesn't really change the underlying model it is just modifying the weights at runtime.

The paper it originates from predates ChatGPT (let alone GPT-4). So I think a charitable interpretation is that it was partly wrong (and definitely unimaginative) at the time, and partly it's the case that very few people were expecting "let's just throw more parameters at it" to work and result in emergent general capabilities, which it appears to.

The next step in the research should be ablation of these time and space neurons to see how it affects accuracy on space/time completions which would help rule out the memorization of linear probes.

It would also be reassuring if they showed that they couldn't reach similar performance when they assigned random values to the target coordinates and used non-spatial words.

Can someone explain to me how they get these graphs? What exactly are they measuring in the model execution?

The interpretation of the LLM having a "world model" is a big stretch in terminology. Encoding a coherent or accurate set of spatial coordinates for places is not equivalent to understanding the actuality of world space and the relationship between items in that space. By this definition an accurate spreadsheet of the lat-long of every major city would also constitute a "world model". It shouldn't be surprising to anyone that a model trained on (likely many) such spreadsheets would also encode that coordinate data. What's notable is that it still takes a team of human researchers to plot these raw numbers onto a map and to interpret them as experiential differentiators in physical space.

There's no doubt that an LLM can uncover whatever is structurally included in its training data, even if that encoding is implicit. What's less believable is that the LLM somehow achieves a grounded understanding of physical aspects of experience purely from streams of raw text.

I think the paper engages in some equivocation here, as the abstract differentiates between "an enormous collection of superficial statistics" and "a coherent model of the data generating process", without admitting that the first should well imply the second. But that doesn't then further imply a "world model" in the sense that we understand it as sentient beings. For us, an internal model is useful, but when our model or texts don't agree with the actuality of the world, the actuality of the world takes precedence. For an LLM there is no distinguishing between its trained representations and an actual exterior place, or even any sense that an exterior space exists.

"Encoding a coherent or accurate set of spatial coordinates for places is not equivalent to understanding the actuality of world space and the relationship between items in that space."

What do you mean by the "actuality" of world space? It certainly has memorized an enormous data set of geographic points for various important locations and can do some pretty sophisticated reasoning and inference from them, and with the code interpreter it can compute various metrics derived from them. What can this thing do that humans can't?

Current LLMs clearly have much coarser models of the world than humans do. Their training data do not include as many modalities and training data with spatiotemporal dimensions are inadequate.

Up and coming multimodal models are changing this.

"Multimodal" models only promise an additional layer of indirection and interpretation from the world through sensors into the same conceptual mechanism. An image is converted to tokens, a sensor is converted to coordinates, and it is all fed back into a correlating model. But this isn't adequate either. The famous dress illusion picture has RGB values closer to gold and white, but the actual physical dress was black and blue. The difference between perception and reality is something we don't have a handle on when it comes to these models.

Tokens are just another representation of the image, like 3 matrices or electrical impulses in our brain. They probably encode most of the relevant information, the question is whether the linear algebra at the core of a transformer is good enough to use this data as efficiently as we do. I don't think we can currently say if it is or isn't adequate.

> There's no doubt that an LLM can uncover whatever is structurally included in its training data, even if that encoding is implicit. What's less believable is that the LLM somehow achieves a grounded understanding of physical aspects of experience purely from streams of raw text.

I was responding to the point in your first paragraph in your upper level comment and this quoted paragraph. Multimodal models will address that.

Your new point about perception could still be a problem for multimodal models, but it’s also possible that with sufficient data, the new models could address that (eg it could say people perceive the colors differently, as A and B).

Multimodal models map between human language and raw sensory data. It’s fundamentally similar, in principle, to part of how we learn our first language. (Not saying we do only that though.)

> coherent model of the data generating process -- a world model.

I'd never seen a definition of world model but this seems deficient in several ways because it does not mention anything about abstraction and logical reasoning.

Those aren't part of the model though. The model is just a representation of the world. An abstract representation can be useful, but so can more concrete ones. A representation that is amenable to logical reasoning is also useful, but the reasoning isn't part of the model.

Then why call it a world model? It's a "data" model and not actually a "world" model. World models would need to have logical relationships between the entities being modeled which would include non-trivial logical relations, R(x,y), and properties, P(x). Current language models do not have such representations and fail miserably on any task that is not explicitly included in the training set (sudoku being an obvious example).

>World models would need to have logical relationships between the entities being

No they wouldn't. There is no evidence, basically none whatsoever that logical can model reality or that general "perfect logical reasoning" is a thing that actually exists in the real world in real world relationships. None.

No animal we've observed does it. Humans certainly don't do it. The only realm this idea actually seems to work is Fiction. and this was not like for a lack of trying. Some of the greatest minds worked on this for decades and some people still don't seem to get it. Logic doesn't scale. It breaks at real world relationships.

We were using logic for image, speech, language synthesis and just had to throw it away once something that actually worked came about.

If logic was so great at modeling the world, we wouldn't be using LLMs like GPT-4 , we'd be using Cyc, the system with decades of a headstart.

Logic systems are that guy in the stands yelling that he could've made the shot, while he's not even on the field.

>included in the training set (sudoku being an obvious example)

LLMs can play sudoku just fine. https://arxiv.org/abs/2305.08291

Code is logic. LLMs are expressed with logical syntax. I recommend meditating on this fact for some time and trying to understand its implications instead of buying into the hype about LLMs being world models.

LLMs aren't world models. They're linguistic models. However, with the right training database, it turns out that a good parsimonious linguistic model will have a (multiple, probably) world model as a subset, as it is so much better at compressing/predicting the textual stream.

Logic is part of the real world. Logic is not the real world. It does not scale to all of reality. This is obvious. anything without clear definitions and unambiguous axioms and it falls apart. You don't have to take my word for it.

I’m seeking a brief almost hands-on introduction to how probing is done: in earlier papers and here. Suggestions? (Preferably with some photographs of napkin drawings.))

LLMs have an internal temporal model and yet ChatGPT still can't explain to me what happened in Tenet

I’ve watched Tenet and I’m not sure I can explain what happened either …

To be fair, Christopher Nolan made a whole 2+ hour movie attempting to explain the concept and couldn’t…

GPT4 explains it pretty clearly, I think:

"Tenet" is a 2020 science fiction action-thriller film written and directed by Christopher Nolan. The story revolves around concepts of time manipulation and inversion, making it a complex narrative that can be challenging to understand on first viewing.

Here's a basic summary: The protagonist (referred to as "The Protagonist", played by John David Washington) is a secret agent who gets involved in a mission to prevent World War III. He is introduced to a concept called "inversion," where the entropy of people or objects can be reversed, making them move backwards in time.

His mission leads him to cross paths with a Russian oligarch named Andrei Sator (played by Kenneth Branagh), who is collecting pieces of an algorithm that can invert the entropy of the entire world, effectively reversing time and destroying the present in favor of the future.

The Protagonist also meets Kat (played by Elizabeth Debicki), Sator's estranged wife, who becomes a crucial part of the mission. The Protagonist and his partner Neil (played by Robert Pattinson) use inversion to their advantage in several action sequences, including a car chase and a final battle at Sator's secret city, where they successfully get the algorithm and prevent the destruction of the present.

The twist at the end of the film reveals that Neil was recruited by a future version of The Protagonist, and that they have been working together for much longer than the duration of the film's events. This means that the organization "Tenet", which they work for, was created by The Protagonist himself in the future. Neil's character is seen sacrificing himself to ensure the mission's success, highlighting the theme of fate and predestination in the movie.

"Tenet" is a complex film that uses its time manipulation concept to construct a narrative that loops in on itself, with events and characters revealing their true significance only as the story progresses or even after the film ends. Nolan's movie plays with the concepts of time, fate, and free will, requiring viewers to actively engage with and decipher its narrative structure.

That's a better explanation than I can muster.

I bet if we were able to get better output from our brains we would see something similar, for those of us that have any spatial awareness capabilities

But we don't have the output to analyze aside from our language and assumption of shared experience

“Have an internal world model,” or “give the appearance of having an internal world model”?

(Disclaimer: I’ve only read the summary, not the full paper)

Recent papers about advanced Go-playing AIs have made it abundantly clear that, at best, their conceptual understanding of Go is deeply flawed, and at worst, completely absent. Yet under most circumstances they give the appearance of super-human understanding based on superhuman play.

Do we have any reason to think LLMs are different? That they understand anything beyond “this is the best word to go next here.”

To be clear, LLM output is remarkable, all the more so if the above is true.

I think sometimes it is good to zoom out a little bit and look at things on a higher level. The LLMs are not an organic creature. It doesn't in any shape or way relate to a human being except the knowledge it was trained on. A bacteria shares over 50% of its genetic sequences with a human and we don't see it as anything more than a glorified bionic automaton. An LLM shares 0% with us. In fact its "genetic sequence" is coded on a completely different physics altogether. It is unreasonable to expect it to understand or think the same way an organic creature do.

But the outcome of its action is irrefutable. It plays and does things better better than us. That is all we need to evaluate it on. I know that the topic is about "sentience" but my point is we are evaluating a complete alien being using human's standards. Of course we will see that it is lacking. It isn't human.

I’m most surprised that the embedding is actually linear. I wonder if that makes extrapolation perform better

Feed forward networks are pairwise linear in respect to inputs, they are effectively DAGs.

In theory you could represent an entire LLM as a single 2 dimensional graph of linear line segments.

It wouldn't be useful for much as the parameters are clustered in dense patches.

So are my synapses, and by and large my cortex is 2-dimensional as well (folded and scrunched in to a super complex shape by the limitations of my roughly spherical cranium)

Everything from ion channel response up is non-linear and not a deterministic Turing machine.

Random cite

https://www.jneurosci.org/content/13/1/334?ijkey=9f8730dd3d0...

The papers cites earlier papers that posit a ‘linear representation hypothesis’. Does your surprise “factor in” these papers? Do you suspect something different in play for the current paper?

Didn’t have that factored into my surprise because I’m not that informed. I just thought that the latent space representation being linear was interesting.

Maybe something to do with the Transformers are mesa optimizers thing?

Space and time (the abstractions) are lingisuric concepts, so this is perhaps not too surprising, but nevertheless interesting.

The space and time mapping they pull out of the LLM here is non-linguistic.

No, no they don't. People looking at the output of LLMs project their own internal models of the world, time, and space onto the output and project that model as if it were the LLM's.

The paper is looking at internal representations, not output.

You’re responding to the headline. Go look at the paper