This feels to me more like incremental belief building than "thinking ahead of time" (which is not what the paper is claiming).

The model only has partial observability of the program it is working on (whatever tool call outputs are present in the context), as well as the trajectory of actions it has taken, and from this is building up some internal beliefs about the program - the probes used were looking for pretty crude things like "is this program well-formed" and "is this program correct (will it pass tests)".

The paper says that these program "properties" (beliefs) predict future state of the program up to 25 "steps" ahead, but given the setup this seems to be expected. An agent is trying to fix a program and/or maintain it in a working state, so it doesn't seem surprising that current well-formedness and correctness persist into the future, or that the model is correctly "optimistic" about the outcome of the next action it is planning/predicting.

This incremental belief building from partial observability reminds me of the ability of LLMs to predict valid chess moves when only given a truncated history of the games moves so far (e.g. last 20 moves, not all moves back to start of the game).

> more like incremental belief building than "thinking ahead of time"

concretely, what's the difference here?

(I suppose you could define "thinking ahead of time" as explicitly using something like "thinking tokens" which might be roughly analogous to system1/system2 thinking, but note that we still call system 1 thinking "thinking")

Well to be fair to the author "thinking ahead of time" wasn't how they described it - that is just the HN title. "Thinking ahead" sounds like the model reasoning about the future state of the program, which is fundamentally unknown. The most the model could have learnt during RL-training, if trained in a harness, is what kind of outcomes to generally expect such as agentic progamming generally being successful (trend towards well-formed and passes tests), except perhaps for certain context smells where it appears to be going down the tubes!

My hunch is that the model isn't thinking/planning head in order to achieve these probe results, but rather that they just reflect the models current take/beliefs on "how it's going" right now, and any correlation with "how it's going" in 25 steps time is just due to the inherent biases in the setup - well-formed programs tend to stay well-formed, etc.

Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.

If you want to take it that far, we've had results like this since the 60s (solomonoff induction). But of course if you state it like that, your (rightful) objection is that it's pretty vacuous (if I had the computational omniscience to just brute force all possible turing machines, whatever that even means, then sure, any 'f' gets subsumed into this paradigm).

A lot of philosophers, mathematicians, scientists, etc. effectively say, "Yeah, everything's just f(inputs of world) -> outputs!" That 'f' is doing a lot of heavy lifting. Which is kind of the point of mechanistic interperability - to make sure we're not jumping ahead of ourselves, and to make sure we're careful when we claim what "deep" and "structure" means, when it pertains to that 'f'.

Isn't that what "Attention is all you need" was about anyway? Does not sound like news to me.

It possibly goes a little further, though, since we’re talking about multi-step agentic loops here. So in addition to next token prediction there are of iterations of collecting input from outside sources.

I’m still hesitant to interpret this as “thinking ahead” without at least seeing some more back-and-forth in the literature first, though. This just seems like one of those spots where it makes sense to give other researchers some time to come up with additional hypotheses to explain the observations instead of focusing on the first one anyone proposes in isolation.

This is all obvious and not news, but there's a lot of people doing fighting retreats against LLM intelligence where the degree of obviousness matters.

Anticipatory token selection was obvious when you used structured output on gpt 3.5.

The model would construct/hallucinate pre-conditions to satisfy some final output value that the model was predisposed to.

I think that's reading a little too much into it. The paper shows the hidden states contain signals about whether the code is good right now, and whether the run is probably going to work out. That's interesting, for sure. But it doesn't mean the model has some detailed idea of what it's going to write 25 steps later.

A lot of that signal could be much simpler stuff. This task is hard. The agent seems stuck. The tests are getting better. The current approach looks promising. All of those things make future success easier to predict without the model actually "knowing what comes next" in any strong sense.

Also, their 25 steps are agent turns, not 25 code edits. The median run had something like 52 steps but only two edits, and the program label stays the same between edits. So "25 steps ahead" may sometimes just mean basically the same codebase, with a bunch of reading and test output in between.

So yeah, I'd say it's consistent with Sutskever's view. But "consistent with" and "confirmatory of" are doing very different amounts of work here.

I think that's the practical implication. Whether it's "thinking ahead" or not, if the model can reliably predict a failing trajectory several agent turns in advance, an agent could backtrack or branch much earlier instead of spending tokens on a dead end. That seems more interesting than the terminology debate.

> To effectively predict the next token it needs a good idea of what comes after the next token.

And that's all it needs. Not reasoning.

Save us from the reasoning / sentience / consciousness / thinking semantic quicksand.

Babbage’s Analytical Engine didn’t actually analyze anything, and terminology hadn’t gotten any more clear-cut since.

+1. Inputs, processing & memory do the job.

I suspect exact and/or universal definitions for intelligence, self-awareness, 'feelings' etc will prove to be elusive, and best we'll get is systems/robots etc behaving as if possessing those qualities. With some tests to put a number on them.

Downside is that may apply to us humans too.

At some level of performance reasoning becomes the most effective method to predict the next token

Chatbot "reasoning" arises from some level of the performance of the particular user, not of the chatbot.

I’m curious what this means? I think the evidence is pretty convincing that, while brittle, there is reasoning going on (though it depends on your definition of reasoning which I’m curious what that is for you).

Isn't "reasoning" in LLMs just training it to have an internal monologue to think through problems like a human would? i.e. extra tokens.

How do you define reasoning in a measurable way?

I keep asking the same question, and I think the steelman version would be "has metacognitive patterns similar to humans"

I think the anthropomorphic view of this is dangerous in the long term as it starts the argument that anything that isn't reasoned by a human isn't reasoning at all. This just changes the argument from LLMs can reason like a human to LLMs can't reason at all while ignoring the third possibility of "LLMs can reason not like a human".

One of the biggest things I've learned after the event of LLMs is that humans definitions of intelligence/thinking/reasoning/consciousness/etc are very poorly defined. Not just across society at large, but the sciences themselves.

> I think the anthropomorphic view of this is dangerous in the long term as it starts the argument that anything that isn't reasoned by a human isn't reasoning at all.

Something like this is actually a stance in the tradition of inferentialism (see the term sapience). Though "reasoning" isn't like, turing machine computability in this space; from what I understand, it's some abstract notion of the "space of reasons". I don't really understand it, honestly.

There's some merit to this, IMO. When an LLM goes wrong, do you blame the person or the LLM? As in, would you throw said LLM in jail, and hold the LLM accountable? Not right now, at least. I'm not sure if that's what is meant by the "space of reasons", but the intuition is that 'reason' can mean a lot of different things, pragmatically speaking. Reason as a legible audit trail is one of those ways.

But that's arguably getting into the social aspect of 'reason' (important!) and not like, what STEM people traditionally think of as 'reason'.

I may be wrong but I think the word you want is anthropocentric, not anthropomorphic. Anthropomorphism is turning something that isn't human into something human, e.g. an AI, while anthropocentrism is taking human as the center and default state of things.

> possibility of "LLMs can reason not like a human"

what would be the difference between not reasoning and reasoning not like a human?

I think you're overestimating the thinking here. I ask this question very often, and the persons I ask if of practically never answer.

I ask it to highlight that most people have no clear answer to what we mean by reasoning. It's a notoriously hard term to define with an precision, and even harder to define by precision in a way that would exclude LLM's, which is usually what the people I ask it of want to do.

I don't think your steelman version really "works" for that purpose because the immediate response would be that it makes "reasoning" largely irrelevant if you define the term that restrictively.

What’s the difference?

Is regexp or compiler lookahead reasoning? No.

Mistaking chatbot lookahead as reasoning comes with being gulled by the "artificial intelligence" sales pitch.

Why are those things not reasoning?

They are narrow and limited, but still primitive forms of reasoning.

It's been known for several years that LLM activations encode future tokens ahead of time (e.g. https://arxiv.org/abs/2404.00859).

But this has only been shown on simple tasks, so I think this paper is still quite neat. The interesting thing is that they show "future horizon length" varies across models.

Thank you for sharing. The way I reasoned about it myself: to make better predictions, we should know what type of outcomes are likely. We can express these outcomes by doing computations in some of the layers, and the training signal adjusts them so our model becomes more correct.

Of course, an interesting question what part of this internal computation is modeling for the future compared to guessing based on the given context (the past).

This is true of truly skillful people in their craft as well.

I know people, who initialize all required variables and write the logic after. which used to feel bonkers to me until I realized, they've done enough practice and memorization to be able to figure what they would need 10 steps down the line.

this does show that, models have a better model of the task and the expected end state.

The paper is saying 'context is context'?

That after a model has context about a project, the probes indicate a state that validates that?

Seems that the paper is highlighting the very nature of what LLMs are and what we expect them to be?

And that there is no 'thinking' here, it's just the state of the model?

How is this news? isn't it an obvious fact from the Transformer architecture?

> isn't it an obvious fact

Just below your question is a very confidently incorrect take about "parroting"... So, not obvious at all, at least for some people :)

Looking at the way they defined their test criteria, these seem like things that could be predicted with better than 50 percent accuracy by a trivial strategy. Am I mistaken? If I predict that on most edits, the program will not be fully correct but the number of passing tests will have gone up without regressing, then I ought to be correct a good deal of the time as long as the coding agent is half decent. I would expect that strategy to be less effective at predicting well-formedness, but that is also by far the worst-performing criterion they evaluated.

It makes intuitive sense. How else could you write a 500-line script top-to-bottom with no backspace key and no arrow keys and get all the imports etc. right upfront?

...by inferring both the imports and the script body from the same context? I think you're suggesting there's some kind of information flow from the anticipated body of the script back up to the imports, but I don't see why that would be necessary. Infer imports from context, infer body from context + imports. All strictly causal.

Sure, try it. It's harder than you think. It's not just imports, it's the entire program.

> you're suggesting there's some kind of information flow from the anticipated body of the script back up to the imports

Yes, I am suggesting this. I don't think it is possible to write programs without either anticipating what you're going to write down below before you get there, or else being able to go back and edit what you already wrote.

Of course agent harnesses allow the latter, but raw models without a harness can still do an exceptionally, superhumanly, good job of straight-line programming with no editing.

> Infer imports from context, infer body from context + imports. All strictly causal.

Of course it's causal, that's kind of a reductive way to look at it.

Just infer the entire program from context and then type what you inferred.

> It's harder than you think.

well, yeah, I'm not a large language model

In what ways are you different?

Can we use backspace key?

see parent comment

Has a life?

In what ways regarding the processing involved.

How much can it be reduced and simulated and be essentially the same principle?

That suggests a straightforward mapping between context -> program. Maybe for trivial programs but I don’t see how you can argue that for sufficiently complex programs.

Reasoning models perform better than non-reasoning models because they’re able to refine their code in multiple steps. That allows any part of the program to influence any other part of the program, not just from start -> end.

Human thinking serves a similar purpose. Basically intelligence needs to be able to backtrack if you want better performance.

Right, reasoning models are almost more notable for what they don't do, which is: try to generate the first token of the response from a single forward pass through the weights.

Non-reasoning models were still surprisingly good at generating working programs. I think it makes sense when you consider that it's still iterative; it doesn't have to generate the entire import block in a single pass, for example, more like decide at the end of each line whether there are more libraries it should import or whether to move on to the body of the program. It's less surprising that a single forward pass is able to make that type of decision IMO.

Taking Python as an example, you see LLMs use a lot of scoped imports and I think this is partly because it makes the output more resilient against failure to think forward at the start. Even a reasoning model will write out (the first iteration of) an entire file in one tool call, so once it starts generating it still has to keep going until it finishes, no pausing for breath.

Also it's interesting that non-reasoning models can still do quite well in a harness that lets them test and fix up their code. Like you said, it gives them the ability to iterate.

> you see LLMs use a lot of scoped imports and I think this is partly because it makes the output more resilient against failure to think forward at the start.

What makes you think an LLM has any such motivation? Or any motivation at all - beyond generating the most likely next token?

The fact that RLVR rewards successful trajectories.

Notice I didn't use the word motivation, which you decided to use. The reason LLMs use a technique is simply because it works, and that comes straight from training (specifically RLVR post-training).

I used to ask my coding agent to present two alternatives to choose from while implementing a task, and include a tokens required for each alternative to implement. (So that I can choose one which needs less token vs one which needs more rigour depending on task)

Finally there is evidence that the model kinda actually knows the correct token spend on each method.

>> Probes trained to predict the outcome of future edits (before they are materialized and written on disk) achieve performance above chance up to roughly 25 steps in advance.

Are these probes effectively run in parallel? The way this reads is more about predicting a future outcome than keeping the current token relevant based on past tokens.

In other words, since the next semantic prediction for forecasting the future is built on the training dataset, it's hard for anything truly new to emerge.

Then how do humans create something 'creative'—something that didn't exist before? I think it might be because the process of simplifying the complex system of nature differs between individuals. The data being learned now is all labeled by humans and simplified through human cognition. Within that kind of information, creativity seems hard to emerge.

Ultimately, with data that already contains interpretation, no matter how much you repeat the learning, it just becomes an encyclopedia that only explores within human knowledge, repeating predictions within human interpretation. So I wonder if we actually need a different encoder that interprets raw data—not based on human interpretation.

In reality, what changed Newton's absolute time to Einstein's relativity was a conclusion derived simply from observing the world. Newton's interpretation was supported by a lot of evidence in its time. If an AI studied all the medieval data from Newton's era, could it actually come up with the theory of relativity?

I'm always curious about this. I think AI is already very good at coding and will soon become better than humans. Logical structures are ultimately human interpretations, and reasoning within that framework is something AI can probably do more logically than humans. In other words, once humans create the framework, stacking the logical Jenga blocks within it—AI will be better at that.

But true creativity lies in breaking the framework itself, and I'm skeptical about whether AI can do that. The encoder also seems insufficient. There will likely be limits. I might be trapped in my own biases.

But the limitations of the current approach seem too clear to ignore.

When I look at the approach of these papers, it feels like an argument that adding shadows that imitate the world will eventually make them become the objects themselves.

I think the text, code, images, papers, and conversations that humans leave behind are not the world itself, but rather shadows of the world that have passed through human cognition and language. No matter how much you learn from those shadows, whether that leads to the ability to actually engage with the objects themselves seems like a separate issue.

I feel like something different is needed. But I'm not intellectually sharp enough to reason this through logically.this is just my intuition

>Ultimately, with data that already contains interpretation, no matter how much you repeat the learning, it just becomes an encyclopedia that only explores within human knowledge, repeating predictions within human interpretation. So I wonder if we actually need a different encoder that interprets raw data—not based on human interpretation.

That will still not create anything new-new, just more new, still dound by just being "an encyclopedia that only explores within the universe" at best.

We need to ask deeper questions on what human creativity actually is.

Why didn't humans 10,000 years ago make a car or a spaceship? We had the same minds back then from what we can tell biologically. Why in the 1500's did the precursors of these ideas start to come about? Data is needed, along with some method of analogy. Quite often when big breakthroughs happen there has been a massive amount of information gathered over the years. This is why said breakthroughs are not generally random. They are by people with the time, wealth, and information ability to put the pieces together.

>The data being learned now is all labeled by humans and simplified through human cognition.

Eh, that was a 'few years ago' thinking at this point. AI learning is working with a large amount of self gathered/generated training data now. At the same time almost everything you gather information wise is based on the interpretation of the society you live in. Reality tunnel is the term for this. Entire societies, millions of people, can be blind to something you see as obvious. Humans are not standalone machines, throw us in the woods as babies and you don't get a person that sees the world differently, you get a feral child that may never be capable of higher learning.

In this sense AI may be hobbled for some time. There are very few large models and they have a lot of the same biases, it's like a world that only 10 people live in for AI. Maybe over time training models will get far cheaper and then we'll be able to explore the frontier of having models 'do crazy shit for the fun of it' kind of like humans do quite often.

Is this methodologically similar to Anthropic's recent J-Space paper?

So this is essentially a predictive mental model similar to the one real programmers have while they're coding?

> A coding agent solving a software-engineering task spends dozens of steps reasoning

No. That's simple PR hype. Parrotry is not reasoning.

Why not? I think there’s fairly strong evidence that there is something that convincingly looks like reasoning. I think anthropic has done some nice circuit tracing and mechanistic interpretability work on this for instance.

> I think there’s fairly strong evidence that there is something that convincingly looks like reasoning.

No one is disputing that, but there is an enormous difference between evidence of 'something that looks like a thing' and evidence of the thing itself.

I tend to agree but to adjudicate that someone has to define what they consider reasoning to be.

What is 'reasoning' or IQ or more importantly consciousness are very difficult open questions in science.

An approach that might shed some light is instead to define what consciousness ISN'T or what thinking ISN'T. Naively let us say consciousness is NOT a large list of weights (i.e. an LLM).

The uncanny emergent ability of an LLM depends entirely on training data. A mathematical model is used to match output against training data (via loss functions etc). The training data contains all the human ingenuity, logic, rational, patterns and features.

Try giving an LLM model the alphabet ALONE and see what it comes up with? Why are you able to immediately reason that given the alphabet alone it could not 'reason', 'think' or produce much of anything useful.

To address briefly the idea of reasoning and something assembling reasoning somehow implying the same thing. Try the following thought experiment. Given a simulated world (e.g. The Matrix), no matter how good the simulation you would not actually get WET.

Crucially it's the responsibility of those who are making the claim here to bring evidence and definitions. The other approach, very popular on HN, has led to endless carping by people making outlandish claims without evidence and then demanding that every term possible be defined before they can be dismissed out of hand.

The claim that any two things are equivalent just because they look similar is a strong one, and the onus is on the person advancing it to bring evidence, not to insist that they are considered correct until it is disproved. That's how it works with the flat earth and religion and everything else; we shouldn't accept an unsupported conclusion just because some people really, really like AI and don't want to have to justify their claims.

The traditional response to such claims is ridicule [1]; we know trivially from all sorts of examples that presentation is not identity, and so 'but it looks similar' is not a convincing stance.

1 https://sites.psu.edu/sierraastle/2019/10/21/behold-a-man/

> Crucially it's the responsibility of those who are making the claim here to bring evidence and definitions

Ok, I claim that if something draws reasonable conclusions to questions it hasn't previously seen by performing steps that look like reasoning, then it is reasoning.

If you disagree then please define reasoning.

This is actually a perfect example of what I was referring to in my earlier comment; the strident [1] and rather aggressive insistence that your unsupported belief be proved wrong, rather than you providing literally any evidence that two distinct things are identical other than the assertion itself.

You're making a strong claim here; the burden of proof [2] lies with you, in much the same way that it would if you declared that horoscopes know the future. We have a sufficient explanation for LLM behaviour that we have no reason to discard based solely on your whim; if you want to be convincing, then you'll need more than assertion.

Similar appearance does not mean identical nature, and your position does not warrant serious consideration until you provide support for it.

[1] https://en.wikipedia.org/wiki/Sealioning [2] https://en.wikipedia.org/wiki/Burden_of_proof_(philosophy)

It's simply amazing how you can be so confidently wrong.

You got a perfectly reasonable response, clear, short. Yet, you chose to ignore it and attack the guy by claiming that the fact that LLMs give a reasonable response are his "beliefs", and "unsupported" at that. LoL.

Is you claim that LLMs don't give a reasonable answers? Did you try to ask them some complex questions in the last 12 months?

Or is your line of reasoning more in the line of "they heard all the questions before, and just memorized the answers?"

Or, do you not agree with a statement that giving reasonable answers is reasoning?

I think you're struggling her with the distinction between a statement and evidence for the statement. I know what the commenter believes, but they don't have any evidence for it beyond appearance.

The appearance of similarity in two things we can already explain without having to posit identical nature is not proof that they are identical. Screwdrivers and hammers, planes and birds: we know about all sorts of things with similar effects but different mechanisms.

If you want to argue for more similarity, you need more evidence than just your belief and superficial resemblance. That's got nothing to do with the results or capabilities of LLMs (and it's either blinkered or disingenuous that you're trying to elide that).

> do you not agree with a statement that giving reasonable answers is reasoning?

Note the seventh word: it's a statement. The other commenters knows it's a statement, you know it's a statement, not an argument or an explanation. but you are seemingly desperate for people to accept it as a proven fact without putting in the effort to prove it.

It's unscientific and frankly embarrassing to be so certain without being able to do anything but hector about it; if it's so obvious, prove it.

I provided my definition and argument. I can't engage further if you don't refute any of it.

You didn't provide either; you made a claim--in your own words--and nothing more. There's nothing to refute [1].

[1] https://en.wikipedia.org/wiki/Hitchens%27s_razor

> there is something that convincingly looks like reasoning

Convincing ... people who believe chatbots are intelligent? Well sure.

I don’t mean to say you are wrong, I just mean to ask what criteria you would consider necessary to satisfy in order for you to consider a system to be reasoning.

There is a certain amount of irony in your comment that I hope you appreciate.

You keep repeating this throughout the thread, one could even say just like a certain class of exotic birds :)

Why couldn't parroting, after a certain level and complexity of the parroting infrastructure, be reasoning? What a priori restriction forbids it from being such?

Flipping a NAND is not calculation either. Billions of them? Things change.

> Flipping a NAND is not calculation either.

But it is.

Well, processing a weighted model to respond is reasoning too.

What exactly do you consider reasoning?

Copying someone else's reasoning process (as best you understand it) is still a limited form of reasoning, even if it might be considered as cargo-cult reasoning (copying without understanding).

> Copying someone else's reasoning process (as best you understand it) is still a limited form of reasoning

The LLM is not copying someone else's process. All it knows it someone else's product, and its only process is to compute the most likely next token within that.

True, and this is why I said "as best you understand it". The visible (copyable) circumstances in which someone makes a reasoning decision aren't the same as the internal decision making process, so this is just cargo-cult copying - copy the surface form without understanding the reason for it.

The perhaps surprising thing is that it works so well some of the time - specifically for math and coding.

Its surprises people just as did a web seach 30 years ago. "Wow, it found my page amongst millions - in seconds!" Just because raw dumb huge computer power is beyond the average user's compehension.