ML Newsnew | past | best | rss | submit about
Modeling Others' Minds as Code (arxiv.org)
66 points by PaulHoule 348 days ago | hide | past | pdf | 42 comments on HN

In plain words: Everyday routines like "wait for the green light, then go" can be written as short code-like scripts; a language model drafts possible scripts, and probability rules pick which one best fits a person's few observed actions. From just a handful of observations, this predicted behavior in game and household simulations up to 50% better than standard pattern-copying and language-model approaches.

Abstract

Accurate prediction of human behavior is essential for robust and safe human-AI collaboration. However, existing approaches for modeling people are often data-hungry and brittle because they either make unrealistic assumptions about rationality or are too computationally demanding to adapt rapidly. Our key insight is that many everyday social interactions may follow predictable patterns; efficient "scripts" that minimize cognitive load for actors and observers, e.g., "wait for the green light, then go." We propose modeling these routines as behavioral programs instantiated in computer code rather than policies conditioned on beliefs and desires. We introduce ROTE, a novel algorithm that leverages both large language models (LLMs) for synthesizing a hypothesis space of behavioral programs, and probabilistic inference for reasoning about uncertainty over that space. We test ROTE in a suite of gridworld tasks and a large-scale embodied household simulator. ROTE predicts human and AI behaviors from sparse observations, outperforming competitive baselines -- including behavior cloning and LLM-based methods -- by as much as 50% in terms of in-sample accuracy and out-of-sample generalization. By treating action understanding as a program synthesis problem, ROTE opens a path for AI systems to efficiently and effectively predict human behavior in the real-world.

Kunal Jha, Aydan Yuenan Huang, Eric Ye, Natasha Jaques, Max Kleiman-Weiner
arXiv:2510.01272 · cs.AI, cs.LG · submitted Sep 29, 2025
abstract · pdf · html

add comment on HN

palmotea 348 days ago [–]
> Our key insight is that many everyday social interactions may follow predictable patterns; efficient "scripts" that minimize cognitive load for actors and observers, e.g., "wait for the green light, then go." We propose modeling these routines as behavioral programs instantiated in computer code rather than policies conditioned on beliefs and desires.

Aren't there already materials (made for people with autism) that catalog these scripts and make them explicit?

Edit: e.g. https://suelarkey.com.au/promoting-social-understanding-soci...

reply
snthpy 347 days ago | parent [–]
https://youtu.be/VLq8RPwDnMc?si=hKQWkGpA2ahTJoDn&t=397

> Pond party later. Too many ducks, too many voices, too many rules. None written down.

Still a good duck, always a good duck!

reply
Fizz43 347 days ago | parent [–]
This is for children though. Where is the adult version?
reply
SubiculumCode 348 days ago [–]
If you AI folks would spend more time reading actual semantic/episodic/working memory literatures, you wouldn't have to "invent" all these old ideas from scratch, over and over. Scripts are mental templates for routine situations. They are super important construct supporting memory and learning.
reply
naasking 348 days ago | parent [–]
Why would AI researchers assume that AI models are like human brains? Sleep is also super important for brains, but AI models don't need it.

Unless you're asserting that AI models and brains do share some things in common?

reply
SubiculumCode 348 days ago | parent [–]
'''Our key insight is that many everyday social interactions may follow predictable patterns; efficient "scripts" that minimize cognitive load for actors and observers, e.g., "wait for the green light, then go." We propose modeling these routines as behavioral programs instantiated in computer code rather than policies conditioned on beliefs and desires. ''' yet, as far as I can tell, did not cite the relevant literature.
reply
Darmani 348 days ago | parent [–]
Au contraire. DreamCoder is a stupendous work. https://dl.acm.org/doi/pdf/10.1145/3453483.3454080
reply
obeseneocortex 348 days ago | parent [–]
well, they speak the same language for one thing
reply
sarreph 348 days ago [–]
I've been thinking a lot recently about how much we'd be able to model the human existence as a foundation model (or multiple models representing each core part of the brain) hooked up to a load of sensors (such as 'optic nerve feed', 'temperature', 'cortisol levels') as input and as a response to tool calls -- and have all of this stream out as structured output controlling speech, motor movement, and other physiological functions.

I don't know if anyone is working on modelling the human existence (via LMMs) in this way... It feels like a Frankensteinian and eccentric project idea, but certainly a fun one!

reply
alganet 348 days ago [–]
I think this has applications in virtual fences. For example, if you want to restrict someone's behavior to a certain pattern.

Also, you can think of it as the subject (or subjects) programming the modeling agent: if the modeled mind is able to recognize it's being modeled, it's reasonable to consider the possibility that it can influence how those inferred scripts are created just by shaping its own behavior.

It can't be like an EEG, right? "Please be quiet and predictable while I model you, sir".

Of course, what humans think of predictability might not hold water. One could think he's behaving in a random way but in reality following well-known patterns (unknown to him).

It's an interesting problem. Absolutely terrifying stuff.

reply
FrustratedMonky 348 days ago [–]
Is this basically the "Fidelity" testing in Westworld?
reply
joaquincabezas 348 days ago [–]
when someone behaves in a very predictable way I use to say "I could code you in C!". Well, turns out is Python!
reply
shagie 348 days ago | parent [–]
Long ago... Think Geek T-shirt: "Go away or I will replace you with a very small shell script"

https://web.archive.org/web/20081204045017/http://www.thinkg...

reply
palmotea 348 days ago | parent [–]
> Long ago... Think Geek T-shirt: "Go away or I will replace you with a very small shell script"

Whoever bought that shirt could probably use some social skills coaching. It's not a good idea to wear a shirt that indiscriminately broadcasts contempt in all directions. I get the purchasers probably confused it for humor, but there's an important difference between humor that works on a viewer TV show and and humor embedded in the interaction of you with another real person.

I had this though recently at Walmart, after seeing the third such shirt (a visual pun meaning "fuck you"). Geeks often have the same attitude problems.

reply
saagarjha 348 days ago | parent [–]
Ideally you pair the shirt with a personality that never leaves any doubt that it is a joke.
reply
palmotea 348 days ago | parent [–]
> Ideally you pair the shirt with a personality that never leaves any doubt that it is a joke.

Ideally, but that still doesn't really solve the problem. It's not really practical to counter an indiscriminate broadcast of contempt with point to point interactions. People who don't know you or don't know you well will always see your shirt, if you wear it out.

You want to do the opposite: indiscriminately broadcast a kind personality, then deploy the sarcasm in point to point contexts "that never [leave] any doubt that it is a joke".

reply
saagarjha 348 days ago | parent [–]
I think this is pretty unreasonable, though. Are you against the husband/wife “I’m with stupid/I’m stupid” shirts? Generally it’s pretty obvious this is meant as a joke and not that one spouse genuinely degrades the other in public.
reply
AppleBananaPie 348 days ago | parent [–]
I see way too many couples that are passive aggressive with each other that I would wonder.

This made me realize just now the functional* couples I know would never wear somethings so blunt.

This is all purely anecdotal but just sharing my personal observations :)

*I need a better word that just describes the objective truth with no baggage please help lol

reply
mindslight 348 days ago | parent [–]
Geeks were just awkwardly ahead of the curve, as usual. The shirt you saw was being marketed to a wider audience, right? Also the prescience of "Fuck you, I'm eating"
reply
qsort 348 days ago | parent [–]
There's the old joke that many people who fear they could be replaced by AI are in fact too self-aggrandizing, they could be replaced by a 12-line python script.

If you want to get a bit meaner, you could profitably replace some people with the empty python script.

reply
emp17344 348 days ago | parent [–]
Weirdly misanthropic. Jobs exist for a reason - people who could be replaced by a python script already have been.
reply
constantcrying 348 days ago | parent [–]
If you work in any large organization you know that there are people who exists so that other people can not do their jobs.
reply
keybored 348 days ago | parent [–]
- I could program a person in C

- They could be replaced by a 12-line Python script

Predictably HN-misanthropic is more like it.

reply
more_corn 348 days ago | parent [–]
I’ve encountered many jobs that could be replaced with a script. When I was young and dumb I proposed replacing a whole department with a simple web app. The app was already finished and showed better success rates than the team of 6. The proposal was rejected.
reply
saagarjha 348 days ago | parent [–]
You’re assuming that companies are efficient at discovering which jobs these are.
reply
emp17344 348 days ago | parent [–]
You’re making an assumption that you can effectively judge who contributes value to the business and who does not.
reply
more_corn 348 days ago | parent [–]
Correct. For everyone’s sake I would hope this is possible.
reply
saagarjha 348 days ago | parent [–]
Sometimes!
reply
atoav 348 days ago | parent [–]
That is how it should be yeah.

In reality you will find pockets of utter incompetence in nearly every organization of considerable size. And I don't mean people who sometimes have bad days (who hasn't?) or struggle with particularly hard tasks (who doesn't?).

I mean long-time employee who lack the ability to wield the core tools and lack the core skills needed in their job. Imagine a blacksmith that doesn't know how to use a hammer and while they can talk very entertainingly and deeply about metals they certainly seem to fail at doing anything with it.

Now you may think I am exaggerating. I am not. Anyone in this thread who has worked in first level IT support will be probably agree. Now I am an educator, with a strong believe that nobody (aside those affected by certain medical conditions) is outside of learning and becoming better. I am known for my extreme patience and have won my provinces teaching price. Take this into account when I continue describing here.

We are talking about secretaries whose main tool (as a fraction of their workday) is the email client and calendar functionality, yet they fail to grasp the fundamental "IT for seniors" concepts of even the most basic version of the software they interact with more than 6 hours a day. In fact it is worse, they know they are bad and still file repeated advice into the mental equivalent of a paper shredder. I know of a person who has been doing this for 10 years now. Don't get me wrong, they somehow manage to not have it falling apart, but it is even exhausting to look at it from afar.

What would you think of a truck driver that after years on the job repeatedly asked you how to start the ignition?

reply
qsort 348 days ago | parent [–]
I'm joking, that wasn't meant to be serious commentary. I don't actually agree with the idea that most jobs are bullshit.
reply
deadbabe 348 days ago | parent [–]
Some people could be replaced with nothing at all.
reply
palmotea 348 days ago | parent [–]
> Some people could be replaced with nothing at all.

The kicker is it's often not the guy you think it is.

reply
bgwalter 348 days ago | parent [–]
I remember the joke differently. I heard it first way before the "AI" craze from an Italian philosopher (the original program must have been recorded in the 1980s and then rebroadcast):

"People who think they can be replaced by AI will be replaced."

In other words, the cheerleaders are so dumb that they probably could be replaced.

reply
PaulHoule 348 days ago | parent [–]
The classic book Remember me to God

https://www.amazon.com/Remember-Me-to-God/dp/B000LQ2SHG

talks about how people in low social positions (say a Bank Teller) have no opportunities to distinguish themself but have opportunities to make mistakes that they'll be held accountable for. Whereas if you are in a high social position you get to grade your own paper, get credit for your successes, and "fail up" when you screw up.

Given that neural networks get it wrong some of the kind they might be better to fill the high status positions (make up crazy stuff to say for Satya Nadella and Eric Schmidt for instance)

reply
atoav 348 days ago | parent [–]
If someone pissed off our sysadmin he would say something among the lines of: "Quiet, you are aware I could replace you with a simple script?"
reply
truelson 348 days ago [–]
Most automatic human behavior uses very simple logic. I spend a lot of time "not present" as my conscious part is often lost in something complicated. My automatic, unconscious actions are, well, pretty simple and subject to failure when I'm not "present."

I really really want this other part of my unconscious behavior modeled well. Would be very useful.

reply
more_corn 348 days ago | parent [–]
One good way to model your unconscious behavior is to examine the ways adversaries exploit your unconscious behavior.

By examining the common attacks on distracted people you can build a simple rule set that accounts for a large part of unconscious behavior. The attack I love to hate is the “subscribe now” popup. It inserts into your OODA loop at exactly the moment when your mind is engaged with important or interesting concepts. It is designed to compromise your decision making. I would use that as the foundation of my model because it sets out not only the behavior but the conditions under which the behavior is active.

Another set of rules can be inferred from ways phishing tricks people. (Activating urgency, fear, irritation, authority, avarice)

A third source of rules might be inferred from the practices of illusionists and cup and ball scams. Attention is finite, I’ve got it here so it’s not available in the important direction.

reply
About | RSS | RSS (all) | HN arXiv