about
User Model Extraction via Belief Self-Distillation (arxiv.org)
1 point by sbulaev 4 days ago | hide | past | pdf | discuss on HN

In plain words: A technique extracts the hidden picture a chatbot keeps of who it's talking to and writes edited versions back in, learning from unlabeled chats. Editing just that picture changed whether the bot refused the same request, beating direct nudges of its internal states.

Abstract

Large language models (LLMs) implicitly infer attributes of their users and adapt their behavior accordingly, yet these beliefs remain difficult to inspect and causally manipulate. We introduce Belief Self-Distillation (BSD), a unified read-write framework that bridges linear and causal probing by learning a compact user representation that can be both decoded and written back into the model. The frozen LLM acts as its own teacher, distilling beliefs from natural conversations without external annotations. Unlike conventional probing, BSD isolates not only information present in activations, but a state whose causal role can be directly tested. Across multiple model families, BSD faithfully recovers user beliefs and enables substantially stronger interventions than matched hidden-state steering. Crucially, we find that refusal depends not only on the request, but on the model's inferred user intent: changing this belief alters refusal while holding the request fixed. We further uncover a striking cross-model regularity: independently trained LLMs converge on a shared geometry for representing their users. Together, these results reveal implicit user models as readable and causally writable internal states with direct implications for AI safety, shaping how models condition safety decisions on whom they believe they are interacting with.

Ali Holmov, Yiran Huang, Kirill Bykov, Zeynep Akata
arXiv:2609.31603 · cs.LG, cs.CL · submitted Sep 25, 2026
abstract · pdf · html

add comment on HN