about
Sky sphere representation in language models (29 Jul 2026) (arxiv.org)
2 points by throwaway81523 23 days ago | hide | past | pdf | discuss on HN

In plain words: Probing large language models' inner workings, this study checks whether they hold a map of the night sky, showing where each star sits. Most open models do: a simple readout of their internal activity places objects with a median error as low as 12°.

Abstract · Sky sphere representation in language models

We analyze whether language models of size ~100B have a representation of the night sky map that is decodable from their residual stream. We find that most of the considered open-source models do have such a representation, and it often even surfaces to the top principal components on prompts that ask questions like ``what is close to this object in the night sky''. In all but one model this representation showed significant scores in LOO testing, containing up to 65-85% of variance ($R^2$-score) and having median angular error down to $12^\circ-21^\circ$. We verify that our representation is not a simple leak from a correlated flat representation. To our knowledge, this representation is the first example of a curved high-dimensional irreducible feature manifold. Codes used in the paper are published at https://github.com/l3erdnik/Decodable-sky

Aleksandr Berdnikov, Yevgeny Liokumovich
arXiv:2607.27092 · cs.LG · submitted Jul 29, 2026
abstract · pdf · html

add comment on HN