about
The Geometry of Concepts: Sparse Autoencoder Feature Structure (arxiv.org)
2 points by roboboffin on Oct 30, 2024 | hide | past | pdf | discuss on HN

In plain words: They mapped how a language model's concept vectors are arranged, from tiny patterns to clusters to the whole cloud's shape. Removing global noise like word length made parallelogram patterns such as man−woman = king−queen much sharper than before.

Abstract

Sparse autoencoders have recently produced dictionaries of high-dimensional vectors corresponding to the universe of concepts represented by large language models. We find that this concept universe has interesting structure at three levels: 1) The "atomic" small-scale structure contains "crystals" whose faces are parallelograms or trapezoids, generalizing well-known examples such as (man-woman-king-queen). We find that the quality of such parallelograms and associated function vectors improves greatly when projecting out global distractor directions such as word length, which is efficiently done with linear discriminant analysis. 2) The "brain" intermediate-scale structure has significant spatial modularity; for example, math and code features form a "lobe" akin to functional lobes seen in neural fMRI images. We quantify the spatial locality of these lobes with multiple metrics and find that clusters of co-occurring features, at coarse enough scale, also cluster together spatially far more than one would expect if feature geometry were random. 3) The "galaxy" scale large-scale structure of the feature point cloud is not isotropic, but instead has a power law of eigenvalues with steepest slope in middle layers. We also quantify how the clustering entropy depends on the layer.

Yuxiao Li, Eric J. Michaud, David D. Baek, Joshua Engels, Xiaoqing Sun, Max Tegmark
arXiv:2410.19750 · q-bio.NC, cs.AI, cs.LG · submitted Oct 10, 2024 · updated Mar 30, 2025
abstract · pdf · html · 16 pages, 12 figures

add comment on HN
Also discussed: Nov 2024 (2 points, 0 comments) · Oct 2024 (2 points, 0 comments)