In plain words: It traces how information moves through a language model, keeping only the steps that matter for each prediction. Unlike testing hand-written examples one at a time, it needs one run, and in Llama 2 it found always-important attention heads and parts specialized for code.
Abstract
Information flows by routes inside the network via mechanisms implemented in the model. These routes can be represented as graphs where nodes correspond to token representations and edges to operations inside the network. We automatically build these graphs in a top-down manner, for each prediction leaving only the most important nodes and edges. In contrast to the existing workflows relying on activation patching, we do this through attribution: this allows us to efficiently uncover existing circuits with just a single forward pass. Additionally, the applicability of our method is far beyond patching: we do not need a human to carefully design prediction templates, and we can extract information flow routes for any prediction (not just the ones among the allowed templates). As a result, we can talk about model behavior in general, for specific types of predictions, or different domains. We experiment with Llama 2 and show that the role of some attention heads is overall important, e.g. previous token heads and subword merging heads. Next, we find similarities in Llama 2 behavior when handling tokens of the same part of speech. Finally, we show that some model components can be specialized on domains such as coding or multilingual texts.
Javier Ferrando, Elena Voita
arXiv:2403.00824 · cs.CL, cs.AI · submitted Feb 27, 2024 · updated Apr 16, 2024
abstract · pdf · html