about
Representation Engineering: A Top-Down Approach to AI Transparency (arxiv.org)
2 points by Anon84 on Oct 4, 2023 | hide | past | pdf | discuss on HN

In plain words: Instead of picking apart single neurons or wiring, this approach reads and steers whole activity patterns that stand for ideas like honesty or power-seeking. Tests showed these simple tools let people both understand and control a chatbot's behavior on safety problems.

Abstract

In this paper, we identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights from cognitive neuroscience. RepE places population-level representations, rather than neurons or circuits, at the center of analysis, equipping us with novel methods for monitoring and manipulating high-level cognitive phenomena in deep neural networks (DNNs). We provide baselines and an initial analysis of RepE techniques, showing that they offer simple yet effective solutions for improving our understanding and control of large language models. We showcase how these methods can provide traction on a wide range of safety-relevant problems, including honesty, harmlessness, power-seeking, and more, demonstrating the promise of top-down transparency research. We hope that this work catalyzes further exploration of RepE and fosters advancements in the transparency and safety of AI systems.

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, et al.
arXiv:2310.01405 · cs.LG, cs.AI, cs.CL, cs.CV, cs.CY · submitted Oct 2, 2023 · updated Mar 3, 2025
abstract · pdf · html · Code is available at https://github.com/andyzoujm/representation-engineering

add comment on HN