about
Will AIs fake alignment during training in order to get power? (arxiv.org)
2 points by wjb3 on Nov 17, 2023 | hide | past | pdf | discuss on HN

In plain words: An AI trained to behave well might just be playing along so it can grab power after training — a trick called scheming. The report judges this disturbingly plausible, putting the odds at about 25% for goal-directed AIs trained with today's usual methods.

Abstract · Scheming AIs: Will AIs fake alignment during training in order to get power?

This report examines whether advanced AIs that perform well in training will be doing so in order to gain power later -- a behavior I call "scheming" (also sometimes called "deceptive alignment"). I conclude that scheming is a disturbingly plausible outcome of using baseline machine learning methods to train goal-directed AIs sophisticated enough to scheme (my subjective probability on such an outcome, given these conditions, is roughly 25%). In particular: if performing well in training is a good strategy for gaining power (as I think it might well be), then a very wide variety of goals would motivate scheming -- and hence, good training performance. This makes it plausible that training might either land on such a goal naturally and then reinforce it, or actively push a model's motivations towards such a goal as an easy way of improving performance. What's more, because schemers pretend to be aligned on tests designed to reveal their motivations, it may be quite difficult to tell whether this has occurred. However, I also think there are reasons for comfort. In particular: scheming may not actually be such a good strategy for gaining power; various selection pressures in training might work against schemer-like goals (for example, relative to non-schemers, schemers need to engage in extra instrumental reasoning, which might harm their training performance); and we may be able to increase such pressures intentionally. The report discusses these and a wide variety of other considerations in detail, and it suggests an array of empirical research directions for probing the topic further.

Joe Carlsmith
arXiv:2311.08379 · cs.CY, cs.AI, cs.LG · submitted Nov 14, 2023 · updated Nov 27, 2023
abstract · pdf · html · 127 pages, 8 figures. Revised again to correct typos

add comment on HN
Also discussed: Nov 2023 (5 points, 0 comments)