about
OpenAI o1 System Card (2024) (arxiv.org)
2 points by ronfriedhaber 47 days ago | hide | past | pdf | discuss on HN

In plain words: The models are trained to think step by step before answering, and to weigh the safety rules while they think, so they can spot risky requests. This made them better at refusing illicit advice, stereotypes, and jailbreak tricks than earlier safety training.

Abstract · OpenAI o1 System Card

The o1 model series is trained with large-scale reinforcement learning to reason using chain of thought. These advanced reasoning capabilities provide new avenues for improving the safety and robustness of our models. In particular, our models can reason about our safety policies in context when responding to potentially unsafe prompts, through deliberative alignment. This leads to state-of-the-art performance on certain benchmarks for risks such as generating illicit advice, choosing stereotyped responses, and succumbing to known jailbreaks. Training models to incorporate a chain of thought before answering has the potential to unlock substantial benefits, while also increasing potential risks that stem from heightened intelligence. Our results underscore the need for building robust alignment methods, extensively stress-testing their efficacy, and maintaining meticulous risk management protocols. This report outlines the safety work carried out for the OpenAI o1 and OpenAI o1-mini models, including safety evaluations, external red teaming, and Preparedness Framework evaluations.

OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.
arXiv:2412.16720 · cs.AI · submitted Dec 21, 2024 · updated Apr 30, 2026
abstract · pdf · html

add comment on HN