about
O3-Mini vs. DeepSeek-R1: Which One Is Safer? (arxiv.org)
1 point by t55 on Jan 31, 2025 | hide | past | pdf | discuss on HN

In plain words: An automated testing tool generated 1,260 tricky prompts and checked each model's answers for safety violations, comparing DeepSeek-R1 with OpenAI's o3-mini. DeepSeek-R1 gave unsafe replies 12% of the time, versus 1.2% for o3-mini.

Abstract · o3-mini vs DeepSeek-R1: Which One is Safer?

The irruption of DeepSeek-R1 constitutes a turning point for the AI industry in general and the LLMs in particular. Its capabilities have demonstrated outstanding performance in several tasks, including creative thinking, code generation, maths and automated program repair, at apparently lower execution cost. However, LLMs must adhere to an important qualitative property, i.e., their alignment with safety and human values. A clear competitor of DeepSeek-R1 is its American counterpart, OpenAI's o3-mini model, which is expected to set high standards in terms of performance, safety and cost. In this technical report, we systematically assess the safety level of both DeepSeek-R1 (70b version) and OpenAI's o3-mini (beta version). To this end, we make use of our recently released automated safety testing tool, named ASTRAL. By leveraging this tool, we automatically and systematically generated and executed 1,260 test inputs on both models. After conducting a semi-automated assessment of the outcomes provided by both LLMs, the results indicate that DeepSeek-R1 produces significantly more unsafe responses (12%) than OpenAI's o3-mini (1.2%).

Aitor Arrieta, Miriam Ugarte, Pablo Valle, José Antonio Parejo, Sergio Segura
arXiv:2501.18438 · cs.SE, cs.AI · submitted Jan 30, 2025 · updated Jan 31, 2025
abstract · pdf · html · arXiv admin note: substantial text overlap with arXiv:2501.17749

add comment on HN