In plain words: A collection of biographies, history, news, and science articles asks AI models to pick and rank key events; a judge model reads those choices to reveal each model's personality. Traits differ: Orca 2 leans emotional and interpersonal, while Qwen 2.5 is strategic and analytical.
Abstract · Supernova Event Dataset: Interpreting Large Language Models' Personality through Critical Event Analysis
Large Language Models (LLMs) are increasingly integrated into everyday applications. As their influence grows, understanding their decision making and underlying personality becomes essential. In this work, we interpret model personality using our proposed Supernova Event Dataset, a novel dataset with diverse articles spanning biographies, historical events, news, and scientific discoveries. We use this dataset to benchmark LLMs on extracting and ranking key events from text, a subjective and complex challenge that requires reasoning over long-range context and modeling causal chains. We evaluate small models like Phi-4, Orca 2, and Qwen 2.5, and large, stronger models such as Claude 3.7, Gemini 2.5, and OpenAI o3, and propose a framework where another LLM acts as a judge to infer each model's personality based on its selection and classification of events. Our analysis shows distinct personality traits: for instance, Orca 2 demonstrates emotional reasoning focusing on interpersonal dynamics, while Qwen 2.5 displays a more strategic, analytical style. When analyzing scientific discovery events, Claude Sonnet 3.7 emphasizes conceptual framing, Gemini 2.5 Pro prioritizes empirical validation, and o3 favors step-by-step causal reasoning. This analysis improves model interpretability, making them user-friendly for a wide range of diverse applications. Project Page - https://www.supernova-event.ai/
Pranav Agarwal, Ioana Ciucă
arXiv:2506.12189 · cs.CL, cs.AI · submitted Jun 13, 2025 · updated Jun 22, 2025
abstract · pdf · html · Accepted at Actionable Interpretability Workshop at ICML 2025