about
Measuring and Understanding LLM Identity Confusion (arxiv.org)
21 points by _____k on Dec 28, 2024 | hide | past | pdf | 1 comment on HN

In plain words: Tested 27 chatbots by checking their documentation, asking each who it is, and comparing their outputs to spot copying. About a quarter (25.93%) misidentified themselves, and the cause was making things up rather than copying — which hurt user trust more than logic errors.

Abstract · I'm Spartacus, No, I'm Spartacus: Measuring and Understanding LLM Identity Confusion

Large Language Models (LLMs) excel in diverse tasks such as text generation, data analysis, and software development, making them indispensable across domains like education, business, and creative industries. However, the rapid proliferation of LLMs (with over 560 companies developing or deploying them as of 2024) has raised concerns about their originality and trustworthiness. A notable issue, termed identity confusion, has emerged, where LLMs misrepresent their origins or identities. This study systematically examines identity confusion through three research questions: (1) How prevalent is identity confusion among LLMs? (2) Does it arise from model reuse, plagiarism, or hallucination? (3) What are the security and trust-related impacts of identity confusion? To address these, we developed an automated tool combining documentation analysis, self-identity recognition testing, and output similarity comparisons--established methods for LLM fingerprinting--and conducted a structured survey via Credamo to assess its impact on user trust. Our analysis of 27 LLMs revealed that 25.93% exhibit identity confusion. Output similarity analysis confirmed that these issues stem from hallucinations rather than replication or reuse. Survey results further highlighted that identity confusion significantly erodes trust, particularly in critical tasks like education and professional use, with declines exceeding those caused by logical errors or inconsistencies. Users attributed these failures to design flaws, incorrect training data, and perceived plagiarism, underscoring the systemic risks posed by identity confusion to LLM reliability and trustworthiness.

Kun Li, Shichao Zhuang, Yue Zhang, Minghui Xu, Ruoxi Wang, Kaidi Xu, Xinwen Fu, Xiuzhen Cheng
arXiv:2411.10683 · cs.CR · submitted Nov 16, 2024
abstract · pdf · html · 16 pages, 8 figure, 6 tables

add comment on HN

They don't understand their identity, because they don't understand the world. They only understand language, and some of the underlying connections that allow the language to be strung together. In other words, they may have something that seems like a world model, but it's a 2nd order construct that grows out of language, and is therefore quite weak.