In plain words: The study compared how sure coding AI models felt about their answers with how often they were actually right, across many programming languages. Weaker models and rare languages showed the biggest gap, with confidence far outrunning real skill.
Abstract
As artificial intelligence systems increasingly collaborate with humans in creative and technical domains, questions arise about the cognitive boundaries and biases that shape our shared agency. This paper investigates the Dunning-Kruger Effect (DKE), the tendency for those with limited competence to overestimate their abilities in state-of-the-art LLMs in coding tasks. By analyzing model confidence and performance across a diverse set of programming languages, we reveal that AI models mirror human patterns of overconfidence, especially in unfamiliar or low-resource domains. Our experiments demonstrate that less competent models and those operating in rare programming languages exhibit stronger DKE-like bias, suggesting that the strength of the bias is proportionate to the competence of the models.
Mukul Singh, Somya Chatterjee, Arjun Radhakrishna, Sumit Gulwani
arXiv:2510.05457 · cs.AI, cs.CL, cs.SE · submitted Oct 6, 2025
abstract · pdf · html