about
Empirical Evaluation of Large Language Models in Automated Program Repair (arxiv.org)
5 points by Bluestein on Jul 9, 2025 | hide | past | pdf | discuss on HN

In plain words: Tested four open-source language models at fixing real bugs in Java, C/C++, and Python, trying four prompt styles and checking over 600,000 suggested fixes. Code-focused models beat larger general ones, and simply making a model bigger did not reliably fix more bugs.

Abstract

The increasing prevalence of software bugs has made automated program repair (APR) a key research focus. Large language models (LLMs) offer new opportunities for APR, but existing studies mostly rely on smaller, earlier-generation models and Java benchmarks. The repair capabilities of modern, large-scale LLMs across diverse languages and scenarios remain underexplored. To address this, we conduct a comprehensive empirical study of four open-source LLMs, CodeLlama, LLaMA, StarCoder, and DeepSeek-Coder, spanning 7B to 33B parameters, diverse architectures, and purposes. We evaluate them across two bug scenarios (enterprise-grades and algorithmic), three languages (Java, C/C++, Python), and four prompting strategies, analyzing over 600K generated patches on six benchmarks. Key findings include: (1) model specialization (e.g., CodeLlama) can outperform larger general-purpose models (e.g., LLaMA); (2) repair performance does not scale linearly with model size; (3) correct patches often appear early in generation; and (4) prompts significantly affect results. These insights offer practical guidance for designing effective and efficient LLM-based APR systems.

Jiajun Sun, Fengjie Li, Xinzhu Qi, Hongyu Zhang, Jiajun Jiang
arXiv:2506.13186 · cs.SE · submitted Jun 16, 2025
abstract · pdf · html

add comment on HN