about
An Empirical Study on Why LLMs Struggle with Password Cracking (arxiv.org)
1 point by gnabgib 311 days ago | hide | past | pdf | discuss on HN

In plain words: They tested several pre-trained language models by giving them a person's name, birthdate, and hobbies and asking for likely passwords. Even counting a hit if the password appeared in their top ten guesses, every model scored under 1.5%, far below rule-based guessing tools.

Abstract · When Intelligence Fails: An Empirical Study on Why LLMs Struggle with Password Cracking

The remarkable capabilities of Large Language Models (LLMs) in natural language understanding and generation have sparked interest in their potential for cybersecurity applications, including password guessing. In this study, we conduct an empirical investigation into the efficacy of pre-trained LLMs for password cracking using synthetic user profiles. Specifically, we evaluate the performance of state-of-the-art open-source LLMs such as TinyLLaMA, Falcon-RW-1B, and Flan-T5 by prompting them to generate plausible passwords based on structured user attributes (e.g., name, birthdate, hobbies). Our results, measured using Hit@1, Hit@5, and Hit@10 metrics under both plaintext and SHA-256 hash comparisons, reveal consistently poor performance, with all models achieving less than 1.5% accuracy at Hit@10. In contrast, traditional rule-based and combinator-based cracking methods demonstrate significantly higher success rates. Through detailed analysis and visualization, we identify key limitations in the generative reasoning of LLMs when applied to the domain-specific task of password guessing. Our findings suggest that, despite their linguistic prowess, current LLMs lack the domain adaptation and memorization capabilities required for effective password inference, especially in the absence of supervised fine-tuning on leaked password datasets. This study provides critical insights into the limitations of LLMs in adversarial contexts and lays the groundwork for future efforts in secure, privacy-preserving, and robust password modeling.

Mohammad Abdul Rehman, Syed Imad Ali Shah, Abbas Anwar, Noor Islam, Hamid Khan
arXiv:2510.17884 · cs.CR, cs.AI, cs.LG · submitted Oct 18, 2025 · updated Dec 31, 2025
abstract · pdf · html

add comment on HN