about
Evaluating the Performance of ChatGPT for Spam Email Detection (arxiv.org)
1 point by PaulHoule on Mar 1, 2024 | hide | past | pdf | discuss on HN

In plain words: ChatGPT was asked to spot spam in English and Chinese emails using prompts with a few example messages, and compared with five standard trained classifiers. It lost to the deep learning models on the large English dataset but won on the small Chinese one.

Abstract

Email continues to be a pivotal and extensively utilized communication medium within professional and commercial domains. Nonetheless, the prevalence of spam emails poses a significant challenge for users, disrupting their daily routines and diminishing productivity. Consequently, accurately identifying and filtering spam based on content has become crucial for cybersecurity. Recent advancements in natural language processing, particularly with large language models like ChatGPT, have shown remarkable performance in tasks such as question answering and text generation. However, its potential in spam identification remains underexplored. To fill in the gap, this study attempts to evaluate ChatGPT's capabilities for spam identification in both English and Chinese email datasets. We employ ChatGPT for spam email detection using in-context learning, which requires a prompt instruction with (or without) a few demonstrations. We also investigate how the number of demonstrations in the prompt affects the performance of ChatGPT. For comparison, we also implement five popular benchmark methods, including naive Bayes, support vector machines (SVM), logistic regression (LR), feedforward dense neural networks (DNN), and BERT classifiers. Through extensive experiments, the performance of ChatGPT is significantly worse than deep supervised learning methods in the large English dataset, while it presents superior performance on the low-resourced Chinese dataset. This study provides insights into the potential and limitations of ChatGPT for spam identification, highlighting its potential as a viable solution for resource-constrained language domains.

Shijing Si, Yuwei Wu, Le Tang, Yugui Zhang, Jedrek Wosik, Qinliang Su
arXiv:2402.15537 · cs.CL, cs.AI, cs.CY, cs.LG · submitted Feb 23, 2024 · updated Feb 12, 2025
abstract · pdf · html · 12 pages, 4 figures; Accepted by Pacific Journal of Optimization (PJO)

add comment on HN