about
Synthetic Data Generation with Large Language Models for Text Classification (arxiv.org)
1 point by PaulHoule on Oct 19, 2023 | hide | past | pdf | discuss on HN

In plain words: They tested whether how subjective a classification task is affects models trained on text examples a language model makes up instead of collecting by hand. More subjective tasks and examples led to worse results, so made-up data works best for clear-cut labels.

Abstract · Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations

The collection and curation of high-quality training data is crucial for developing text classification models with superior performance, but it is often associated with significant costs and time investment. Researchers have recently explored using large language models (LLMs) to generate synthetic datasets as an alternative approach. However, the effectiveness of the LLM-generated synthetic data in supporting model training is inconsistent across different classification tasks. To better understand factors that moderate the effectiveness of the LLM-generated synthetic data, in this study, we look into how the performance of models trained on these synthetic data may vary with the subjectivity of classification. Our results indicate that subjectivity, at both the task level and instance level, is negatively associated with the performance of the model trained on synthetic data. We conclude by discussing the implications of our work on the potential and limitations of leveraging LLM for synthetic data generation.

Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, Ming Yin
arXiv:2310.07849 · cs.CL, cs.AI · submitted Oct 11, 2023 · updated Oct 13, 2023
abstract · pdf · html · EMNLP 2023

add comment on HN