about
Analogical Reasoning-Augmented Interactive Data Annotation (arxiv.org)
1 point by azhenley on May 22, 2024 | hide | past | pdf | discuss on HN

In plain words: A labeling helper pairs a trained model with a lookup of similar already-labeled examples, trusting the lookup more when the model's guess looks shaky, so people fix fewer wrong suggestions. Across tasks, it cut human correction work by 11% on average versus standard interactive labeling.

Abstract · ARAIDA: Analogical Reasoning-Augmented Interactive Data Annotation

Human annotation is a time-consuming task that requires a significant amount of effort. To address this issue, interactive data annotation utilizes an annotation model to provide suggestions for humans to approve or correct. However, annotation models trained with limited labeled data are prone to generating incorrect suggestions, leading to extra human correction effort. To tackle this challenge, we propose Araida, an analogical reasoning-based approach that enhances automatic annotation accuracy in the interactive data annotation setting and reduces the need for human corrections. Araida involves an error-aware integration strategy that dynamically coordinates an annotation model and a k-nearest neighbors (KNN) model, giving more importance to KNN's predictions when predictions from the annotation model are deemed inaccurate. Empirical studies demonstrate that Araida is adaptable to different annotation tasks and models. On average, it reduces human correction labor by 11.02% compared to vanilla interactive data annotation methods.

Chen Huang, Yiping Jin, Ilija Ilievski, Wenqiang Lei, Jiancheng Lv
arXiv:2405.11912 · cs.CL, cs.HC · submitted May 20, 2024 · updated Jun 1, 2024
abstract · pdf · html · Accepted to ACL 2024. Camera Ready

add comment on HN