about
What Makes Language Models Good-Enough? (arxiv.org)
3 points by PaulHoule on Jun 15, 2024 | hide | past | pdf | discuss on HN

In plain words: They built a 7,680-example test of sentences with human judgments to see which model shapes copy our habit of doing just enough work to get by. Models with fewer layers and fewer attention spots (parts that weigh words) matched the full model on those shortcuts.

Abstract · What Makes Language Models Good-enough?

Psycholinguistic research suggests that humans may build a representation of linguistic input that is 'good-enough' for the task at hand. This study examines what architectural features make language models learn human-like good-enough language processing. We focus on the number of layers and self-attention heads in Transformers. We create a good-enough language processing (GELP) evaluation dataset (7,680 examples), which is designed to test the effects of two plausibility types, eight construction types, and three degrees of memory cost on language processing. To annotate GELP, we first conduct a crowdsourcing experiment whose design follows prior psycholinguistic studies. Our model evaluation against the annotated GELP then reveals that the full model as well as models with fewer layers and/or self-attention heads exhibit a good-enough performance. This result suggests that models with shallower depth and fewer heads can learn good-enough language processing.

Daiki Asami, Saku Sugawara
arXiv:2406.03666 · cs.CL · submitted Jun 6, 2024
abstract · pdf · html · To appear in Findings of ACL2024

add comment on HN