In plain words: A new dataset labels 10.5 million paragraphs from arXiv preprints with 13 kinds of scientific statements, so models can learn to tell them apart. The best tested model reached 0.91 on a 0-to-1 correctness score, matching typical results for this task.
Abstract · Scientific Statement Classification over arXiv.org
We introduce a new classification task for scientific statements and release a large-scale dataset for supervised learning. Our resource is derived from a machine-readable representation of the arXiv.org collection of preprint articles. We explore fifty author-annotated categories and empirically motivate a task design of grouping 10.5 million annotated paragraphs into thirteen classes. We demonstrate that the task setup aligns with known success rates from the state of the art, peaking at a 0.91 F1-score via a BiLSTM encoder-decoder model. Additionally, we introduce a lexeme serialization for mathematical formulas, and observe that context-aware models could improve when also trained on the symbolic modality. Finally, we discuss the limitations of both data and task design, and outline potential directions towards increasingly complex models of scientific discourse, beyond isolated statements.
Deyan Ginev, Bruce R. Miller
arXiv:1908.10993 · cs.CL, cs.AI, cs.DL · submitted Aug 29, 2019
abstract · pdf · html