about
Does Pretraining Teach Chemical Language Models Chemistry? (arxiv.org)
3 points by yz-exodao 13 days ago | hide | past | pdf | discuss on HN

In plain words: They tested whether pretraining on molecules teaches chemistry by comparing pretrained chemical language models across 64 property-prediction tasks with classical-feature baselines. More pretraining brought no gains and stayed below those baselines; only pretraining on physicochemical descriptors helped, and some was worse than starting untrained.

Abstract

Chemical language models (CLMs) are pretrained on large corpora of molecules with the aim of learning representations that transfer to diverse downstream tasks. Here, we systematically study the effects of pretraining using a series of such CLMs across 64 molecular property prediction datasets. We find that the downstream gains of pretraining in CLMs do not scale and performance saturates below gradient-boosting baselines. Across the regimes evaluated, only supervised pretraining on traditional physico-chemical descriptors consistently improves downstream performance for the CLMs tested whereas some forms of pretraining result in models worse than a randomly initialized encoder. While models can resolve molecular structure well, they do so no better than a randomly initialized model, suggesting that, at best, CLMs learn descriptor-like properties in addition to low-level token statistics during pretraining, but surprisingly little chemistry beyond what classical molecular features already capture.

Leif Sieben, Yoel Zimmermann
arXiv:2610.18651 · cs.LG, cs.CL, physics.chem-ph · submitted Oct 13, 2026 · updated Oct 14, 2026
abstract · pdf · html

add comment on HN