about
Graph Masked Language Models (arxiv.org)
1 point by PaulHoule on Mar 25, 2025 | hide | past | pdf | discuss on HN

In plain words: A text encoder and a graph network that knows relation types swap information and are trained to match each node's text and graph views. On five graphs where linked nodes differ, this pairing beat graph-only and text-only systems on four, by over 8% on one.

Abstract · GMLM: Bridging Graph Neural Networks and Language Models for Heterophilic Node Classification

Integrating Pre-trained Language Models (PLMs) with Graph Neural Networks (GNNs) remains a central challenge in text-rich heterophilic graph learning. We propose a novel integration framework that enables effective fusion between powerful pre-trained text encoders and Relational Graph Convolutional Networks (R-GCNs). Our method enhances the alignment of textual and structural representations through a bidirectional fusion mechanism and contrastive node-level optimization. To evaluate the approach, we train two variants using different PLMs: Snowflake-Embed (state-of-the-art) and GTE-base, each paired with an R-GCN backbone. Experiments on five heterophilic benchmarks demonstrate that our integration method achieves state-of-the-art results on four datasets, surpassing existing GNN and large language model-based approaches. Notably, Snowflake-Embed + R-GCN improves accuracy on the Texas dataset by over 8\% and on Wisconsin by nearly 5\%. These results highlight the effectiveness of our fusion strategy for advancing text-rich graph representation learning.

Aarush Sinha
arXiv:2503.05763 · cs.CL, cs.AI, cs.LG · submitted Feb 24, 2025 · updated Oct 8, 2025
abstract · pdf · html

add comment on HN