about
Big Mood: Relating Transformers to Explicit Commonsense Knowledge (arxiv.org)
3 points by sel1 on Oct 19, 2019 | hide | past | pdf | 1 comment on HN

In plain words: It blends a language model's word meanings with facts from a commonsense knowledge graph, matching words that don't line up so the facts sharpen each word's sense. The mix answered questions more accurately than the language model alone, placing fifth in the shared task.

Abstract · BIG MOOD: Relating Transformers to Explicit Commonsense Knowledge

We introduce a simple yet effective method of integrating contextual embeddings with commonsense graph embeddings, dubbed BERT Infused Graphs: Matching Over Other embeDdings. First, we introduce a preprocessing method to improve the speed of querying knowledge bases. Then, we develop a method of creating knowledge embeddings from each knowledge base. We introduce a method of aligning tokens between two misaligned tokenization methods. Finally, we contribute a method of contextualizing BERT after combining with knowledge base embeddings. We also show BERTs tendency to correct lower accuracy question types. Our model achieves a higher accuracy than BERT, and we score fifth on the official leaderboard of the shared task and score the highest without any additional language model pretraining.

Jeff Da
arXiv:1910.07713 · cs.CL · submitted Oct 17, 2019
abstract · pdf · html · Accepted to EMNLP Commonsense (COIN)

add comment on HN

Hey, this is my paper! Thanks for sharing :)