 |
ML Newsnew | past | best | rss | submit | about |
| Benchmark for Assessing Large Language Models in Knowledge Graph Engineering (arxiv.org) | |
| 1 point by PaulHoule on Sep 5, 2023 | hide | past | pdf | discuss on HN | |
|
In plain words: A test suite checks how well language models handle building knowledge graphs — networks of linked facts — like fixing errors, extracting facts, and creating data, and scores their answers automatically. The tests show these models can't yet build one from a plain prompt alone. Abstract · Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph EngineeringAs the field of Large Language Models (LLMs) evolves at an accelerated pace, the critical need to assess and monitor their performance emerges. We introduce a benchmarking framework focused on knowledge graph engineering (KGE) accompanied by three challenges addressing syntax and error correction, facts extraction and dataset generation. We show that while being a useful tool, LLMs are yet unfit to assist in knowledge graph generation with zero-shot prompting. Consequently, our LLM-KG-Bench framework provides automatic evaluation and storage of LLM responses as well as statistical data and visualization tools to support tracking of prompt engineering and model performance. | |
| add comment on HN |