about
Screenshot to HTML Code Dataset (arxiv.org)
1 point by radeeyate on Mar 18, 2024 | hide | past | pdf | discuss on HN

In plain words: They built a synthetic set of 2 million screenshots paired with the HTML code that builds each one, and trained a vision model on it to write that code. It rebuilds pages as working HTML, something barely attempted before for lack of training data.

Abstract · Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset

Using vision-language models (VLMs) in web development presents a promising strategy to increase efficiency and unblock no-code solutions: by providing a screenshot or a sketch of a UI, a VLM could generate the code to reproduce it, for instance in a language like HTML. Despite the advancements in VLMs for various tasks, the specific challenge of converting a screenshot into a corresponding HTML has been minimally explored. We posit that this is mainly due to the absence of a suitable, high-quality dataset. This work introduces WebSight, a synthetic dataset consisting of 2 million pairs of HTML codes and their corresponding screenshots. We fine-tune a foundational VLM on our dataset and show proficiency in converting webpage screenshots to functional HTML code. To accelerate the research in this area, we open-source WebSight.

Hugo Laurençon, Léo Tronchon, Victor Sanh
arXiv:2403.09029 · cs.HC, cs.AI, cs.CV · submitted Mar 14, 2024
abstract · pdf · html

add comment on HN