about
WinClick: GUI Grounding with Multimodal Large Language Models (arxiv.org)
1 point by PaulHoule on Mar 20, 2025 | hide | past | pdf | discuss on HN

In plain words: A Windows helper that looks only at screenshots to find where to click, instead of relying on page code that many apps don't expose. Extra practice at matching instructions to screen spots made it beat earlier tools on a new Windows test set.

Abstract

Graphical User Interface (GUI) tasks are vital for automating workflows such as software testing, user interface navigation. For users, the GUI is the most intuitive platform for interacting with a computer. Previous work identified a key challenge in developing visual GUI agents: GUI grounding - the ability to accurately locate screen elements based on instructions. However, most existing GUI agents rely on structured data formats like DOM or HTML files in training or inferencing, which are inaccessible across all applications, particular in a general desktop environments such as Windows OS. To address this, we introduce WinClick, a novel visual GUI agent developed in Windows platform. WinClick leverages screenshots to detect actionable regions. To overcome the challenge of GUI grounding, we enhance WinClick with GUI grounding pre-training and propose an LLM-based method for aligning GUI grounding data. Additionally, we introduce WinSpot, the first comprehensive benchmark for GUI grounding on Windows. Our experiments demonstrate that WinClick, combined with GUI grounding pre-training, significantly outperforms existing baselines, offering a scalable solution for GUI automation in desktop environments. WinSpot is publicly available at https://github.com/zackhuiiiii/WinSpot.

Zheng Hui, Yinheng Li, Dan zhao, Tianyi Chen, Colby Banbury, Kazuhito Koishida
arXiv:2503.04730 · cs.CL, cs.HC · submitted Jan 27, 2025
abstract · pdf · html

add comment on HN