In plain words: A crawler installs real phone apps and taps and drags elements to see what actually happens, giving it correct labels without human crowd-workers. It has run for over 5,000 device-hours, taking half a million actions across 6,000 apps to train three vision models.
Abstract · Never-ending Learning of User Interfaces
Machine learning models have been trained to predict semantic information about user interfaces (UIs) to make apps more accessible, easier to test, and to automate. Currently, most models rely on datasets that are collected and labeled by human crowd-workers, a process that is costly and surprisingly error-prone for certain tasks. For example, it is possible to guess if a UI element is "tappable" from a screenshot (i.e., based on visual signifiers) or from potentially unreliable metadata (e.g., a view hierarchy), but one way to know for certain is to programmatically tap the UI element and observe the effects. We built the Never-ending UI Learner, an app crawler that automatically installs real apps from a mobile app store and crawls them to discover new and challenging training examples to learn from. The Never-ending UI Learner has crawled for more than 5,000 device-hours, performing over half a million actions on 6,000 apps to train three computer vision models for i) tappability prediction, ii) draggability prediction, and iii) screen similarity.
Jason Wu, Rebecca Krosnick, Eldon Schoop, Amanda Swearngin, Jeffrey P. Bigham, Jeffrey Nichols
arXiv:2308.08726 · cs.HC · submitted Aug 17, 2023
abstract · pdf · html
It's like the UX/UI designers have taken twitch video games as the gold standard for a UI, and therefore the visual acuity and reaction times of a twentysomething are baked-in as prerequisites for operating a simple thing like a mobile phone with your banking application, or a desktop computer viewing a normal news article.
In these cases, it's not sufficient to simply learn the UI, it's necessary to anticipate it, and lead your target so that you're prepared to click it when the opportune moment presents itself and before that moment disappears again.