about
Listen to the Image (arxiv.org)
1 point by stagas on May 14, 2020 | hide | past | pdf | discuss on HN

In plain words: Rather than asking blind people to judge each way of turning pictures into sounds, a computer model turns the sound back into an image and scores the encoding by its accuracy. Across several encodings its ratings matched human tests closely, making tuning faster and cheaper.

Abstract

Visual-to-auditory sensory substitution devices can assist the blind in sensing the visual environment by translating the visual information into a sound pattern. To improve the translation quality, the task performances of the blind are usually employed to evaluate different encoding schemes. In contrast to the toilsome human-based assessment, we argue that machine model can be also developed for evaluation, and more efficient. To this end, we firstly propose two distinct cross-modal perception model w.r.t. the late-blind and congenitally-blind cases, which aim to generate concrete visual contents based on the translated sound. To validate the functionality of proposed models, two novel optimization strategies w.r.t. the primary encoding scheme are presented. Further, we conduct sets of human-based experiments to evaluate and compare them with the conducted machine-based assessments in the cross-modal generation task. Their highly consistent results w.r.t. different encoding schemes indicate that using machine model to accelerate optimization evaluation and reduce experimental cost is feasible to some extent, which could dramatically promote the upgrading of encoding scheme then help the blind to improve their visual perception ability.

Di Hu, Dong Wang, Xuelong Li, Feiping Nie, Qi Wang
arXiv:1904.09115 · cs.CV, cs.HC, cs.MM, cs.SD, eess.AS · submitted Apr 19, 2019
abstract · pdf · html · Accepted by CVPR2019

add comment on HN