about
Learned Single-Pass Multitasking Perceptual Graphics for Immersive Displays (arxiv.org)
3 points by PaulHoule on Aug 25, 2024 | hide | past | pdf | discuss on HN

In plain words: A single lightweight model takes an image plus a text prompt, like "lightly denoised," and applies the described visual effects in one pass. It keeps quality consistent like chaining separate effect models together, but without the heavy compute and juggling of many models.

Abstract

Emerging immersive display technologies efficiently utilize resources with perceptual graphics methods such as foveated rendering and denoising. Running multiple perceptual graphics methods challenges devices with limited power and computational resources. We propose a computationally-lightweight learned multitasking perceptual graphics model. Given RGB images and text-prompts, our model performs text-described perceptual tasks in a single inference step. Simply daisy-chaining multiple models or training dedicated models can lead to model management issues and exhaust computational resources. In contrast, our flexible method unlocks consistent high quality perceptual effects with reasonable compute, supporting various permutations at varied intensities using adjectives in text prompts (e.g. mildly, lightly). Text-guidance provides ease of use for dynamic requirements such as creative processes. To train our model, we propose a dataset containing source and perceptually enhanced images with corresponding text prompts. We evaluate our model on desktop and embedded platforms and validate perceptual quality through a user study.

Doğa Yılmaz, He Wang, Towaki Takikawa, Duygu Ceylan, Kaan Akşit
arXiv:2408.07836 · cs.CV, cs.GR, eess.IV · submitted Jul 31, 2024 · updated Aug 7, 2025
abstract · pdf · html

add comment on HN