about
Unifying Point Cloud Perception, Generation and Editing with LLMs (arxiv.org)
2 points by PaulHoule on Feb 13, 2024 | hide | past | pdf | discuss on HN

In plain words: A language model turns 3D point clouds into one shared form, so it can understand a scene, create objects in it, and edit them by following plain-language commands. Tests showed it handled these jobs well, and adding scene perception improved its generation and editing.

Abstract · Uni3D-LLM: Unifying Point Cloud Perception, Generation and Editing with Large Language Models

In this paper, we introduce Uni3D-LLM, a unified framework that leverages a Large Language Model (LLM) to integrate tasks of 3D perception, generation, and editing within point cloud scenes. This framework empowers users to effortlessly generate and modify objects at specified locations within a scene, guided by the versatility of natural language descriptions. Uni3D-LLM harnesses the expressive power of natural language to allow for precise command over the generation and editing of 3D objects, thereby significantly enhancing operational flexibility and controllability. By mapping point cloud into the unified representation space, Uni3D-LLM achieves cross-application functionality, enabling the seamless execution of a wide array of tasks, ranging from the accurate instantiation of 3D objects to the diverse requirements of interactive design. Through a comprehensive suite of rigorous experiments, the efficacy of Uni3D-LLM in the comprehension, generation, and editing of point cloud has been validated. Additionally, we have assessed the impact of integrating a point cloud perception module on the generation and editing processes, confirming the substantial potential of our approach for practical applications.

Dingning Liu, Xiaoshui Huang, Yuenan Hou, Zhihui Wang, Zhenfei Yin, Yongshun Gong, Peng Gao, Wanli Ouyang
arXiv:2402.03327 · cs.CV, cs.AI, cs.CL · submitted Jan 9, 2024
abstract · pdf · html · 10 pages, 6 figures

add comment on HN