ImmerseSketch: Transforming Creative Prompts into Vivid 3D Environments in VR

Alfred Lan, Tai-Chen Tsai, Chih-Chuan Huan, Pu Ching, Tse-Yu Pan, Min-Chun Hu

National Tsing Hua University, National Taiwan Universtiy of Science and Technology
SIGGRAPH 2024 Posters

Demo video showcasing ImmerseSketch.

Abstract

We propose ImmerseSketch, a framework designed to transform creative prompts into vivid and detailed 3D content within a Virtual Reality (VR) environment. Our aim is to inspire creative ideation and to provide creators with 3D reference content for painting. However, the computational demands of diffusion models for 3D model generation and the scarcity of high-quality 3D datasets limit the diversity and intricacy of the generated environments. To overcome 0these obstacles, we focus on generating initial panoramic images through diffusion models and then converting these images into rich 3D environments with the aid of depth estimation. The pilot study shows that the proposed ImmerseSketch can provide an immersive environment and assist the process of creation.

Overall framework of our system. The framework is seperated into 2 parts, the Unity-based user interface, and the remote computer that handles intensive calculations. Commands are sent from the user computer to generate assets on the remote computer. Generated assets are sent back to the user computer to be displayed.

Our method of generating our drawing environment mesh. We first perform super resolution on a panorama generated from user-given prompt to generate more detail. We lower the resolution again to lower the file size of the generated environment mesh. Then combined with a depth map attained from depth estimation, convert the panorama into a 3D mesh.

Photo of a participant testing out ImmerseSketch during our pilot study.

Sketches drawn by participants during our pilot study. Results show that participants were influenced by the given environment and themes, yet still diverse in style and subject. Showcasing our system's assistance in the participants' artistic process.

Poster

BibTeX


@incollection{lan2024immersesketch,
  title={ImmerseSketch: Transforming Creative Prompts into Vivid 3D Environments in VR},
  author={Lan, Alfred and Tsai, Tai-Chen and Huang, Chih-Chuan and Ching, Pu and Pan, Tse-Yu and Hu, Min-Chun},
  booktitle={ACM SIGGRAPH 2024 Posters},
  pages={1--2},
  year={2024}
}