3D Generation
GaussianCity
Generative Gaussian Splatting for Unbounded 3D City Generation Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, Ziwei Liu S-Lab, Nanyang Technological University visitors arXiv 2406.06526 🤗 Hugging Face Featured Space Stars TL;DR: GaussianCity is a framework for efficient unbounded 3D city generation using 3D Gaussian Splatting. Abstract 3D city generation with NeRF-based methods shows promising generation results but is computationally inefficient. Recently 3D Gaussian Splatting (3D-GS) has emerged as a highly efficient alternative for object-level 3D generation. However, adapting 3D-GS from finite-scale 3D objects and humans to infinite-scale 3D cities is non-trivial. Unbounded 3D city generation entails significant storage overhead (out-of-memory issues), arising from the need to expand points to billions, often demanding hundreds of Gigabytes of VRAM for a city scene spanning 10km2. In this paper, we propose GaussianCity, a generative Gaussian Splatting framework dedicated to efficiently synthesize unbounded 3D cities with a single feed-forward pass. Our key insights are two-fold: 1) Compact 3D Scene Representation: We introduce BEV-Point as a highly compact intermediate representation, ensuring that the growth in VRAM usage for unbounded scenes remains constant, thus enabling unbounded city generation. 2) Spatial-aware Gaussian Attribute Decoder: We present spatial-aware BEV-Point decoder to produce 3D Gaussian attributes, which leverages Point Serializer to integrate the structural and contextual characteristics of BEV points. Extensive experiments demonstrate that GaussianCity achieves state-of-the-art results in both drone-view and street-view 3D city generation. Notably, compared to CityDreamer, GaussianCity exhibits superior performance with a speedup of 60 times (10.72 FPS v.s. 0.18 FPS).
CityDreamer
CityDreamer: Compositional Generative Model of Unbounded 3D Cities Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, Ziwei Liu S-Lab, Nanyang Technological University visitors arXiv 2309.00610 🤗 Hugging Face Stars TL;DR: CityDreamer learns to generate unbounded 3D cities from Google Earth imagery and OpenStreetMap. Abstract In recent years, extensive research has focused on 3D natural scene generation, but the domain of 3D city generation has not received as much exploration. This is due to the greater challenges posed by 3D city generation, mainly because humans are more sensitive to structural distortions in urban environments. Additionally, generating 3D cities is more complex than 3D natural scenes since buildings, as objects of the same class, exhibit a wider range of appearances compared to the relatively consistent appearance of objects like trees in natural scenes. To address these challenges, we propose CityDreamer, a compositional generative model designed specifically for unbounded 3D cities, which separates the generation of building instances from other background objects, such as roads, green lands, and water areas, into distinct modules. Furthermore, we construct two datasets, OSM and GoogleEarth, containing a vast amount of real-world city imagery to enhance the realism of the generated 3D cities both in their layout and appearance. Through extensive experiments, CityDreamer has proven its superiority over state-of-the-art methods in generating a wide range of lifelike 3D cities.