4DGS World Foundation Model Scene Integration with Relighting

Built by Adam Chernick, this experiment explores how AI-generated 3D worlds can provide dynamic lighting for streamed 4D Gaussian Splat (4DGS) performances.

The demo combines a World Labs generated 3DGS environment with a streamed Gracia 4DGS performer in a single browser scene. The generated world is then used to relight the volumetric capture, creating a more cohesive and immersive result.

Highlights

The project investigates how AI-generated spatial environments can drive contextual relighting for volumetric human performance and beyond.

• World Foundation Model generated 3DGS environments
• Streamed 4DGS human performance playback
• Runtime cube map capture from the generated world
• Spherical harmonics relighting applied to the 4DGS volumetric capture
• Interactive scene switching, navigation, and lighting controls

The goal is to test a simple creative workflow where a generated world does not just surround a performer, but actively informs how that performer is lit, creating a cohesive and dynamic immersive environment.

How It Works

A World Labs 3DGS scene is rendered in the browser using Spark.js, with a streamed Gracia 4DGS performer added as a dynamic volumetric layer.

When a scene loads, the application captures the surrounding environment into a low-resolution cubemap, converts it into spherical harmonics coefficients, and feeds those into Gracia's lighting system. This allows the performer to inherit low-frequency lighting from the generated world, producing contextual relighting that updates as the environment changes.

The result is a lightweight browser demo where AI-generated worlds and volumetric performances coexist and share lighting information in real time.

Tech Stack

• World Labs / Marble — AI-generated 3DGS environments
• Spark.js — Browser-based Gaussian Splat rendering
• Gracia Web SDK — Streamed 4DGS performer playback
• Three.js — Scene orchestration, camera controls, and cubemap capture
• Spherical Harmonics - Environment-based relighting

What’s Next

• Extended contextual worlds for integration with larger scale 4DGS volumetric captures.
• Pipeline to generate contextual worlds dynamically based on 4DGS asset input.
• Deeper experiments at the intersection of 4DGS and World Foundation Models.