Scaling Robotic Simulation with Marble

Overview

High-quality, diverse simulation data is one of the biggest limiting factors in robotics research. Training robots to perceive, move, and interact safely requires thousands of environments — yet building those environments by hand is slow, expensive, and often inconsistent.

Researchers Hang Yin and Abhishek Joshi set out to explore a new approach: using Marble’s world-generation technology to automatically create scalable, physically accurate 3D scenes for simulation and data collection.

Both Hang and Abhishek are active researchers in robotics and embodied AI. Hang’s work focuses on maintaining the BEHAVIOR-1K project and the OmniGibson simulator, while Abhishek has contributed to major simulation frameworks such as RoboSuite and Infinigen-Articulated (previously Infinigen-Sim).

Their combined experiments demonstrate how Marble can serve as a generative foundation for robot training — combining rapid world creation with the realism and physical fidelity needed for advanced simulation research.

Reimagining Robot Simulation

For decades, robotics simulation has relied on manually curated environments — warehouses, kitchens, offices — each painstakingly modeled, textured, and tested. These environments are vital for robot learning but impossible to scale at the pace AI models demand.

Marble changes that. By generating 3D worlds directly from text or image prompts, it allows researchers to create thousands of photorealistic scenes quickly. Each scene includes depth, lighting, and geometry data, along with an exportable collider mesh for accurate physical interaction.

This means researchers can test perception, planning, and control algorithms across unlimited visual and structural variations — essential for domain randomization, the practice of training robots in diverse synthetic worlds to improve performance in the real one.

Project Demonstrations

Abhishek Joshi — Kitchen Teleoperation & Scalable Simulation Data

Abhishek’s work explored how Marble-generated environments could be used to scale simulation datasets for robot learning. He focused on an indoor teleoperation scene featuring a kitchen workspace designed for manipulation research.

Marble generated the static kitchen environment — including the architectural layout of the Gaussian splat. Separately, 3D models such as the robot, pan, boxes, microwave, and trash can were imported from external tools such as Infinigen and RoboSuite.

A robotic manipulator was remotely controlled to interact with these objects — traversing the kitchen environment and completing structured tasks. The Marble scene was exported with a full collision mesh and imported into MuJoCo. From there, Abhishek constructed an environment in RoboSuite using the Marble scene and controlled a robot to complete a long-horizon manipulation task.

The demonstration was then exported to Blender and rendered with the Gaussian splat scene.

As a contributor to RoboSuite, Abhishek compared this process to traditional simulation authoring, where a single high-quality environment can take weeks to curate and validate. With Marble, he generated multiple usable worlds in hours while maintaining photorealistic fidelity and valid collision meshes.

Hang Yin — Legged Locomotion & Industrial Manipulation

Hang conducted two experiments to explore how Marble-generated environments can enhance robotic training and evaluation inside NVIDIA Isaac Sim.

Legged Locomotion & Multi-Robot Awareness

In the first demo, an ANYmal quadruped navigates a Marble-generated house, inspecting furniture and navigating obstacles while visualizing its perception pipeline — from object detection through depth and collision mapping.

A Spot robot performs a parallel inspection task in the kitchen, demonstrating multi-robot awareness as both agents share spatial understanding within the same environment.

Marble’s PLY splats and GLB colliders were converted to USD/USDZ formats and imported into Isaac Sim, where Omniverse RTX Neural Rendering (NuRec) capabilities enabled real-time visualization of the 3D Gaussian Splats alongside physical interaction.

The result is a visually rich, simulation-ready world that supports both performance benchmarking and research on collaborative autonomy.

Industrial Manipulation

The second demo shows a UR10 robotic arm completing a bin-stacking task inside a Marble-generated warehouse.

The warehouse scene and layout were created entirely using Marble, while the robot, conveyor belt, and boxes were external assets manually added later in Isaac Sim.

A GLB collider mesh exported from Marble provided accurate contact physics, allowing Hang to simulate grasping and stacking behaviors under realistic spatial conditions.

The workflow demonstrated how Marble scenes can serve as reusable, high-fidelity backdrops for industrial robotics research.

Why Marble

Both researchers identified the same core advantages that make Marble uniquely suited for robotics research:

• Rapid world generation: Create thousands of simulation-ready environments from minimal input.
• Physical accuracy: Exported mesh-based colliders enable true physical interaction with generated geometry.
• Cross-platform integration: Works seamlessly with Isaac Sim, MuJoCo, Omniverse RTX, and RoboSuite.
• Visual variety: Built-in randomization of lighting, materials, and scene structure for domain diversity.
• Scalability: Reduces manual environment creation cost and time by orders of magnitude.

For both Hang and Abhishek, Marble proved that world generation can become as automated as model training — shifting robotics from hand-built datasets to described datasets defined by scene parameters and constraints.

Technical Workflow

Both projects followed similar pipelines, adapted for different simulators:

Scene Generation: Text or image prompts used to generate diverse indoor and industrial worlds in Marble.

Scene Composition: Larger environments assembled in Marble Composer, then exported as PLY splats and GLB collider meshes.

Simulation Integration:
Hang Yin: Converted PLY/GLB → USD/USDZ and integrated into Isaac Sim, leveraging Omniverse RTX's NuRec rendering.
Abhishek Joshi: Integrated the scene into MuJoCo and RoboSuite, then used a USD bridge to export results into Blender for visualization.

Task Setup:
UR10 bin stacking (industrial manipulation)
ANYmal + Spot locomotion and awareness
Franka Panda robotic arm teleoperation (kitchen scene)

Evaluation: Teleoperation control and policy inference performed across generated environments; perception pipelines visualized and logged for later analysis.

This end-to-end flow — from world generation to simulation — can now be completed in hours, compared to days or weeks with traditional 3D curation.

Results & Impact

Marble significantly improved efficiency and data diversity for both researchers.

• Faster iteration: New simulation worlds generated and tested within minutes.
• Diverse data at scale: Thousands of unique variations for perception and control tasks.
• Reduced human effort: Environment curation time reduced by over 90%.
• Improved sim-to-real transfer: Photorealistic geometry and lighting enable more accurate domain alignment.

The results point to a broader opportunity: controlled, semantic world generation. Future versions of Marble could let researchers define scene-level constraints — such as “a cluttered kitchen with open drawers” or “an office corridor with two robots present” — and automatically generate hundreds of meaningful variations.

Looking Ahead

Both researchers plan to extend their Marble-based workflows in upcoming experiments:

• Articulated environments: Support for functional doors, drawers, and tools to enable interaction learning.
• Semantic scene control: Defining object-level parameters and constraints for structured data generation.
• Multi-room and outdoor worlds: Testing long-horizon navigation and collaborative robot scenarios.
• Automated variation pipelines: Procedurally generating benchmarks for vision, planning, and control.

For Abhishek and Hang, Marble represents a step toward autonomous simulation data generation — where instead of designing environments, researchers describe them.