Star Trek Holodeck Recreated Using ChatGPT

Researchers from the University of Pennsylvania have recreated Stark Trek’s holodeck – a program capable of generating immersive, interactive 3D environments – using ChatGPT.

While in Star Trek the Holodeck is used to prepare crew members for missions or for entertainment, this study targeted virtual interactive environments for simulation training for robots before real-world deployment.

Creating complex, detailed virtual worlds is currently a labor- and time-intensive process, typically requiring artists to manually create the environments. 

As such, the research team pitched using AI and large language models as a viable alternative, using billions of data points to rapidly create complex worlds. A user simply describes the environment they want and the AI agents interpret and respond to the request. 

The Holodeck then fills the space with virtual objects from Objaverse, a library of premade digital objects. 

“Generative AI systems like ChatGPT are trained on trillions of words and image generators like Midjourney and DALL.E  are trained on billions of images,” said Chris Callison-Burch, study co-author. “We only have a fraction of that amount of 3D environments for training so-called ‘embodied AI’.

“If we want to use generative AI techniques to develop robots that can safely navigate in real-world environments, then we will need to create millions or billions of simulated environments.” 

Related:Nvidia Expands Robotic Simulation Platform

The novel Holodeck was created in collaboration with researchers at Stanford, the University of Washington and the Allen Institute for Artificial Intelligence (AI2).

To evaluate the Holodeck’s realism and accuracy, the researchers generated 120 scenes using both Holodeck and ProcTHOR, a tool created by AI2 that uses human-created rules rather than AI-generated text. Several hundred Penn Engineering students were then asked which version they preferred.

For every criterion — object selection, layout coherence and overall preference — the students consistently rated the Holodeck-generated environments higher.  

Finally, the team tested the Holodeck’s capabilities as a testbed for a robotic agent.

“The ultimate test of Holodeck,” said Mark Yatskar, study co-lead, “is using it to help robots interact with their environment more safely by preparing them to inhabit places they’ve never been before.”

The robotic simulation was deployed across a range of virtual environments, including offices, gyms and arcades and was found to be able to navigate across each new space with ease.

The team is set to present Holodeck in June at the 2024 Institute of Electrical and Electronics Engineers and Computer Vision Foundation Computer Vision and Pattern Recognition Conference in Seattle, Washington.

Related:Amazon Creates Human-Robot Simulation Platform