Microsoft Controls Robots with ChatGPT

Microsoft has been testing the use of OpenAI’s language model, ChatGPT, to enable remote control of household and industry assistive robots.

The trials aim to create a new and more streamlined means of achieving human-robot interactions without the need for users to learn programming languages.

“Our goal with this research is to see if ChatGPT can think beyond text and reason about the physical world to help with robotics tasks,” the company wrote in a blog post. Using the chatbot, the team said it was able to control "multiple platforms such as robot arms, drones, and home assistant robots intuitively with language.”

Currently, robotic commands are programmed by an engineer who has to write specific code for each desired action and system. Such a process is often time-consuming and costly and excludes users who may not understand code. Using ChatGPT, code can be generated automatically, responding to specific scenarios as they occur.

In trials, the team used ChatGPT to control a drone, write code and solve robotics puzzles.

“ChatGPT…proved to be an extremely intuitive language-based interface between the non-technical user and the robot,” the team wrote. “ChatGPT asked clarification questions when the user’s instructions were ambiguous and wrote complex code structures for the drone such as a zig-zag pattern to visually inspect shelves.”

The team identified a challenge in teaching ChatGPT to understand and take into account a robot’s physical presence and how its actions change the world around it.

Microsoft developed a series of design principles in response to this challenge, including special prompting structures, high-level APIs, and human feedback via text. Using these design principles, the researchers could guide ChatGPT’s understanding of its surroundings and the task in front of it.

Microsoft also announced the introduction of PromptCraft, a “collaborative open-source platform” where the company has invited people to share examples and tips for prompting strategies for different robotics categories.

“Besides prompt design, we hope to also include multiple robotics simulators and interfaces to allow users to test their ChatGPT-generated algorithms,” the team wrote. “As a start, we also release an AirSim environment with ChatGPT integration that anyone can use to get started with these ideas.”