System Enables Human-to-Humanoid Robot Operation

Researchers have developed a new system that allows a human operator to remotely control a humanoid robot, in what the team says is the first known demonstration of whole-body humanoid teleoperation.

The team, from Carnegie Mellon University (CMU) based the system on reinforcement learning. Utilizing only an RGB camera, similar to those found in webcams, the platform translates an operator's actions into the humanoids’ movements. 

A video demonstration shows the system being used to make a humanoid robot from Unitree perform simple tasks including picking and placing items, as well as more dynamic actions such as boxing, kicking a football and pushing a pram.

To create a large-scale dataset of human movements for humanoid robots to copy, the team deployed a scalable "sim-to-data" process to filter and identify feasible motions using a privileged motion imitator. 

“Afterwards, we train a robust real-time humanoid motion imitator in simulation using these refined motions and transfer it to the real humanoid robot in a zero-shot manner,” the team wrote. “We successfully achieve teleoperation of dynamic whole-body motions in real-world scenarios, including walking, back jumping, kicking, turning, waving, pushing, boxing, etc.

“To the best of our knowledge, this is the first demonstration to achieve learning-based real-time whole-body humanoid teleoperation.”

Related:Humanoid Robot Shows off Boxing Skills

The team said the design marks a breakthrough in teleoperation and opens the door to more streamlined human-to-robot interactions and control. 

Writing on X, formerly Twitter, lead researcher Tairan He said the framework “holds the key for collecting large-scale state-action pairs to train sensorimotor models for autonomous agents.”

“In the future, we can remotely teleoperate a humanoid to perform chores, dangerous missions and complex tasks that require a human-like level of dexterity and adaptability,” he wrote.