What to watch out for

There are several things to keep in mind when watching the surge of robot demonstration videos and even livestreams. First, such robotic demonstrations are not necessarily indicative of robots operating autonomously without human control or oversight, said Dipam Patel, a PhD candidate in computer science at Purdue University and a research assistant at the US Army DevCom Army Research Lab. Many demonstrations still rely on human operators directly controlling the robots’ actions through teleoperation.

“Unless a research paper or a company is explicitly mentioning that [the robot] is completely autonomous, you should take it with a very big pinch of salt,” Patel told Ars.

Another question to consider is whether the demonstration shows robots tackling a completely new test environment for the first time, or whether the robots are simply repeating a task they had already learned to do in that specific training environment. The new test environment would be significantly more impressive at showcasing robots capable of doing tasks autonomously in a generalized way, Patel said.

It is also worth checking the video playback speed for any robot demonstration, because “usually the robots are very slow” for safety and other reasons, Patel said. Companies may sometimes disclose that a robot demonstration video is running at two times or four times normal speed—meaning the robot could be taking twice as long or four times as long as a human to do the same task.

Robot demonstration videos can also vary wildly in their informative value and transparency. Some are clearly intended to be performative entertainment clips that can go viral on social media, or polished promotional videos from companies seeking new clients and investors. Others may provide more of a behind-the-scenes look at the robot training process while acknowledging robot mistakes along the way.

But even if a robot demo video appears incredibly impressive and authentic while coming from a more reputable company or research lab, just keep in mind that it’s still a small glimpse of the bigger picture. The real indicators of progress in robotic capabilities are not so easily packaged for Internet audiences.