Well, ChatGPT has been the demo of the month in pretty much every domain you can think of.
So, why not robotics, too?
This winter, researchers at Microsoft Research report experiments using GPTChat to program robots [4]. (Perhaps they will try Bing Chat later. With Bing’s hostile attitude, that may be an interesting experiment!)
The basic idea seems to be, “let’s use general purpose chat to tell a robot what you want it to do”. ChatGPT’s language model is being used to interpret and create natural descriptions of tasks. Behind the scenes, the chat conversation is turned into high level commands to a home robot.
“Wouldn’t it be amazing to just tell your home assistant robot: “Please warm up my lunch“, and have it find the microwave by itself?”
“We believe that language-based robotics control will be fundamental to bring robotics out of science labs, and into the hands of everyday users.”
(from [1])
As the technical report puts it, the key technical concept is “the creation of a high-level function library.” ([4], p.2) (This doesn’t seem like a huge innovation to me. This is kind of what Turtlebots did, way back when.)
What ChatGPT does is kind of “tell a story” of how to do what the user wants. This can be refined in a dialog, to get a (sort of) natural language description of the desired action. This can then be passed along to software that turns this into high level operations (a la Turtlebots).
This is kind of neat, and truly embodies the vision of human computer interaction as a form of collaborative narrative. (This concept was galvanizing when it was put forward way back when, e.g., [3] and [2]).
Glancing at the example, though, I’m struck by the underwhelming, not to say, idiotic, tasks performed. It would be easier to just get a drink myself rather than several iterations of detailed dialog with a robot. And why do I want my drone to “look at my orchid”, if I can already see both the robot and the orchid?
Other examples are slightly more reasonable tasks, but executing them with ChatGPT is, as it’s name implies, pretty darn chatty. Guiding a UAV involves extremely non-natural dialogs about how the UAV is supposed to behave.
E..g., “I want you to make sure to fly only if there is at least 10 meters of distance in front of the drone. If not, you should rotate the drone until there is at least 10 meters of clearance in the front and then take a step in the direction that the drone is currently facing. Don’t forget to face the goal again after taking a step. Got it?”
([4], p.11)
My point is, this general purpose chat bot doesn’t seem to be all that great as an interface to controlling robots. I guess the news is that it works at all, and that it works by iterative dialog, which it uses to disambiguate and clarify the user’s task.
Still, this interface would drive me nuts in a very few minutes.
And, of course, the big question is, what is it that you want a robot for?
They ask: “Wouldn’t it be amazing to just tell your home assistant robot: “Please warm up my lunch“, and have it find the microwave by itself?”
And I answer: For me, no, thanks. Don’t really care. In fact, really don’t want a robot to operate the microwave for me. (Not that it would even fit in my kitchen.)
So basically, this is a really complicated (and probably expensive) way to do something I don’t want to do anyway.
From what we hear of the Microsoft Bing Chat, it would probably tell you to “do it yourself, you lazy b**” or something equally rude. Now that would be amusing… but only once.
- Autonomous Systems and Robotics Group, ChatGPT for Robotics: Design Principles and Model Abilities, in Microsoft Research – Autonomous Systems and Robotics Group -Articles February 20, 2023. https://www.microsoft.com/en-us/research/group/autonomous-systems-group-robotics/articles/chatgpt-for-robotics/
- Myron W. Krueger, Artificial Reality II, Reading, Mass., Addison Wesley, 1991.
- Brenda Laurel, Computers as Theatre, Reading, MA, Addison Wesley, 1991.
- Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor, ChatGPT for Robotics: Design Principles and Model Abilities. Microsoft, 2023. https://www.microsoft.com/en-us/research/publication/chatgpt-for-robotics-design-principles-and-model-abilities/
Robot Wednesday
Robert McGrath's Blog 
One thought on “ChatGPT for Robotics?”