A Robot Puppeteer from ETH

String puppeteering is amazing to me, almost magical.  How do they do it?

I was very interested to see the report from ETH of a robot puppeteer, which works pretty amazingly well [2].  Wow!  How did they do it?

What they didn’t do was design a “theory of puppets”, or a programming language (the “Pinocchio” language?).

What they did do is apply motion planning and physics simulation to rapidly predict the effects of gravity, strings, and the articulation of the puppet.  This planner develops optimal motions of the robot to move the puppet to the desired position.  If I understand correctly, the simulation runs constantly, computing the next one second of motion, to make the optimal moves.

“we devise a predictive control model that accounts for the dynamics of the marionette and kinematics of the robot puppeteer.” ([2], p. 1)

This works amazingly well!

The researchers also can design optimal “paddles” for particular puppets and motions.  The simulation system would also enable the development of new puppets in silico, and it would be possible to modify the design of existing puppets.

The researchers suggest that this planning model could be useful for challenging manipulation tasks, such as cloth, soft parcels, or cables.  (I think this betrays their affiliation with an engineering institute.)

I think it would also be very interesting to see how this model matches the behavior of human experts.  The results look similar, but is this an accurate description of what a human puppeteer “knows”?    As Evan Ackerman points out, expert human puppeteers seem to be a lot better, so what is missing from the simulation?

It is very possible that humans have alternative ways to do it, and if so, it might be interesting to incorporate these “algorithms” into a simulation.  In addition, it would be interesting to see how puppeteers learn the craft, which might offer lessons for the development of the models.  (In other words, this is an opportunity to do some “biomimetic” design mimicing a somewhat mysterious human skill.)

Finally, seeing this simulation made me think about making much more complicated “puppets” by creating puppet puppeteers.  In principle, a computational puppeteer could manipulate a puppet that is attached to another puppet, pulling strings that make other strings pull, etc.

---

1. Evan Ackerman, ETH Zurich Demonstrates PuppetMaster Robot, in IEEE Spectrum – Robotics. 2019. https://spectrum.ieee.org/automaton/robotics/robotics-hardware/eth-surich-puppetmaster-robot
2. Simon Zimmermann, Roi Poranne, James M. Bern, and Stelian Coros, PuppetMaster: robotic animation of marionettes. ACM Transactions on Graphics, 38 (4) July 2019. https://dl.acm.org/citation.cfm?doid=3306346.3323003

 

Robot Wednesday