For several years now, Andrew Straw and colleagues at Albert-Ludwigs-University Freiburg have been doing interesting development of Virtual Reality for non-human species. This work applies the basic idea of (mainly visual) Virtual Reality for non-humans.
This is trickier than it sounds, because VR depends on a deep understanding of the subjective experience of the world through vision. For humans, we have both research and extensive experience to guide development. For other species, we have no experience and it is much more difficult to understand the individuals subjective experiences.
At the same time, if you can make VR work, it is has a lot of advantages for learning about non-human perception. Conventional experiments require the animal to be restrained while test stimuli are presented. Even if not uncomfortable, this is an unnatural situation, and precludes a fully natural response to the stimuli.
An all round VR experiment that allows the animal to move naturally is a much more realistic situation, that can elicit normal behavior. In addition, a VR space can be configured and reconfigured in many ways, and can include an extensive virtual world. These capabilities open the way for extensive experimentation without seriously harming the animals.
Building on earlier work which created a VR experience for flies, the research group has extended and generalized the system so it can be used with other species. In their recent paper they report on experiments with flies, mice, and fish [2]. (Land, sea, and air—get it?)
The FreemoVR system exploits contemporary 3D video game technology to rapidly render a realistic scene all around the animal. It also uses computer vision to non-invasively track the position of the animal. The system rapidly renders the correct perspective view of the virtual world as the animal moves.
To prove out the system with different species, a species-appropriate VR world has to be created for each.
For a given species, the virtual world needs to be designed that reflects the natural environment of the animal, and that is rendered for their sensory apparatus. A fly’s world is different from a fish, and their eyes see differently.
The computer vision also needs to be trained to recognize the body pose and motion for each species. The VR depends on accurately tracking both the position and where the animal is looking.
In earlier work, they showed how the system can reveal how the fly uses visual cues to navigate. The current work illustrates other creative experiments. For instance, the fish were presented choices in the form of “teleportation” ports, which instantly shifted the fish to a new scene. (Apparently, this didn’t distress the fish as much as it would upset me!)
This is a classic single user VR system that presents the world registers to one point of view. It isn’t suitable for experiments with multiple animals at the same time, because the viewpoint is correct only for one of them. It is, as they say, a CAVE for animals.
However, they are able to present some group or even “social” situations, by projecting other animals nearby. And, in the case of the fish, they simulate a school of fish, and the subject swims along with them. These effects make it possible to explore interactions, at least based on visual cues.
Indeed, they also presented a world full of cartoonish “space invaders”, which did seem to worry the fish a bit.
The technology is open source, but kind of complicated, building on video game VR and computer vision libraries. The also use Robot Operating System (ROS) as the framework, presumably because it is a modular real time operating system.
Cool stuff!
- Charles Q. Choi, Virtual Reality Platform Created For Lab Animals, in IEEE Spectrum – The Human OS. 2017. http://spectrum.ieee.org/the-human-os/computing/hardware/virtual-reality-platform-created-for-lab-animals
- John R. Stowers, Maximilian Hofbauer, Renaud Bastien, Johannes Griessner, Peter Higgins, Sarfarazhussain Farooqui, Ruth M. Fischer, Karin Nowikovsky, Wulf Haubensak, Iain D. Couzin, Kristin Tessmar-Raible, and Andrew D. Straw, Virtual reality for freely moving animals. Nat Meth, advance online publication 08/21/online 2017. http://dx.doi.org/10.1038/nmeth.4399