Multimedia support for the article

Animat Vision: Active Vision in Artificial Animals

by Demetri Terzopoulos and Tamer F. Rabie

Notes: Click on any image in this document to view a larger version. Try free-fusing the stereo images and stereo MPEG () movies by crossing your eyes.


Artificial Fishes in their Virtual World

The artificial fishes in their physics-based virtual world as it appears to an underwater observer. The 3 reddish fish (center) are engaged in mating behavior, the greenish fish (upper right) is a predator hunting for small prey, the remaining 3 fishes are feeding on plankton (white dots). Seaweeds grow from the ocean bed and sway in the current.

More images courtesy of Xiaoyuan Tu:

Here are (low-resolution) movie clips of artificial fishes:

movie clip from the animation ``Go Fish!'' by X. Tu, D. Terzopoulos, E. Fiume, in ACM SIGGRAPH Video Review Issue 91: SIGGRAPH'93 Electronic Theater, Anaheim, CA, August, 1993;

QuickTime movie including clips from ``Go Fish!'' and from the animation ``A National Geo-Graphics Society Special: The Undersea World of Jack Cousto'' by X. Tu, D. Terzopoulos, R. Grzeszczuk, presented at the SIGGRAPH'95 Electronic Theater Evening Program, Los Angeles, CA, August, 1995.

Retinal Imaging

The fish animat is equipped with a pair of mobile eyes and is capable of color, binocular vision. The stereoscopic view at the right is acquired by the animat as it gazes at a red fish in the distance.

The animat has foveated vision. Its retinas have a 64x64 high-acuity fovea surrounded by three annular peripheral regions with progressively diminishing resolution (hence the increasing image blockiness towards the periphery). The picture to the left reveals the structure of a retinal image by demarcating the peripheral regions with white borders.

Active Vision Based Sensorimotor Control

The task of vision-guided navigation in a dynamic environment serves to illustrate the active vision abilities of the fish animat. Despite their limited quality, MPEG movies can better convey the dynamic scenarios. Here is an movie demonstrating the pursuit of the red target fish by the observer animat (centered in the video frame) and an movie of the moving target as seen by the eyes of the pursuer. The red fish tries to get away, but the observer can maintain a hot pursuit by keeping the target foveated and using controlled muscle actions to steer the body in accordance with the gaze direction. These MPEGs give an idea of the performance of early versions of the animat vision algorithms.

The image at the right shows the format of the ensuing MPEG movies which are more recent and better demonstrate the robustness of the animat vision system (note: these large 640x480 MPEGs will run at less than the normal frame rate on most computers). The top half of the frame shows the left and right retinal images of the observer animat. At the lower left is a side view of the action, with the observer at the center of the frame. At the lower right is a top view of the action. The line emanating from each eye of the observer indicates the direction of the gaze. The line is colored white out to the estimated range of the foveated target and yellow thereafter.

The next demonstration shows an observer animat that first detects and foveates a reddish fish and then applies visually-guided navigation to swim towards this target fish and follow it. Here is an movie demonstrating active-vision-based sensorimotor control in the animat.

The retinal image stabilization process of the animat vision system implements an optokinetic reflex for the eye muscles that maintains the stability of the visual world. To demonstrate the difference between stabilized and unstabilized vision during locomotion, let's make the observer animat strabismic by locking the gaze angle of its left eye, while the right eye continually performs saccadic eye movements under control of the optical flow based retinal image stabilization process. Here is an movie demonstrating a fish animat with an inactive left eye and a right eye that actively compensates for the pitching and yawing of the swimming body.

Selective Visual Attention and Behavior

We are extending the animat vision system to support other behaviors in addition to the pursuit of moving targets. We have implemented a module that integrates motion and color for selective attention. For example, the next demonstration shows an observer animat foveating and tracking a red target fish. The observer is also alert to interesting colorful objects moving in its visual periphery. Once it detects another target worth looking at, in this case a bluish target fish that enters its peripheral vision, it foveates the recognized object to obtain a high resolution view and recognize the target. This perceptual processing enables the observer to produce the appropriate behavioral response; for example, if the detected object is food, the animat pursues, or if the object is a predator, the animat attempts escape. Here is an movie demonstrating selective attention. The observer shifts attention from the reddish target to the bluish target at frame number 167 and pursues the new target.

The animat vision system can now also support predation and feeding behavior. The following movies show a large bluish predator fish employing active vision to detect and foveate a small reddish prey fish and swim towards the moving prey using visually-guided navigation. As the picture on the right shows, when the predator manages to get sufficiently close to the prey, it violently sucks the prey into a gaping mouth and devours it. Here is an movie demonstrating the vision-guided predation behavior, as well as an movie showing what the predator sees during the predation behavior sequence.


Demetri Terzopoulos