Abstract: In this paper we consider the problem of 3D object recognition and the role that perceptual grouping processes must play. In particular, we argue that reliance on a single level of perceptual grouping is inadequate, since it is responsible for the specific weaknesses of several well-known recognition techniques. Instead, recognition must use a hierarchy of perceptual grouping processes. We describe an appearance-based system that uses four distinct levels of perceptual grouping to represent 3D objects in a form that allows not only recognition, but reasoning about 3D manipulation of a sort that has been supported in the past only by 3D geometric models. The results of the algorithms have been previously reported, and the main contribution of this paper is the development of the perceptual organization hierarchy.