This document classifies a family of “impossible objects” created by Kokichi Sugihara that are real physical objects but appear to be impossible due to optical illusions and organizes them into a family tree according to the reasons why they look impossible. There are two fundamental sources of ambiguity in reconstructing 3D objects from 2D images that allow us to create impossible objects. One is the lack of depth information in images. The other is the freedom in choosing the viewpoint from which we see the images. The proposed family tree will provide a new framework for further research on impossible objects and 3D optical illusions.