Diagramme von Gesten. Eine zeichentheoretische Analyse digitaler Bewegungsspuren

Abstract

Summary. This paper presents a semiotic investigation of how diagrams are produced and used in empirical, motion-capture based gesture research for the purposes of repre­sentation and analysis of co-speech gestures. First, we discuss the notion of diagram, drawing on Peircean semiotics and more recent accounts of diagrammatic thinking, and traditional ways of recording and representing co-speech gestures. We then employ analytic tools stemming from semiotic theory to a) account for the notational procedu­res used in motion-capture technology and b) offer a diagrammatic interpretation of the signs resulting from them. Our hypothesis is that motion-capture diagrams are products of operative diagrammatic processes and that they are comparable with empirical mea­surements. The “epistemic force of the trace” (Krämer 2011) consists in the iconic visu­alization of movement trajectories which become perceptual gestural gestalts through projecting an indexical sign onto a coordinate system that virtually structures gesture space. We further highlight diagrammatic processes underpinning co-speech gestures themselves, which may be visualized as holistic gestalts with the help of optical moti­on-capture technology. These cognitive-semiotic processes manifest themselves dyna­mically and spatially through embodied image schemas, relational representations of abstract concepts, and gestural diagrams based on mental maps (for example, when conversational partners are jointly planning a journey).