Patterns of synchrony and rhythm

The interest in relating prosody (the patterns of vocalization) to body movement arose from the author’s work in Japan with researchers working on Aizuchi (backchannels) in speech at NTT and ATR for the design of multi-modal interfaces. It was possible to feel the intent in the prosodic qualities of the backchannels and fillers that the Japanese researchers were defining and codifying. The author drew upon (1) this work on backchannels and fillers (Shimojima et al. 1998; Kawamori et al. 1995, 1998) and (2) conversation moves which express how people are understanding what each other are saying, and (3) research on gesture, to reflect on how our bodies mediate knowledge transfer when we communicate. The first study analysed how landscape architects’ bodies mediate knowledge whilst they are sketching a design together. What emerged was a coding system for pragmatic acts of body movement with speech, termed Body Moves, which were identified as having a rhythmic quality.

These salient rhythms could not be fully explained through the conversational feedback model of turn taking in conversation moves and speech backchannels. One of these in particular (Gill et al. 2000) posed a difficulty in using the idea of feedback as it involved both the bodies of the participants moving at exactly the same time and also in time with the voice of one of them. This was called the parallel coordinated move as the bodies were performing different tasks, so they were not imitating each other. This movement that lasted 2 s could be said to be equivalent to the experience of musicians playing together in ‘unison’ (Levinson 2011).

A further sketching study was undertaken to test the existence of these Body Moves (Gill and Borchers 2004). This time participants (undergraduates at Stanford University) working in pairs drew upon a computer-based interface that permitted only one person to touch the surface at a time (SmartBoard). This was expected to block the moment of ‘unison’. Indeed, the parallel coordinated move was inhibited but so was all rhythmic interpersonal synchrony around the surface of the board. This finding suggested that there is a relationship between the various levels and forms of rhythmic and synchronous coordination that needs further investigation.

The other study mentioned above, of non-face-to-face interaction, that compared Japanese and English speakers, gives support to this suggestion of a multi-level framework. The study involved pairs of Japanese participants speaking in Japanese and English participants speaking in English, located in separate rooms and talking through a microphone. The study found that the body movements of each participant are finally tuned to the other’s voice and to their own when they speak. For the Japanese, the movements were predominantly head-nodding actions, and for the British, they involved a variety of body movements including shoulder movements, body sway, and head movements. The study also showed that these rhythms are aligned with the language spoken and that they differ between the two languages. Kita and Ide’s (2007) analysis of Japanese aizuchi (akin to backchannels but occurring on the part of the speaker as well as listener) proposes that any cultural comparison needs to understand how culture itself shapes the patterns of synchronization, and they provide examples that illustrate the need to go beyond the traditional feedback model. A surprising finding in the non-face-to-face condition is that body movements can occur simultaneously in both participants, and this may occur without speech. These occurrences are not arbitrary. The fine tuning of body movement with self and other speech indicates that these movements are for the maintenance of self-synchrony that is needed in order to be synchronized with the other person. Condon and Ogston (1966, 1974) provide detailed evidence for the relation of self (intra)-synchrony with interpersonal synchrony, for human entrainment in the face-to-face condition. And as indicated in Kita et al. (2007) study, cultural differences are key here.

The findings about patterns of synchrony and rhythm from both the studies of collaborative sketching activity and the non-face-to-face communication of Japanese and English speakers suggest a complex picture of movement at multiple levels. It is not clear how a particular level is working with another. Furthermore, Body Moves (pragmatic rhythms) occurred in both face-to-face and non-face-to-face communication but not in the same way, and it is expected that the speech-mediated rhythms may also differ in the face-to-face condition with its visual information of gesture and body movement.