Visually grounded virtual accelerometers : a longitudinal video investigation of dyadic bodily dynamics around the time of word acquisition

Abstract

Human movement encodes information about internal states and goals. When these goals involve dyadic interactions, such as in language acquisition, the nature of the movement and proximity become representative, allowing parts of our internal states to manifest. We propose an approach called Visually Grounded Virtual Accelerometers (VGVA), to aid with ecologically-valid video analysis investigations, involving humans during dyadic interactions. Utilizing the Human Speechome (HSP) [1] video corpus database, we examine a dyadic interaction paradigm taken from the caregiver-child ecology, during language acquisition. We proceed to characterize human interaction in a video cross-modally; by visually detecting and assessing the child's bodily dynamics in a video, grounded on the caregiver's bodily dynamics of the same video and the related HSP speech transcriptions [2]. Potential applications include analyzing a child's language acquisition, establishing longitudinal diagnostic means for child developmental disorders and generally establishing a metric of effective human communication on dyadic interactions under a video surveillance system. In this thesis, we examine word-learning transcribed video episodes before and after the age of the word's acquisition (AOA). As auditory stimulus is uttered from the caregiver, points along the VGVA tracked sequences corresponding to the onset and post-onset of the child-caregiver bodily responses, are used to longitudinally mark and characterize episodes of word learning. We report a systematic shift in terms of caregiver-child synchrony in motion and turning behavior, tied to exposures of the target word around the time the child begins to understand and thus respond to instances of the spoken word. The systematic shift, diminishes gradually after the age of word acquisition (AOA).