Exploring real-time video interactivity with Scratch

Abstract

Real-time video interactivity is becoming increasingly popular in today's world with the advent of better and more affordable video input devices. With the recent release of the Microsoft Kinect followed by an official Kinect SDK, there has been an explosion of activity around utilizing this now easily-accessible video sensor data. Many creative uses have surfaced including object recognition, gesture recognition, and more. The audience capable of taking full advantage of these video technologies continues to be a technical crowd, likely with a background in computer science. But what if such video technology were made accessible to a much more diverse crowd? This thesis presents a set of computer vision tools for exploration of the real-time video interactivity space in the context of Scratch (scratch.mit.edu), a graphical block-based programming language accessible to all ages. To decide what functionality to provide to Scratch users, various computer vision algorithms are tested, including object detection, object recognition, face recognition and optical flow. Ultimately, an optical flow implementation is realized and its generative abilities are observed through testing with different user groups.