Video sensing is a type of augmented reality that allows a software to "see" objects in front of the camera and respond to them on-screen with animations, sounds, or any code one can imagine. We use a video feed instead of using a MakeyMakey or other keyboard input devices. Interacting with a computer in such a playful and creative way seemed like a rich tool for computational tinkering activities. We decided to familiarize ourselves with this aspect of Scratch and have been exploring video sensing with Scratch for about a month now (see here and here).
Real world paper objects merged with computer animations on a computer screen.
I created Day to Night as an example of a starting place for visitors. The grass was handmade and imported as a sprite, and the Deanna sprite was left over from a Scratch project that AiR Tim Hunkin made.
Techniques and Scratch Assets
We learned that Scratch can detect motion as well as color. The background color changes to a 50% opacity blue when the video sensing picks up a specified color (in this case, the "moon" on a stick that is light yellow). The code block below is assigned to a large, transparent rectangle sprite. It also triggers the bat to come out and removes the sun.
This code is assigned to the transparent sprite.
The sun sprite appears on whether the transparent sprite detects the moon color.
The bat sprite's appearance is also dependent on the presence of the moon color.
We believe that video sensing can be added to the library of computational tinkering projects. Video sensing encourages visual story telling along the same lines as some of our collaborators like Becca Rose.
As we think about bringing Scratch video sensing out on the floor for visitors, I'm interested in creating inviting set-ups for visitors that encourage a wide variety of possible outcomes.
Where Do We Go From Here?
Scenes. A walking sprite could transition between scenes, like Eric Rosenbaum's Big Tape Adventure. Perhaps the Deanna sprite could go on a walking tour of San Francisco or the Exploratorium, or even go on a safari or to outer space.
Collaboration. Inspired by the drawing-style of exquisite corpse, each participant could contribute an object and code what happens when the object is placed on the screen. In a museum setting, we could add to the same project throughout the day and different visitors could contribute to a larger, collaborative story.
This work was supported by a grant from Science Sandbox, an initiative of the Simons Foundation
This project was made possible through the generous support from the LEGO Foundation




