Skip to main content

Coding and Storytelling with Video Sensing

Deanna Deanna
April 5, 2018
Calculating...
Coding and Storytelling with Video Sensing

Video sensing is a type of augmented reality that allows a software to "see" objects in front of the camera and respond to them on-screen with animations, sounds, or any code one can imagine. We use a video feed instead of using a MakeyMakey or other keyboard input devices. Interacting with a computer in such a playful and creative way seemed like a rich tool for computational tinkering activities. We decided to familiarize ourselves with this aspect of Scratch and have been exploring video sensing with Scratch for about a month now (see here and here).

 

Real world paper objects merged with computer animations on a computer screen.

I created Day to Night as an example of a starting place for visitors. The grass was handmade and imported as a sprite, and the Deanna sprite was left over from a Scratch project that AiR Tim Hunkin made.

video-sensing-setup

Techniques and Scratch Assets
We learned that Scratch can detect motion as well as color. The background color changes to a 50% opacity blue when the video sensing picks up a specified color (in this case, the "moon" on a stick that is light yellow). The code block below is assigned to a large, transparent rectangle sprite. It also triggers the bat to come out and removes the sun.

Screen Shot 2018-04-09 at 4.24.08 PM


This code is assigned to the transparent sprite.

Screen Shot 2018-04-09 at 4.24.29 PM

The sun sprite appears on whether the transparent sprite detects the moon color.

Screen Shot 2018-04-09 at 4.24.40 PM


The bat sprite's appearance is also dependent on the presence of the moon color.

We believe that video sensing can be added to the library of computational tinkering projects. Video sensing encourages visual story telling along the same lines as some of our collaborators like Becca Rose.
As we think about bringing Scratch video sensing out on the floor for visitors, I'm interested in creating inviting set-ups for visitors that encourage a wide variety of possible outcomes.

Where Do We Go From Here?

 

Sounds. After sharing this prototype during a team share-out, we discussed what visitors (or Tinkering Studio members) might want to add to the set-up. Sounds were mentioned as a must have, including daytime sounds (chirping birds, feet walking on grass) and nighttime sounds (crickets, owl hoots).

Characters. With more sounds come the opportunity to add more characters, or sprites in the Scratch vernacular. Populate the scene with more animals and environmental details.

Half-baked Ideas

Scenes. A walking sprite could transition between scenes, like Eric Rosenbaum's Big Tape Adventure. Perhaps the Deanna sprite could go on a walking tour of San Francisco or the Exploratorium, or even go on a safari or to outer space.

Collaboration. Inspired by the drawing-style of exquisite corpse, each participant could contribute an object and code what happens when the object is placed on the screen. In a museum setting, we could add to the same project throughout the day and different visitors could contribute to a larger, collaborative story.

Simons Foundation Hi Res


replace this text

This work was supported by a grant from Science Sandbox, an initiative of the Simons Foundation

 

 

 

replace this text

This project was made possible through the generous support from the LEGO Foundation