Guest post written by Nathan Rabinovitch, tinkering resident with the Brazilian Creative Learning Network. Nathan is a researcher and partner of LSITec at POLI-USP's electrical engineering State University and a practicing teacher at Elementary High School. From an early age, Nathan researches and explores the relationships and complexities that emerge from the different combinations of materials, technologies and narratives.
He would like to thank two friends who helped him make this post. Lina Lopes, for the photos, GIFs and videos and Daniela Lyra, for translating this material into English.
I designed this activity for Scratch Conference 2021 immediately after getting invited by the Tinkering Studio to create something together. I was very excited to be able to explore new ways to interact with the Scratch video sensor. After some initial experiments, I realized that I could use the colors captured by the camera to trigger commands within the code and I took the opportunity to do some tests with music. It was really cool, but it wasn't exactly what I was hoping for, something was still missing. At that moment, I remembered an interactive work I had visited at the Exploratorium science museum. Hole Saw Rhythm was exactly what I was looking for! I wanted a way to make music that was fast, playful and visual.
After finding the inspiration, it took me a few days to organize and polish the activity into the format you can find below. This project describes my exploration and my playful process, but I suggest you try different possibilities, materials and parameters. Think of this post as a starting point for your own creative journey.
Materials
- Computer
- Webcam
- Cardboard
- Scissors
- Paint
- Brush
To create the physical musical notes, we use cardboard and paint. We made simple geometric shapes such as squares, triangles and circles. You can experiment with other shapes, such as animal outlines, letters or any other cool idea you might have.
Writing Your Code
This project uses the Scratch video detection extension.
We will use a vertical bar as a character. There are many other possibilities that are worth exploring, but we will use the bar as a starting point
The Code
In this project, we will use only two groups of blocks.



The wait block adjusts the repetition speed of each note. Try changing its value to numbers between 0 and 1.
Remix
If you feel like a helping hand, jump into this project and make it your own. Don't forget to tune the colors of the notes.
Organize the Space
To make sure the project works well, we need to think in terms of colors. Because the code block tests the colors of the objects that are captured by the camera, it is important to have good lighting. The more homogeneous the better. If the activity takes place near a window, the captured colors may vary throughout the day depending on the amount of sunlight you've got. So, it is important to take into account your surroundings.
To help you get good lighting conditions, you might want to place two lamps, one on each side of your tinkering area.
The First Musical Note



Adding Musical Notes

Further Explorations
It is possible to use other materials and forms to create musical notes. Here are some different suggestions:

.gif)
.gif)
Bottle caps Plants and veggies Fun cardboard shapes
You might want to explore other instruments in the sound blocks.
Scratch has a musical extension that is also worth exploring.
You can add a photo or painting as a stage and then choose a few repeating colors to create new music.
Automata and Music
References
- Tinkering Project: Animate Your World
- Tinkering Activity Guide: Scratch Video Sensing Guide
- Blogpost: Stop Motion Animations
- Brazilian Creative Learning Network
- Scratch Brasil Portal