Buy Tickets
Donate

Making Colorful Music with Scratch Video Detection

Tinkering Community Tinkering Community
November 12, 2021
Calculating...
Making Colorful Music with Scratch Video Detection

replace this text

Guest post written by Nathan Rabinovitch, tinkering resident with the Brazilian Creative Learning Network. Nathan is a researcher and partner of LSITec at POLI-USP's electrical engineering State University and a practicing teacher at Elementary High School. From an early age, Nathan researches and explores the relationships and complexities that emerge from the different combinations of materials, technologies and narratives.


He would like to thank two friends who helped him make this post. Lina Lopes, for the photos, GIFs and videos and Daniela Lyra, for translating this material into English.

I designed this activity for Scratch Conference 2021 immediately after getting invited by the Tinkering Studio to create something together. I was very excited to be able to explore new ways to interact with the Scratch video sensor. After some initial experiments, I realized that I could use the colors captured by the camera to trigger commands within the code and I took the opportunity to do some tests with music. It was really cool, but it wasn't exactly what I was hoping for, something was still missing. At that moment, I remembered an interactive work I had visited at the Exploratorium science museum. Hole Saw Rhythm was exactly what I was looking for! I wanted a way to make music that was fast, playful and visual.

After finding the inspiration, it took me a few days to organize and polish the activity into the format you can find below. This project describes my exploration and my playful process, but I suggest you try different possibilities, materials and parameters. Think of this post as a starting point for your own creative journey.

Materials

  • Computerreplace this text
  • Webcam
  • Cardboard
  • Scissors
  • Paint
  • Brush

To create the physical musical notes, we use cardboard and paint. We made simple geometric shapes such as squares, triangles and circles. You can experiment with other shapes, such as animal outlines, letters or any other cool idea you might have.

Writing Your Code

 

replace this text

This project uses the Scratch video detection extension.

replace this text

We will use a vertical bar as a character. There are many other possibilities that are worth exploring, but we will use the bar as a starting point

 

The Code

In this project, we will use only two groups of blocks.

replace this text
The first sets the video transparency to zero and creates an infinite loop that slides the actor across the screen, swiping left to right in two seconds, then instantly returning to the starting position. If you want the music to repeat faster, just decrease the time the slider slides. If you want it slower, just increase the time the bar slides.

 

replace this text
The second set is responsible for playing a musical note every time the actor passes over a certain color. The block uses the color conditional to trigger the sound.

 

The music notes can be found under the sounds category.

 

The wait block adjusts the repetition speed of each note. Try changing its value to numbers between 0 and 1.

Remix

If you feel like a helping hand, jump into this project and make it your own. Don't forget to tune the colors of the notes.

Organize the Space

To make sure the project works well, we need to think in terms of colors. Because the code block tests the colors of the objects that are captured by the camera, it is important to have good lighting. The more homogeneous the better. If the activity takes place near a window, the captured colors may vary throughout the day depending on the amount of sunlight you've got. So, it is important to take into account your surroundings. 

To help you get good lighting conditions, you might want to place two lamps, one on each side of your tinkering area.

The First Musical Note

 

replace this text
Step 1: Point the computer camera to your tinkering table. If you are working from a laptop, you can lower the screen or tilt the computer with some support. Once the camera is in position, the less it moves, the better. This goes for both the tuning and the timing of playing the song.

 

replace this text
Step 2: To tune the first musical note, place the colored object on your tinkering table so that the camera can capture its color. Then use the eyedropper tool to choose the color that will trigger the sound. In some cases, slight variations of the colors captured from your tinkering table might interfere with playing a note. To help you with that, you might try the block with the selected colors at two different points on the table.

 

replace this text
Step 3: Click on the green flag for the character to start moving. When the bar passes over the programmed color you will hear the musical note selected in the play sound block. Special tip: If any note does not work, repeat the tuning process, making sure the light and color vary as little as possible. Remember that in this activity the camera thinks through colors.

 

Adding Musical Notes

 

To add more musical notes, duplicate the set of blocks that trigger the first note. Then tweak the desired new color, and change the note that will be played.

 

Further Explorations

It is possible to use other materials and forms to create musical notes. Here are some different suggestions:

replace this text
replace this text
replace this text

                          Bottle caps                                                    Plants and veggies                                           Fun cardboard shapes

 

You might want to explore other instruments in the sound blocks.

 

Scratch has a musical extension that is also worth exploring.

 

You can add a photo or painting as a stage and then choose a few repeating colors to create new music.

 

Automata and Music

Discover the work of Brazilian artist Agnaldo Pinho. I like the way he combines automatas with music and other interesting ways to interact with his artwork. 
 

 

References