Flappy Melody

An interactive installation where player control the ferrofluid by singing

ROLE

Interaction Designer

Programmer

TYPE

Academic

Collaborate with Xinyue Li

SKILLS

p5.js & Arduino

Physical Computing

Fabrication

TIMELINE

Nov- Dec 2018

INTRODUCTION

Flappy Melody is an interactive game installation controlling by the human voice. The idea was inspired by Flappy Bird; users control the bird to avoid the coming barriers. In Flappy Melody, we use the ferrofluid as the moving ‘bird’, its position changes by different tones, and the barrier would rotate and change at the same time, followed by the voice. Players would use their voice, sing a melody to control the ferrofluid away from musical notes as barriers. Once users hit the barrier, FerroFluid would drop and barriers go back to origin to start again.

CONCEPT

Inspiration

Flappy Bird was one of the most popular mobile game in 2014. The game is a side-scroller where the player controls a bird, attempting to fly between columns of green pipes without hitting them.

However, Flappy Bird received tons of critics for its addictive nature. So we started to think about what if we build a physical version of the game? And what if instead of using their hands, the players interact with the game in other ways?

0075500286dce71e12f8c02dad2ceb1d (1).gif

Initial Idea

 

We came up with the idea of using music to play the game. In particular, players have to sing specific tones to pass each barrier. In this way, this installation was not only a game but also a tool for music training. And to make the game more engaging, interacting with the game would be a learning process - at first, the players will try several times to understand the patterns of the game, and once they succeed multiple moves, they would find the song we programmed was "Merry Christmas" and finish the game.

To make the game installation more accessible and portable, we built the game in two cylinders. Since we wanted the players to interact without using their hands, all the game mechanics should be inside the cylinders. Thus, we used magnetism for the moving 'bird' mechanism: the bird was made of ferrofluid and would be controlled by a magnet inside. And the barriers were physical ones and we would detect if the bird hit the barriers. Both the barriers and the bird would be trapped between the two cylinders and the inner cylinder would rotate to keep the game looping.

IMG_3311.jpg

Concept illustration

Techniques

 

  • p5.js: Using the sound library of Javascript to detect the frequency from the voice input.

  • Serial Communication between p5.js and Arduino: sending pitch data to Arduino.

  • Arduino: Controlling the rotation of motors to move the ferrofluid and the cylinder. 

  • Electromagnets: For dropping or attracting the ferrofluid for the game setting.

PROTOTYPING & TESTING

Playtest- Round 1

 

To test our basic idea and observe users' behaviors, we made a basic prototype without using the Arduino. It was just a cylinder with some basic hand-drawn barriers. And to avoid making too much mess, we just use a small magnet for the bird and put the ferrofluid in a bottle next to the cylinder. Instead of using motors, we manually move the inner magnet and rotate the cylinder based on the users' tone. The instruction was written on a piece of paper to prevent us from explaining too much.

The first prototype and basic instruction

User exploring the game setting

Key findings


1. The instruction was not clear enough.
It took some time for people to realize that they have to sing to play the game. We should have a real mic to better indicate singing. Also, the word “rhyme” was confusing, we should change it to “song”. Even if they noticed what would happen when they sing, they forgot to press the start button. The instruction seemed problematic for asking people to do something before the game. We should allow people to start the game right away and make the instruction clearer.


2. People got distracted by the ferrofluid.
Having seen the ferrofluid, people were like “That’s so cool!”, “Can I play with it?”. Then they forgot about the controlling thing. We had to think about would that become a problem for our game? And would they treat the ferrofluid as the bird which is supposed to be "flying" through barriers?

 

3. The barriers were confusing and difficult to go through.

Users had some difficulties in figuring out how the barriers were set. And if there's only a small gap between each barrier, players have to sing accurately to succeed, which might be extremely hard for those who could sing well. We could modify the barriers into something related to the music to better indicate how the game works.


4. Having a preset song wasn’t as easy as we thought.
None of the players figured out the barriers representing a song. And they found it hard to know the right pinch. We should probably have a tone calibration before the game starts. Also, some people suggested we should allow users to choose the song. But that would be so difficult to accomplish within just javascript and Arduino. 

Coding
 

The programming of the installation consisted of three parts.  

 

1. Tone detection in p5.js. 

Based on the tone detection library and Prof Tom Igoe's example, we finished the code for tone detection. Particularly, the pinch data was mapped into multiple ranges and the data coming out would be number from 0 to 9.  

To avoid the influence of surrounding noise, we tested and figured out the proper amplitude threshold. 

tone-detection-code.jpg

Core code for tone detection in javascript

2. Serial Communication  

Serial communication was the connection between the javascript code and the electronics. This part was written in p5.js.

 

3. Step motors control in Arduino

By serial communication, the Arduino obtained the pinch data from tone detection. In response, one of the motors would rotate curtain degrees to move the magnet up and down based on the height of frequency. The other motor would rotate the barriers every time the pinch changed to keep the game looping.

View Code

Building the electronic mechanism

 

Since we were using ferrofluid which would be messy in materials other than glass, we had to build the mechanism inside the glass cylinders. It was hard to find bigger glass cylinders, so we had to face the difficulty of the limited space in the cylinders, especially the structure for moving the magnetic ferrofluid vertically. The existing structure such as the linear actuator could not fit in the small cylinder. The biggest challenge was to build our own transmission structure.

So for the vertical movement, we came up with the pulley and timing belt structure controlled by a stepper motor.

Change from 2 pulleys to 3 pulleys

Pulley and timing belt structure for moving the magnet vertically

At first, we used two pulleys together with the timing belt, but after testing, it turned out that the range of movement for the magnet was so limited. So we changed to a triangle shape using three pulleys.

IMG_1938.jpg

Two-pulley system

IMG_2230.jpg

Three-pulley system

Testing the structure with ferrofluid

We also encountered a big challenge of rotating the cylinder to keep the game looping. Motors were not powerful enough to rotate the glass cylinder and all the circuits inside. So instead, we design a lid with all the barriers attached to it and rotate the lid. Thus, all the barriers would move horizontally. And this structure consisted of two pulleys and a timing belt and was driven by another stepper motor.

User Test- Round 2

 

After we finished the mechanism and all the programming, we invited a few people to test the game. Based on the feedback in the playtest, we changed the barriers to musical elements. At this time, we didn't use the electronic magnet.

Key Findings

1. Users had trouble figuring out when to start.

A number of people ask similar questions like "Has the game already start?" We realized that there should have been a clear starting point and the rotation of the barriers shouldn't start until the player starts singing.

 

2. The environment noise still had some influence.

When we didn't use microphone input, we could use the noise cancelation function of the computer. But the microphone input wasn't satisfied. We should have used a microphone that only collected a direct voice

FABRICATION

Materials

 

Besides the glass cylinders, we used acrylic and wood as the main materials. The barriers were made by black acrylic in accord with the singing instruction. And we also made an acrylic base for the pulleys and motors. To hold the cylinders and the breadboard circuit, we made the wood enclosure.

IMG_2243.jpg

Wood enclosure

IMG_2121.jpg

"Barriers"

Circuits

 

The circuit was carefully fixed before assembling to avoid being influenced by the rotation.

Wiring

Fixed circuit

Finalizing

 

After plenty of testing, we realized the electronic magnet was so heavy that its movement through the timing belt would be more and more inaccurate through time. So in the end, we gave up the electronic magnet and used a small magnet instead. Thus, the game would keep going even if the player sings the wrong tone.

REFLECTION

ITP Winter Show

 

Being selected for the ITP Winter Show, we presented the game. During the two-day show, dozens of people tried the game and we received plenty of feedback.

What we've learned 

1. People felt embarrassed about singing in such a crowded environment.

The environment of the Winter Show was much crowded and messy than our user tests before, a lot of players suggested they would prefer speaking rather than singing. Using voice recognition would be an interesting attempt. 

 

2. Players' behaviors didn't meet our goal of making it a learning process. 

Most people find out the movement pattern quite soon, but they didn't realize there was a pre-programmed song. Sometimes, users even played the game successfully just by speaking, rather than singing.

Video demo

© 2020 by Jingyi Wen