© 2018 by Jingyi Wen

Flappy Melody

An interactive installation where player control the ferrofluid by singing

ROLE

Interaction Designer

User Researcher

Programmer

 

TYPE

Academic / Collaborate with Xinyue Li

 

TIMELINE

2018.11 - 2018.12 (5 Weeks)

 

TOOLS

Arduino

p5.js

Microphone

 

SKILLS

Physical Computing,

Usability test,

Prototyping,

Fabrication

 

INTRODUCTION

Flappy Melody is an interactive game installation controlling by the human voice. The idea was inspired by Flappy Bird; users control the bird to avoid the coming barriers. In Flappy Melody, we use the ferrofluid as the moving ‘bird’, its position changes by different tones, and the barrier would rotate and change at the same time, followed by the voice. Players would use their voice, sing a melody to control the ferrofluid away from musical notes as barriers. Once users hit the barrier, FerroFluid would drop and barriers go back to origin to start again.

01 Ideation

Background: Flappy Bird

 

Flappy Bird was one of the most popular mobile game in 2014. The game is a side-scroller where the player controls a bird, attempting to fly between columns of green pipes without hitting them.

However, Flappy Bird received tons of critics for its addictive nature. So we started to think about what if we build a physical version of the game? And what if instead of using their hands, the players interact with the game in other ways?

 

Initial Idea

 

We came up with the idea of using music to play the game. In particular, players have to sing specific tones to pass each barrier. By this way, this installation was not only a game but also a tool of music training. And to make the game more engaging, interacting with the game would be a learning process - at first the players will try several times to understand the patterns of the game, and once they succeed multiple moves, they would find the song we programmed was "Merry Christmas" and finish the game.

To make the game installation more accessible and portable, we built the game in two cylinders. Since we wanted the players to interact without using their hands, all the game mechanics should be inside the cylinders. Thus, we used magnetism for the moving 'bird' mechanism: the bird was made of ferrofluid and would be controlled by a magnet inside. And the barriers were physical ones and we would detect if the bird hit the barriers. Both the barriers and the bird would be trapped between the two cylinders and the inner cylinder would rotate to keep the game looping.

 

 

Illustration of the design

Techniques

 

  • p5.js: Using the sound library of Javascript to detect the frequency from the voice input.

  • Serial Communication between p5.js and Arduino: sending pitch data to Arduino.

  • Arduino: Controlling the rotation of motors to move the ferrofluid and the cylinder. 

  • Electromagnets: For dropping or attracting the ferrofluid for the game setting.

 

02 Prototyping and Testing

1st Round Playtest

 

To test our basic idea and observe users' behaviors, we made a basic prototype without using the Arduino. It was just a cylinder with some basic hand-drawn barriers. And to avoid making too much mess, we just use a small magnet for the bird and put the ferrofluid in a bottle next to the cylinder. Instead of using motors, we manually move the inner magnet and rotate the cylinder based on users' tone. The instruction was written on a piece of paper to prevent us from explaining too much.

 

 

playtest.jpeg

First prototype and basic instruction

testing1.jpeg

User exploring the game setting

What we learned


1. The instruction was not clear enough.
First, it took some time for people to realize that they have to sing to play the game. We should have a real mic to better indicate singing. Also, the word “rhyme” was confusing, we should change it to “song”. Even if they noticed what would happen when they sing, they forgot to press the start button. The instruction seemed problematic for asking people to do something before the game.
We should allow people to start the game right away and make the instruction clearer.


2. People got distracted by the ferrofluid.
Having seen the ferrofluid, people were like “That’s so cool!”, “Can I play with it?”. Then they forgot about the controlling thing. We had to think about would that become a problem for our game? And would they treat the ferrofluid as the bird which is supposed to be "flying" through barriers?

 

3. The barriers were confusing and difficult to go through.

Users had some difficulties in figuring out how the barriers were set. And if there's only a small gap between each barrier, players have to sing accurately to succeed, which might be extremely hard for those who could sing well. We could change the barriers into something related to the music to better indicate how the game works.


4. Having a preset song wasn’t as easy as we thought.
None of the players figured out the barriers represent a song. And they found it hard to know the right pinch.
We should probably have a tone calibration before the game starts. Also, some people suggested we should allow the users to choose the song. But that would be so difficult to accomplish within just javascript and Arduino. 

 

 

 

Programming (View all the codes here

 

 

The programming of the installation consisted of three parts.  

 

1. Tone detection in p5.js. 

Based on the tone detection library and Prof Tom Igoe's example, we finished the code for tone detection. Particularly, the pinch data was mapped into multiple ranges and the data coming out would be number from 0 to 9.  

To avoid the influence of surrounding noise, we tested and figured out the proper amplitude threshold. 

(Reference https://github.com/tigoe/p5js_examples/tree/master/toneDetect)

2. Serial communication

Serial communication was the connection between the javascript code and the electronics. This part was written in p5.js.

 

3. Step motors control in Arduino

By serial communication, the Arduino obtained the pinch data from the tone detection. In response, one of the motors would rotate curtain degree to move the magnet up and down based on the height of frequency. The other motor would rotate the barriers every time the pinch changed to keep the game looping.

 

Building the electronic mechanism

 

Since we were using ferrofluid which would be messy in materials other than glass, we had to build the mechanism inside the glass cylinders. It was hard to find bigger glass cylinders, so we had to face the difficulty of the limited space in the cylinders, especially the structure for moving the magnetic ferrofluid vertically. The existing structure such as the linear actuator could not fit in the small cylinder. The biggest challenge was to build our own transmission structure.

So for the vertical movement, we came up with the pulley and timing belt structure controlled by a stepper motor.

 

Pulley and timing belt structure for moving the magnet vertically

Change from 2 pulleys to 3 pulleys

At first, we used two pulleys together with the timing belt, but after testing, it turned out that the range of movement for the magnet was so limited. So we changed to a triangle shape using three pulleys.

Two-pulley structure 

Three-pulley structure 

Testing the structure with ferrofluid

We also encountered a big challenge of rotating the cylinder to keep the game looping. Motors were not powerful enough to rotate the glass cylinder and all the circuit inside. So instead, we design a lid with all the barriers attached to it and rotate the lid. Thus, all the barriers would move horizontally. And the structure consisted of two pulleys and a timing belt and was driven by another stepper motor.

 

Pulley and timing belt structure for rotating the barriers

2nd Round User Test

 

After we finished the mechanism and all the programming, we invited a few people to test the game. Based on the feedback in the playtest, we changed the barriers to musical elements. At this time, we didn't use the electronic magnet.

 

 

What we learned

 

1. Users had trouble in figuring out when to start.

A number of people ask similar questions like "Has the game already start?" We realized that there should have been a clear starting point and the rotation of the barriers shouldn't start until the player start singing.

 

 

2. The environment noise still had some influence.

When we didn't use microphone input, we could use the noise cancelation function of the computer. But the microphone input wasn't satisfied. We should have used a microphone that only collected direct voice. 

 

03 Fabrication

Materials

 

Besides the glass cylinders, we used the acrylic and wood as the main materials. The barriers were made by black acrylic in accord with the singing instruction. And we also made an acrylic base for the pulleys and motors. To hold the cylinders and the breadboard circuit, we made the wood enclosure.

 

 

Wood enclosure for the cylinder

Fixing the circuit

 

The circuit was carefully fixed before assembling to avoid being influenced by the rotation.

The circuit

Making sure the wires were tight

Finalizing

 

After plenty of testing, we realized the electronic magnet was so heavy that its movement through the timing belt would be more and more inaccurate through time. So in the end, we gave up the electronic magnet and used a small magnet instead. Thus, the game would keep going even if the player sing a wrong tone.

04 Presentation

ITP Winter Show

 

Being selected for the ITP Winter Show, we presented the game. During the two-day show, dozens of people tried the game and we received numerous feedback.

What we learned

 

1. People felt embarrassed in singing in such a crowded environment.

The environment of the Winter Show was much crowded and messy than our user tests before, a lot of players suggested they would prefer speaking rather than singing. Using voice recognition would be an interesting attempt. 

 

2. Players behaviors didn't meet our goal of making it a learning process. 

Most people find out the movement pattern quite soon, but they didn't realize there was a pre-programmed song. Sometimes, users even played the game successfully just by speaking, rather than singing.