Grandma's Companion

An "Instagram-story" box for grandparents to visually check-in with faraway grandkids


Interaction Designer




Graduate Thesis Project


Node. js 

Physical Computing



Feb- May 2020


While technologies are making remote communication easier for most of us, millions of senior citizens such as my grandma still find it challenging to use these technologies. As a senior with a very limited educational level and a mindset of being so accustomed to a “tech-free” lifestyle, my grandma feels not confident using any digital devices other than an old fashion flip phone. Video calls and text messages are not options for her, not to mention any apps or social media. So how can I make our remote communication better and strengthen the bonding even I'm far away from home? 

Grandma's companion is an accessible installation that provides awareness of my well-being and daily activities to my grandma via displaying photos and narrations.


Pain Points

When I'm away from home, my grandma and I do phone calls about once a week. But phone calls have the following problems:

1. Information delivery - Words are not enough to show her my life nor to stop her from being worried about me

2. Time difference - We have 12h/13h time difference and it's hard for my grandma to calculate and remember that

3. Lack of emotion - Phone calls are cold

Initial Idea


We came up with the idea of using music to play the game. In particular, players have to sing specific tones to pass each barrier. In this way, this installation was not only a game but also a tool for music training. And to make the game more engaging, interacting with the game would be a learning process - at first, the players will try several times to understand the patterns of the game, and once they succeed multiple moves, they would find the song we programmed was "Merry Christmas" and finish the game.

To make the game installation more accessible and portable, we built the game in two cylinders. Since we wanted the players to interact without using their hands, all the game mechanics should be inside the cylinders. Thus, we used magnetism for the moving 'bird' mechanism: the bird was made of ferrofluid and would be controlled by a magnet inside. And the barriers were physical ones and we would detect if the bird hit the barriers. Both the barriers and the bird would be trapped between the two cylinders and the inner cylinder would rotate to keep the game looping.


Concept illustration



  • p5.js: Using the sound library of Javascript to detect the frequency from the voice input.

  • Serial Communication between p5.js and Arduino: sending pitch data to Arduino.

  • Arduino: Controlling the rotation of motors to move the ferrofluid and the cylinder. 

  • Electromagnets: For dropping or attracting the ferrofluid for the game setting.


Playtest- Round 1


To test our basic idea and observe users' behaviors, we made a basic prototype without using the Arduino. It was just a cylinder with some basic hand-drawn barriers. And to avoid making too much mess, we just use a small magnet for the bird and put the ferrofluid in a bottle next to the cylinder. Instead of using motors, we manually move the inner magnet and rotate the cylinder based on the users' tone. The instruction was written on a piece of paper to prevent us from explaining too much.

The first prototype and basic instruction

User exploring the game setting

Key findings

1. The instruction was not clear enough.
It took some time for people to realize that they have to sing to play the game. We should have a real mic to better indicate singing. Also, the word “rhyme” was confusing, we should change it to “song”. Even if they noticed what would happen when they sing, they forgot to press the start button. The instruction seemed problematic for asking people to do something before the game. We should allow people to start the game right away and make the instruction clearer.

2. People got distracted by the ferrofluid.
Having seen the ferrofluid, people were like “That’s so cool!”, “Can I play with it?”. Then they forgot about the controlling thing. We had to think about would that become a problem for our game? And would they treat the ferrofluid as the bird which is supposed to be "flying" through barriers?


3. The barriers were confusing and difficult to go through.

Users had some difficulties in figuring out how the barriers were set. And if there's only a small gap between each barrier, players have to sing accurately to succeed, which might be extremely hard for those who could sing well. We could modify the barriers into something related to the music to better indicate how the game works.

4. Having a preset song wasn’t as easy as we thought.
None of the players figured out the barriers representing a song. And they found it hard to know the right pinch. We should probably have a tone calibration before the game starts. Also, some people suggested we should allow users to choose the song. But that would be so difficult to accomplish within just javascript and Arduino. 


The programming of the installation consisted of three parts.  


1. Tone detection in p5.js. 

Based on the tone detection library and Prof Tom Igoe's example, we finished the code for tone detection. Particularly, the pinch data was mapped into multiple ranges and the data coming out would be number from 0 to 9.  

To avoid the influence of surrounding noise, we tested and figured out the proper amplitude threshold. 


Core code for tone detection in javascript

2. Serial Communication  

Serial communication was the connection between the javascript code and the electronics. This part was written in p5.js.


3. Step motors control in Arduino

By serial communication, the Arduino obtained the pinch data from tone detection. In response, one of the motors would rotate curtain degrees to move the magnet up and down based on the height of frequency. The other motor would rotate the barriers every time the pinch changed to keep the game looping.

View Code

Building the electronic mechanism


Since we were using ferrofluid which would be messy in materials other than glass, we had to build the mechanism inside the glass cylinders. It was hard to find bigger glass cylinders, so we had to face the difficulty of the limited space in the cylinders, especially the structure for moving the magnetic ferrofluid vertically. The existing structure such as the linear actuator could not fit in the small cylinder. The biggest challenge was to build our own transmission structure.

So for the vertical movement, we came up with the pulley and timing belt structure controlled by a stepper motor.

Change from 2 pulleys to 3 pulleys

Pulley and timing belt structure for moving the magnet vertically

At first, we used two pulleys together with the timing belt, but after testing, it turned out that the range of movement for the magnet was so limited. So we changed to a triangle shape using three pulleys.


Two-pulley system


Three-pulley system

Testing the structure with ferrofluid

We also encountered a big challenge of rotating the cylinder to keep the game looping. Motors were not powerful enough to rotate the glass cylinder and all the circuits inside. So instead, we design a lid with all the barriers attached to it and rotate the lid. Thus, all the barriers would move horizontally. And this structure consisted of two pulleys and a timing belt and was driven by another stepper motor.

User Test- Round 2


After we finished the mechanism and all the programming, we invited a few people to test the game. Based on the feedback in the playtest, we changed the barriers to musical elements. At this time, we didn't use the electronic magnet.

Key Findings

1. Users had trouble figuring out when to start.

A number of people ask similar questions like "Has the game already start?" We realized that there should have been a clear starting point and the rotation of the barriers shouldn't start until the player starts singing.


2. The environment noise still had some influence.

When we didn't use microphone input, we could use the noise cancelation function of the computer. But the microphone input wasn't satisfied. We should have used a microphone that only collected a direct voice




Besides the glass cylinders, we used acrylic and wood as the main materials. The barriers were made by black acrylic in accord with the singing instruction. And we also made an acrylic base for the pulleys and motors. To hold the cylinders and the breadboard circuit, we made the wood enclosure.


Wood enclosure





The circuit was carefully fixed before assembling to avoid being influenced by the rotation.


Fixed circuit



After plenty of testing, we realized the electronic magnet was so heavy that its movement through the timing belt would be more and more inaccurate through time. So in the end, we gave up the electronic magnet and used a small magnet instead. Thus, the game would keep going even if the player sings the wrong tone.


ITP Winter Show


Being selected for the ITP Winter Show, we presented the game. During the two-day show, dozens of people tried the game and we received plenty of feedback.

What we've learned 

1. People felt embarrassed about singing in such a crowded environment.

The environment of the Winter Show was much crowded and messy than our user tests before, a lot of players suggested they would prefer speaking rather than singing. Using voice recognition would be an interesting attempt. 


2. Players' behaviors didn't meet our goal of making it a learning process. 

Most people find out the movement pattern quite soon, but they didn't realize there was a pre-programmed song. Sometimes, users even played the game successfully just by speaking, rather than singing.

Video demo

© 2020 by Jingyi Wen