Milestone 5: User Study I
May 12, 2014
Visit cardcollage.herokuapp.com for the functional prototype.
Introduction
Now that we have a working prototype, we want to expand our user testing into real situations where people want to make cards and flyers for each other. This is crucial, because it will test whether the product is actually useful for people. It will also allow us to see whether people like the resulting collages enough to send them to a recipient. This will form the basis for our main driving question in this week’s user study.
On Thursday’s class, the instructors also underlined a point that had come up for us during user testing - that scaffolding the creative participation of the collaborators is a key part of Card Collage. If a template is unclear or uninspiring, then nobody will want to engage enough to make a nice collage, and the whole project falls apart. It therefore makes sense to present the collaborators with a range of template patterns and see what engages people to take creative photos. We also asked this question in our follow-up user study interviews.
To summarize, our main driving question is: Given that the collaborators know they are contributing a collage to a real recipient that they personally know, how authentic is their participation? We hypothesize they will be genuine in their participation, sending pictures that match the template and personal messages.
In addition, we also asked this question in our follow-up interviews: What template would the collaborators have liked to contribute to, in sending a gift for the same recipient?
Methods
To answer our main driving question, we assembled four separate groups to each create their own collage for three different recipients.
The four recipients are Dan’s grandmother Zoe, Dan’s friend Reba, and Jim’s friends John and Ellen. All recipients live far away, which fits with our theme of forming weak connection with people we don’t usually contact. Zoe, Ellen, and John have birthdays coming up, and Reba is currently abroad, away from her friends. The collaborators are family members of Zoe and friends of Reba, Ellen, and John respectively. We recruited them by sending them an email message asking them to participate in making a birthday or miss-you message to the recipients. To Zoe, Ellen, and John, the message will be “Happy birthday, (Name), (heart collage)” and to Reba it will be “We miss you, Reba, (heart collage)” The collaborators collaborated in making a 4-tiles heart collage as well as leave a personal video or text message. In total, we gather 9 collaborator data points, out of a max of 13 data points (one for each letter, minus three since we will start out the first tile for three of the tiles). The core task is making a tile - photo, video, and text message. In addition, we also tested our prototype with other non-targeted users, for additional feedback about the design.
We collected both quantitative and qualitative data points. Quantitatively, we measured how many times collaborators retake the picture and video. Qualitatively, we measured how well they follow the instructions of the program, and how closely their pictures resemble the shape they are supposed to make and how genuine is their video message.
After the user study, we also interviewed the collaborators about their experience. In particular, we first asked them about their engagement with the current template. Then we gave them a blank template and asked them to provide us with the template they would feel most engaged in and why.
Interview questions:
- How easy or difficult was it to take a photo?
- How did you decide what photo you wanted to take?
- Did anything about using the program surprise you?
- Are you happy with the result?
- What did you think of the template I gave - was it clear what you had to do?
- What other templates might be fun to do?
- Is there anything else you'd like to tell me about the program?
Results
We present our results as short summaries of the actions and responses of each user, and some simple summary statistics.
Collage for Reba
Rachel (bottom left) took a photo with her friend and used her arm to make part of the heart template shape, and she recorded a video. But she didn’t press “Stop Recording” before pressing “Finish”, so her video didn’t get saved.
Nicole (top right) disregarded the heart-shape template when she took her photo, apparently because she didn’t understand that she was supposed to follow the template. She also accidentally clicked on another person’s photo and overrode it. Finally, she claimed that her old computer was unable to support video, but since she was able to take a photo, the problem may actually be that she didn’t allow camera access to Chrome a second time.
Collage for John
Billy (top left) kind of followed the template by using his head as the shape of the heart. However, it did not quite conform well with the heart shape. He also recorded a video but did not press “Stop Recording” before pressing “Finished.” No video was saved.
Brian (top right) did not follow the template but just took a mug shot of himself. His video message was genuinely addressed to wish John a happy birthday.
George (bottom right) also did not follow the template, but he took a creative picture of himself. His video was also pretty creative: he sent funny faces because his mic did not work.
Collage for Ellen
Mary (top left) followed the template by using her hands to make part of the heart shape. She actually clicked on take photo multiple times to get the right shot. She says she didn’t know there was a timer on the picture, so she was anxious about how to take the picture when both her hands are occupied.
Peter and Macy (top right) did not follow the template and did not make an honest attempt. They wrote a happy birthday text message. However, in later interviews, we found out that they did not understand that they were supposed to pose for a template.
Stephanie (bottom left) followed the template and attempted to record an authentic video message. However, she also clicked on “Finish” too early and the video was lost.
Katherine (bottom right) also followed the templated and attempted to record, but clicked on "Finish" too early.
Collage for Zoe
Nobody responded in time for this collage :-(
Summary statistics
Table 1: Summary statistics of users’ attempts at contributing to the collages. “Authentic” here means that users actually tried to make a plausible message for their friend, and didn’t just make a placeholder message to please us.
From Table 1, we see that 8 out of 9 attempts were authentic attempts, 5 out of the 9 attempts followed the heart template, and 4 users did not respond in time to participate.
We also collected some Google Analytics about users’ interactions. 40 unique clicks were recorded for taking picture, suggesting that many users clicked multiple times to get the right picture they wanted. 50 unique take video events were recorded, meaning even more users (among those who successfully recorded) re-recorded their videos until they were satisfied.
Discussions
This user test was extremely informative for our fledgling app. We address our two original user study questions below, and then offer other insights that we didn’t anticipate with our question.
Question 1: Given that the collaborators know they are contributing a collage to a real recipient that they personally know, how authentic is their participation?
The purpose of this question was to see whether people really attempt to use the app in a way that would be a plausible message to a friend. Out of nine participants who contributed to a collage, eight actually made a plausible message. (The one who did not made a throwaway photo and empty text message.) This suggests to us that people understand the purpose of the app, and are willing to make heartfelt messages. Four participants also didn’t use the app in time to get included in this report. Two participates wished their message would be private so that they could leave longer, more secretive messages.
Question 2: What template would the collaborators have liked to contribute to, in sending a gift for the same recipient?
The purpose of this question was to understand how different templates prompt people to be more or less creative in their photos and video messages. We followed up with people and asked what other templates they would have liked. Surprisingly, out of the nine participants who contributed to a collage, only five even followed the heart template that we provided. The vast majority of participants failed to understand the purpose of the template - for creating a pattern or shape across multiple pictures - and instead they just took a selfie. Therefore, only the five who followed the heart template were able to respond to our follow-up question and suggested other shapes like stars, cakes, flower, and letters. In addition, we found that seeding the template with our own photo did not result in other people following the template.
Our users also found many technical bugs and confusing user interface features, listed below:
1. The main page:
- The collage is off-center and isn’t presented like a card or gift
- Guidelines for participation are unclear at the start
- Some people didn’t discover that tiles are flippable
- It’s not clear that you should click on the non-photo tiles
- It’s possible to accidentally overwrite an old photo if you click on a tile that has already been completed
- The font isn’t appealing
- The text messages don’t display properly
2. The photo page:
- Enabling the webcam was very confusing for people
- The two frames on this screen are confusing and unnecessary
- The layout of three buttons is not clear enough, particularly “Done”
- Lots of people disregard the template shape
- People didn’t know that there was a photo timer, and found it difficult to pose for a photo and have to click on “Take photo” at the same time
- You can’t remove your photo once you take it - need a default picture
- Some found it hard to pose for the two upper heart tiles before a webcam while including their face in the frame as well.
3. The video page:
- People don’t stop the video before hitting finish, and they lose the video! Very demoralizing!
- One user had video tech difficulties, but was using an old computer
- Some users want to have private video messages for the recipient
- Some found it confusing that they enable video, and they find out that they have to enable audio again.
Overall, we see that our bugs mostly come from “expert blind spot” - our own understanding of the app caused us to miss unclear interface features for a novice.
Implications
For each of our technical bugs, reproduced below, we list our intended software fixes:
1. The main page:
- The collage is off-center and isn’t presented like a card or gift
- Fix: move the collage to center of screen, change background to make it feel like a nice card on a clean background.
- Guidelines for participation are unclear at the start
- Fix: Landing page with examples of successful collages. One collage can actually be an instructional message explaining Card Collage. Could also provide a short introductory YouTube video.
- Some people didn’t discover that tiles are flippable
- Fix: starting examples or intro video illustrate the flip property.
- It’s not clear that you should click on the non-photo tiles
- Fix: Make the non-photo tiles bright, and grey out the claimed tiles.
- It’s possible to accidentally overwrite an old photo if you click on a tile that has already been completed
- Fix: make an “are you sure” popup so that someone has to confirm if they want to overwrite an old tile.
- The font isn’t appealing
- Fix: change it!
- The text messages don’t display properly
- Fix: take away text messages entirely, because people will already have video messages and text adds little.
2. The photo page:
- Enabling the webcam was very confusing for people
- Fix: Make Card Collage a single-webpage app that uses Javascript to rearrange DOM elements, so that you only need to approve video/audio once.
- The two frames on this screen are confusing and unnecessary
- Fix: combine them into one screen that is a camera when someone is trying to take a photo, and a still image once you have taken the photo.
- The layout of three buttons is not clear enough, particularly “Done”
- Fix: Rename “Done” to “Next”, and move it to the bottom right. Move “Cancel” to the bottom left.
- Lots of people disregard the template shape
- Fix: Not sure yet - see below.
- People didn’t know that there was a photo timer, and found it difficult to pose for a photo
- Fix: rename button to a picture of a camera plus “start photo countdown”.
- You can’t remove your photo once you take it - need a default picture
- Fix: add a “reset” button.
- People don’t stop the video before hitting finish, and they lose the video! Very demoralizing!
- Fix: make the “finish” button greyed out until you stop the video.
- One user had video tech difficulties, but was using an old computer
- Fix: not sure, still have to look at her setup.
- Some users want to have private video messages for the recipient
- Fix: add an option to make a private video, in which case only the user and the recipient can see the video once it is recorded.
These fixes are technically non-trivial, but don’t fundamentally challenge the nature of the program. However, the fact that fully half of our participants failed to follow the card template is troubling. Is it because participants didn’t understand that they were supposed to, because they didn’t want to, or because they were unable to? It seems likely that many of these participants didn’t realize the purpose of the template, and if we made an introductory example or welcome page, we might be able to address this problem.
But what if following a template is simply not that compelling to users? Survey responses showed that they could generate ideas for alternate templates, but it’s still not clear that contributing to a larger picture is a compelling enough reason for people to take photos that match a template. People might just prefer to upload photos of themselves with the recipient, or make other creative use of the tile. But if we remove the template aspect of Card Collage, then it becomes quite similar to Pinterest or Path, and becomes less collaborative and less unique. Our main challenge going forward is therefore to verify the importance of templates as a motivating aspect of collaborative card-making.