May 12, 2014
Now that we have a working prototype, we want to expand our user testing into real situations where people want to make cards and flyers for each other. This is crucial, because it will test whether the product is actually useful for people. It will also allow us to see whether people like the resulting collages enough to send them to a recipient. This will form the basis for our main driving question in this week’s user study.
On Thursday’s class, the instructors also underlined a point that had come up for us during user testing - that scaffolding the creative participation of the collaborators is a key part of Card Collage. If a template is unclear or uninspiring, then nobody will want to engage enough to make a nice collage, and the whole project falls apart. It therefore makes sense to present the collaborators with a range of template patterns and see what engages people to take creative photos. We also asked this question in our follow-up user study interviews.
To summarize, our main driving question is: Given that the collaborators know they are contributing a collage to a real recipient that they personally know, how authentic is their participation? We hypothesize they will be genuine in their participation, sending pictures that match the template and personal messages.
In addition, we also asked this question in our follow-up interviews: What template would the collaborators have liked to contribute to, in sending a gift for the same recipient?
To answer our main driving question, we assembled four separate groups to each create their own collage for three different recipients.
The four recipients are Dan’s grandmother Zoe, Dan’s friend Reba, and Jim’s friends John and Ellen. All recipients live far away, which fits with our theme of forming weak connection with people we don’t usually contact. Zoe, Ellen, and John have birthdays coming up, and Reba is currently abroad, away from her friends. The collaborators are family members of Zoe and friends of Reba, Ellen, and John respectively. We recruited them by sending them an email message asking them to participate in making a birthday or miss-you message to the recipients. To Zoe, Ellen, and John, the message will be “Happy birthday, (Name), (heart collage)” and to Reba it will be “We miss you, Reba, (heart collage)” The collaborators collaborated in making a 4-tiles heart collage as well as leave a personal video or text message. In total, we gather 9 collaborator data points, out of a max of 13 data points (one for each letter, minus three since we will start out the first tile for three of the tiles). The core task is making a tile - photo, video, and text message. In addition, we also tested our prototype with other non-targeted users, for additional feedback about the design.
We collected both quantitative and qualitative data points. Quantitatively, we measured how many times collaborators retake the picture and video. Qualitatively, we measured how well they follow the instructions of the program, and how closely their pictures resemble the shape they are supposed to make and how genuine is their video message.
After the user study, we also interviewed the collaborators about their experience. In particular, we first asked them about their engagement with the current template. Then we gave them a blank template and asked them to provide us with the template they would feel most engaged in and why.
We present our results as short summaries of the actions and responses of each user, and some simple summary statistics.
Rachel (bottom left) took a photo with her friend and used her arm to make part of the heart template shape, and she recorded a video. But she didn’t press “Stop Recording” before pressing “Finish”, so her video didn’t get saved.
Nicole (top right) disregarded the heart-shape template when she took her photo, apparently because she didn’t understand that she was supposed to follow the template. She also accidentally clicked on another person’s photo and overrode it. Finally, she claimed that her old computer was unable to support video, but since she was able to take a photo, the problem may actually be that she didn’t allow camera access to Chrome a second time.
Billy (top left) kind of followed the template by using his head as the shape of the heart. However, it did not quite conform well with the heart shape. He also recorded a video but did not press “Stop Recording” before pressing “Finished.” No video was saved.
Brian (top right) did not follow the template but just took a mug shot of himself. His video message was genuinely addressed to wish John a happy birthday.
George (bottom right) also did not follow the template, but he took a creative picture of himself. His video was also pretty creative: he sent funny faces because his mic did not work.
Mary (top left) followed the template by using her hands to make part of the heart shape. She actually clicked on take photo multiple times to get the right shot. She says she didn’t know there was a timer on the picture, so she was anxious about how to take the picture when both her hands are occupied.
Peter and Macy (top right) did not follow the template and did not make an honest attempt. They wrote a happy birthday text message. However, in later interviews, we found out that they did not understand that they were supposed to pose for a template.
Stephanie (bottom left) followed the template and attempted to record an authentic video message. However, she also clicked on “Finish” too early and the video was lost.
Katherine (bottom right) also followed the templated and attempted to record, but clicked on "Finish" too early.
Nobody responded in time for this collage :-(
Table 1: Summary statistics of users’ attempts at contributing to the collages. “Authentic” here means that users actually tried to make a plausible message for their friend, and didn’t just make a placeholder message to please us.
From Table 1, we see that 8 out of 9 attempts were authentic attempts, 5 out of the 9 attempts followed the heart template, and 4 users did not respond in time to participate.
We also collected some Google Analytics about users’ interactions. 40 unique clicks were recorded for taking picture, suggesting that many users clicked multiple times to get the right picture they wanted. 50 unique take video events were recorded, meaning even more users (among those who successfully recorded) re-recorded their videos until they were satisfied.
This user test was extremely informative for our fledgling app. We address our two original user study questions below, and then offer other insights that we didn’t anticipate with our question.
Question 1: Given that the collaborators know they are contributing a collage to a real recipient that they personally know, how authentic is their participation?
The purpose of this question was to see whether people really attempt to use the app in a way that would be a plausible message to a friend. Out of nine participants who contributed to a collage, eight actually made a plausible message. (The one who did not made a throwaway photo and empty text message.) This suggests to us that people understand the purpose of the app, and are willing to make heartfelt messages. Four participants also didn’t use the app in time to get included in this report. Two participates wished their message would be private so that they could leave longer, more secretive messages.
Question 2: What template would the collaborators have liked to contribute to, in sending a gift for the same recipient?
The purpose of this question was to understand how different templates prompt people to be more or less creative in their photos and video messages. We followed up with people and asked what other templates they would have liked. Surprisingly, out of the nine participants who contributed to a collage, only five even followed the heart template that we provided. The vast majority of participants failed to understand the purpose of the template - for creating a pattern or shape across multiple pictures - and instead they just took a selfie. Therefore, only the five who followed the heart template were able to respond to our follow-up question and suggested other shapes like stars, cakes, flower, and letters. In addition, we found that seeding the template with our own photo did not result in other people following the template.
Our users also found many technical bugs and confusing user interface features, listed below:
1. The main page:
2. The photo page:
3. The video page:
Overall, we see that our bugs mostly come from “expert blind spot” - our own understanding of the app caused us to miss unclear interface features for a novice.
For each of our technical bugs, reproduced below, we list our intended software fixes:
1. The main page:
2. The photo page:
These fixes are technically non-trivial, but don’t fundamentally challenge the nature of the program. However, the fact that fully half of our participants failed to follow the card template is troubling. Is it because participants didn’t understand that they were supposed to, because they didn’t want to, or because they were unable to? It seems likely that many of these participants didn’t realize the purpose of the template, and if we made an introductory example or welcome page, we might be able to address this problem.
But what if following a template is simply not that compelling to users? Survey responses showed that they could generate ideas for alternate templates, but it’s still not clear that contributing to a larger picture is a compelling enough reason for people to take photos that match a template. People might just prefer to upload photos of themselves with the recipient, or make other creative use of the tile. But if we remove the template aspect of Card Collage, then it becomes quite similar to Pinterest or Path, and becomes less collaborative and less unique. Our main challenge going forward is therefore to verify the importance of templates as a motivating aspect of collaborative card-making.