Research Update
October 20, 2022
Doodle & When2Meet: A user study
After implementing prototypes of When2Meet and Doodle, I compared user experiences with a user evaluation study.
Project Dates: October 2022
Built with: HTML, TypeScript, CSS, Ruby
A Doodle prototype representation used in the user evaluation study.
When2Meet and Doodle are a web-based calendar apps designed to help coordinate events for a large number of people. There are substantial UI differences between the two solutions, outlined below, which warrants an exploration of the differences in usability between the two interfaces.
Need for Cognition (NFC)
A concept that became extremely relevant in the study was Need for Cognition (NFC), which reflects a person's willingness to undertake difficult mental activities. It is often associated with exploratory behavior and deep learning activities.
Hypotheses
First, because When2Meet has some made-for-a-pointer UI elements (click and drag), it can be hypothesized that participants on mobile devices will have a harder time selecting their availability. Related to the our study, it can be hypothesized that participants with higher NFC scores will feel more comfortable using the more complicated When2Meet application. However, one can expect that throughout all groups, Doodle will be perceived as more useful and easy to use. Finally, as described by Reinecke et al., it can be hypothesized that some participants will display individualist behavior and try to arrange meeting times that better fit them, even if they don't work for some in the rest of the group.
Research Question
Which of the two applications, When2Meet or Doodle, is perceived as more useful and easy to use?
User evaluation study
Participants were asked to complete four different tasks sequentially. The tasks were split into two clusters, the Doodle and When2Meet cluster, based on which scheduling solution they were testing. To ensure uniformness, the order of the tasks within the two clusters was randomized for every participant; then the presentation order of the task clusters was also randomized. I chose not to interject tasks of a clusters in between two tasks of another cluster, so as to minimize the cognitive load caused by the need to switch between the distinct design concepts of the two scheduling solutions.
To access their need for cognition, participants were asked to rate how characteristic 18 statements were for them personally by choosing a rating of 1-5 (1: Extremely Uncharacteristic, 2: Somewhat Uncharacteristic, 3: Uncertain 4: Somewhat Characteristic, 5: Extremely Characteristic). Questions are detailed in the full report in the Resources section below.
When2Meet Availability Filling
To study how users perceive the When2Meet platform and confirm user attitudes as described by Reinecke et al., participants were asked to complete a When2Meet poll with the goal of scheduling an all-hands meeting with their team. The When2Meet poll contained 5 set responses, with some times being suitable for all participants, and some other times being chosen by two or three participants. To ensure consistency, the participants were provided with a pre-filled work calendar schedule and were told to fill their availability in the When2Meet poll.
Prefilled When2Meet calendar presented to participants to fill.
Doodle Availability Filling
To enable comparisons between platforms, and once again extend the findings of Reinecke et al., participants were also asked to complete a Doodle poll with the goal of scheduling an all-hands meeting with their team. Similar patterns to the When2Meet poll were presented to participants to enable cross-comparisons. However, the availability windows were shifted so as to prevent participants from cross-referencing their answers between the two tasks. In this task, the maybe feature of Doodle was selected for some of the pre-filled responses to test the attitudes of participants towards if-need-be availability, consistent with our third hypothesis.
As described above, the use of When2Meet and Doodle presents two different use cases: filling out one's availability, and selecting the most suitable time for a meeting based on existing responses. The following two tasks examined participants' attitudes in the second use case.
When2Meet Meeting Time Decision
To study how efficiently users can make meeting time decisions based on submitted responses by other users, participants were asked to identify the best timeslot for a one hour meeting based on a pre-filled When2Meet poll. They were given a When2Meet poll that contained 5 set responses, and were provided with a pre-filled work calendar schedule they were told to assume was theirs. The When2Meet responses were designed in a way such that the responders had a timeslot that worked well for all users who filled out the poll, but was not ideal for the participant (either between a chain of meetings or very late into the workday, stretching after 5pm). There were a couple of timeslots that were ideal for the participant but would exclude a poll respondent for at least half the meeting. All other possible timeslots were non-optimal for both the participant and the When2Meet respondents.
Doodle Meeting Time Decision
To enable comparisons between platforms, participants were asked to identify the best timeslot for a one hour meeting based on a pre-filled Doodle poll. Similar patterns to the When2Meet poll were presented to participants to enable cross-comparisons. However, the availability windows were shifted so as to prevent participants from cross-referencing their answers between the two tasks. Two timeslots had one respondent's answer marked as maybe to test the attitudes of participants towards if-need-be availability, consistent with our fourth hypothesis. All other possible timeslots were non-optimal for both the participant and the Doodle respondents.
Participants
To understand how useful and efficient the current iterations of the Doodle and When2Meet are for users, a diverse group of users was invited to participate in a usability evaluation study. The group contained four different subgroups:
8 users that had never used the When2Meet application before, but had familiarity with the Doodle application,
8 users that had never used the Doodle application before, but had familiarity with the When2Meet application,
8 users that had never used either the When2Meet or Doodle applications before, and
8 users that had used both the When2Meet and Doodle applications before.
The study was limited to current Windows, Android, iOS and macOS users that had been using the respective device for at least six months, to ensure that users were familiar on how to operate said devices.
Results
After each task, participants were asked to rate each application on its ease of use and utility on a scale from 1-7, with 1 being "Extremely difficult to use/Would never choose to use again'' and 7 being "Extremely easy to use/Would definitely choose to use every time I needed a tool like this.''
The user study results show a clear difference in participants' perception of utility and ease of use between the two applications, with a strong preference for Doodle. When asked for qualitative feedback, participants shared that the Doodle cell-based look resembled a calendar view they were more familiar with and made it easier to navigate. Mobile platform (iOS and Android) participants shared that the When2Meet drag functionality was extremely difficult to navigate, and when they tried to determine the right timeslot for the meeting in the second set of tasks, the interface was even less responsive.
The results point to a clear upwards trend of the participants' perception of utility and ease of use for both applications. More significantly, the margin between the participants' perceptions of the two applications shrank with an increase of their Need for Cognition. Once again, however, all groups rated Doodle higher for utility and ease of use.
Finally, the results show that participants were mostly successful in choosing the best timeslot for both the participant and the poll respondents. However, as also reported by Reinecke et al., some participants chose the best timeslot for them but a sub-optimal timeslot for some poll respondents. Finally, there were some participants that chose a non-optimal timeslot for both the participant and the poll respondents. Of note is that more participants chose a non-optiomal/sub-optimal timeslot in the When2Meet task compared to the Doodle task. Moreover, the NFC scores were more widespread when working on the When2Meet task, with participants choosing a suboptimal timelsot having a moderate NFC compared to when working on the Doodle task.
Outlook and limitations
There was a clear difference in how participants perceived the utility and ease of use of When2Meet and Doodle based on the device they used. As hypothesized, mobile devices provided a worse experience compared to desktop devices. However, more widespread UI elements (checkboxes) of Doodle provided a better experience across the board compared to the niche UI elements (drag) of When2Meet. Second, participants with a higher tolerance to complexity (higher NFC scores) showed a general willingness to engage with both applications, no matter how complex they are. Nonetheless, as expected, the difference between the two applications in terms of utility and ease of use remained across NFC groups. The study results also verify that there is a group of users who behave in the way that Reinecke et al. reported by selecting a sub-optimal timeslot for the people they are trying to meet with so that the times works better for them. Finally, the results show that a more complicated application (When2Meet) will make more users choose an incorrect timeslot that is non-optimal for everyone involved. This is also supported by the fact that most participants with high/moderate NFC scores chose the best timeslot for all, while many of those participants ended up choosing an incorrect timeslot (increasing the average NFC scores for each of the groups in the When2Meet task).
These results raise a number of questions for future work both in adapting these applications and developing new meeting scheduling applications. Related to the question of the nature of the device to the participant's perception of utility or ease of use of an application, it is still an open question if these results generalize to other scheduling applications. In the area of NFC, it remains to be seen if the mapping of the simplicity of an application to the types of users that will choose to engage with said application generalizes to more application. Finally, a larger study could validate these empirical findings and provide statistically significant evidence for the hypotheses examined in this paper.
Notes
I would like to personally thank Prof. Elena Glassman, who provided helpful comments throughout the writing process of this project.
