Swain
UX Research
Longitudinal Beta Study
Project Overview
Email automation is an essential tool for marketers who wish to scale their email marketing (EMM) operation. At the time of this project, a new generation of email automation functionality had been released in beta to a small segment of our most highly engaged product users. This research project was designed to collect feedback on the discoverability, intelligibility and ease of use of this new feature. Additionally, I sought to identify any important functional omissions and technical bugs in the beta release.
Process
Email automation campaigns take place over an extended period of time, often weeks or even months. Users who were given access to this beta functionality would not be able to experience it fully until they had worked with it for some time. For this reason, I chose a longitudinal research structure with three participant touchpoints, each spaced three to four weeks apart.
The product team responsible for this next generation “autoresponder” functionality provided beta access to 59 active product users who had expressed interest in email automation. After my initial introduction and expectation setting, the first two touch points came in the form of online surveys, while the third and last would be a brief phone interview. I chose to begin with online surveys because they are relatively quick and unobtrusive, and would therefore receive the highest take-rate among study participants.
The first survey, completed by 45 participants, was focused on feature discoverability, intelligibility and usability. Since this is a tool that requires ongoing user engagement over an extended period of time , I was curious about possible differences between initial usability findings and the longer term user experience. The second online survey, completed by 35 participants three weeks after the first, focused on feature utility and the related performance reporting interface. This second survey incorporated a System Usability Scale (SUS) questionnaire so as to provide product stakeholders a usability measure that could be compared to a well established, external usability benchmark.
Three to four weeks after the second online survey was completed, I conducted a 15 minute phone interview with 15 of the most engaged study participants. I chose the interview format over a third survey because it afforded me the opportunity to explore the unique nature of each participant’s experience with the tool. Since EMM can take many forms as it is applied to different industries and according to different marketing strategies, participant experiences with the new functionality varied widely. Additionally, each participant came to the study with a different level of personal experience with email automation, which significantly impacted their perspective on the usability and utility of the functionality.
Outcome & Reflection
This study demonstrated to the satisfaction of product stakeholders that the next generation email automation function was sufficiently discoverable, usable and bug free to release in the production environment. Feedback from the second survey and participant interviews did, however, indicate significant shortcomings in the email performance reporting interface. As a result, product stakeholders targeted this performance reporting interface for future refinement.
The relatively small sample size under consideration in this study limited the statistical reliability of the data generated. No reliably generalizable inferences can be made from a sample size of 45, so the survey results must be treated as qualitative feedback. I opted to incorporate a SUS questionnaire in part because this measure is considered effective even with relatively small sample sizes. In this case, the product stakeholders were comfortable proceeding on the basis of qualitative feedback, but the study itself would have been technically enhanced if the sample population were significantly increased.