CSUQ: An Application for Medical Device Usability
Medical device manufacturers have good reason to focus primarily on safety. The risk of device failure and adverse effects on the human operator are significant. There are also potential regulatory and financial impacts, not to mention the reputation of the manufacturer. However, the development of a medical device should also consider other factors, namely ease of use and satisfaction of the user – especially as more and more medical devices are being designed to be used by a lay person (as opposed to a trained medical professional), and consumers will have a choice between competing devices. This analysis examines the use of a well-established post-test usability assessment, the Computer System Usability Questionnaire (CSUQ) for evaluating pre- and post-design perceptions of usability and satisfaction. The CSUQ is a 19-item inventory developed by J.R. Lewis from IBM1, and measures four constructs of interface usability: Overall, System Usefulness, Interface Quality and Information Quality. This instrument has been used to evaluate software, websites, hardware, and myriad other types of interfaces. Employing valid measurements of usability and user satisfaction are especially critical when the user has a choice in the medical device they are purchasing or using, as was the case for this product. This paper presents data demonstrating its utility for a consumer-oriented medical device2. It specifically demonstrates how applying the user centered design process resulted in higher user satisfaction with the overall interface design as measured by the CSUQ inventory.
Methodology: Establishing a Baseline Initially, we aimed to capture a cross-section of attitudes towards device usability from current users of the product. A comprehensive survey consisting of the 19 CSUQ questions measured using a 7-point Likert scale, as well as 69 additional questions was administered to a sample of 3500 current end-users of the medical device within the United States. Approximately 4513 (12.8%) valid responses4 were collected from this initial sample. This data was used to establish a baseline ahead of a significant re-design of the user interface and product feature set. The table below summarizes the data from the baseline sample. This research process used in this project followed the basic pre/post-test protocol outlined below. Additionally, for this analysis we explored the relationship between the baseline CSUQ measure and an outcome measure of interest to the organization: intent to continue use of the medical device. Because the medical device is adjunctive, or optional, the user has the choice to continue its use or discontinue its use independent of their present medical condition. That is, there are alternatives to managing their disease that do not rely on this specific technology; however, this technology, if designed well, could certainly afford substantial improvements in their daily functioning and the state of their health.
Applying User-Centered Design Practice As the new product development process began, the project team planned on conducting numerous usability engineering activities, including: contextual inquiry, heuristic evaluations, developing multiple design concepts, iterative usability testing, and conducting a final summative validation test. Formative usability testing was used to identify and correct user interface design flaws leading to use errors and the inability of users to complete tasks. A total of nine (9) formative usability tests were conducted during the product development process, culminating in a summative usability test. The CSUQ assessment was administered after the summative usability test. A statistically significant difference was observed between the Baseline mean and the CSUQ mean for the new product (t = 7.92, DF = 37, p < .000).
The Effect on Intent to Continue Use We wanted to evaluate the effect of the CSUQ score on a critical behavioral metric: intended continued use. Since the medical device in question was technically adjunctive (optional), users could decide to discontinue their use of the device at any time. The original baseline CSUQ asked respondents to report their intent to continue using the medical device 6 months from the point of the survey. Of the sample collected, 14% indicated they did not intend to continue using the device, the remaining 86% indicated they intended to continue using the device. While there are a number of variables that could contribute to a user’s stated intent, we focused on a handful of variables that could contribute to this intent. These variables includes: Age, Gender, Years with the Disease, and Device User (smartphone vs. not smartphone user). We used Logistic Regression to develop a predictive model that could model the relationship between these variables, CSUQ score, and the intent to continue using the medical device 6 months from now. The table below shows the results of the Hierarchical Logistic Regression Analysis. The Logistic Regression analysis determined that after considering the effects of age, gender, disease years, and device usage type, the CSUQ score was a statistically significant predictor of the user’s intent to continue using the medical device. In fact, none of the demographic variables were found to be significant predictors at all. Thus, we can be relatively confident that despite differences due to these demographic effects, users that report a higher degree of usability as measured vis a vis the CSUQ are less likely to consider discontinuing use of their adjunctive medical device. Specifically, for each point increase in the CSUQ score (i.e. 4 to 5, 5 to 6, etc.) there is a 72% reduction in the probability of intent to discontinue. The graph below shows the effect of the CSUQ score on the associated continued use probability. Given that sample base rate for intent to discontinue is 14%, we can set an approximate heuristic that any CSUQ score lower than about 5.5 introduces a greater risk of discontinued use than what is naturally occurring in our sample. In terms of accuracy, the overall model is about 73% accurate in classifying cases into the two groups, leaving lots of room to improve the model efficacy by considering other influential variables, etc.
Conclusion The purpose of this analysis was to demonstrate several important aspects of usability engineering and the use of advanced statistical methods to demonstrate how data can be used to achieve specific project goals. Namely, that usability engineering practices resulted in a significant improvement in the user experience relative to a baseline. Secondly, relating objective measures to an important outcome indicator, such as intent to use or stop using an adjunctive product. This analysis, though mostly for demonstrative purposes, identified that usability engineering efforts resulted in a statistically significant improvement in the measured user experience (usability and satisfaction). In turn, the validated measure (CSUQ) of usability predicts a desirable outcome for the organization (and product), and provides a convenient cut-off score by which product teams can use to gauge the design and perceived usability of the product. There is quite a bit of additional analysis that can (and should be done) with this kind of data, and a trained statistician or data scientist can provide the right guidance on designing the right data collection and analysis methods. Designing the right assessment depends heavily on the goals and nature of the product being developed, as well as the readiness of the organization. However, in most cases demonstrating the utility of the data through advanced statistical methods and techniques will help move the organization forward towards data-driven decision-making practices and showcase the ability of your team
