Nancy Lightner, Helen Fuller, Timothy Arnold (2023). Highlighting Usability in Healthcare System Selection. In: Healthcare and Medical Devices. AHFE Open Access, vol. 79, pp. 244–253. https://doi.org/10.54941/ahfe1003493
1 Department of Veterans Affairs, Indianapolis, IN 47340, USA
2 Human Factors Engineering, Clinical Informatics and Data Management Office, Office of Health Informatics, Veterans Health Administration, Department of Veterans, Affairs, Washington, DC, USA ↗
3 Human Factors Engineering, Clinical Informatics and Data Management Office, Office of Health Informatics, Veterans Health Administration, Department of Veterans Affairs, Washington, DC, USA ↗
When an organization plans to replace a healthcare information system, usability initially seems like a critical evaluation component. While leadership often cites “user experience” as important, as the organization must also consider other important components such as cost, functionality, and schedule, they may reduce the emphasis on usability. This paper presents a method to highlight usability concerns throughout the system selection process.
Studies show that usability concerns not only influence Electronic Health Record (EHR) acceptance (Maillet et al., 2015) but have links to patient safety issues (Carayon & Hoonakker, 2019; Pew et al., 2018). While research has recommended principles and processes to improve usability, it usually provides guidance for system design (Gould & Lewis, 1985; Jacko, 2012; van Velsen et al., 2022) or evaluation of a system already in use (Dixit et al., 2023; Edwards et al., 2008; Goldstein et al., 2023). In the meantime, to reduce cost and accelerate deployment, organizations have transitioned from developing custom systems to purchasing Commercially available Off-The-Shelf (COTS) products. This especially holds true for those planning to implement an EHR, since they are complex and involve protected patient and billing data. Though guidance exists for considerations of usability in the software acquisition process (e.g., DAU, 2023), implementation remains challenging.
Several acquisition best practices exist, including identifying functions a new system should perform, evaluating the extent to which each system meets those requirements, and contracting with the supplier to provide them. For example, the Capability Maturity Model® Integration for Acquisition (CMMI-ACQ) “enables organizations to avoid or eliminate barriers in the acquisition process…” (Richter, 2008, p. 3). The ISO/IEC 12207:2017 Systems and software engineering – System life cycle processes (ISO, 2017) indicates that the processes, activities, and tasks also apply during the acquisition of a software system, including those establishing an agreement
Highlighting Usability in Healthcare System Selection 245
between a supplier and the acquirer. The International Organization for Standardization (ISO) defines usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use”(ISO, 2018). The field of Project Management also has interest in COTS acquisition, as it meets the definition of a project as a unique and temporary job (Project Management Institute, 2023; Douglas, 2011). For a description and comparison of these standards, see Alfaraj and Qin (2010) and Pino et al. (2010). The Defense Acquisition University (DAU) also details a process that includes an “Evaluation” step.
The publications mentioned all have similar general steps, but none specify details within them. This paper proposes how to incorporate consideration for evaluating usability when selecting a healthcare related COTS product such as an EHR or related systems.
BACKGROUND
According to the United States (U.S.) Office of the National Coordinator (ONC) for Health Information Technology for Economic and Clinical Health (HITECH), as of 2021, 78% of office-based physicians and 96% of non- federal acute care hospitals have adopted a certified EHR (ONC for Health Information Technology, n.d.; ONC for Health Information Technology, 2022). Many attribute that adoption rate to the HITECH Act of 2009, legislation that placed EHR technologies at the center of health system reform in the U.S. (Peterson & Holman, n.d.). Similarly, the United Nations adopted a sustainable development goal of ensuring healthy lives and promoting well- being for all at all ages, for achievement by 2030 (United Nations General Assembly, 2015). These global goals entice healthcare systems in low- and moderate-income countries to strive to adopt an EHR.
While many associate the use of EHRs with improved healthcare, difficulties remain, especially with clinician burnout resulting from system complexity and the extreme amount of documentation required (Kutney- Lee et al., 2019; Melnick et al., 2020; Melnick et al., 2021). A literature review by Kruse et al. (2022) identified that in 21% of studies mitigation of physician burnout resulting from poor EHR usability required interface redesign. The non-intuitive nature of the interface and interface design issues required longer work hours, resulting in burnout (Colicchio, et al., 2019; Melnick et al., 2019; Melnick et al., 2021; Shulte & Fry, 2019).
The combination of need for EHR functionality, likely from a COTS product, and the importance of system usability, plus the lack of specific guidance on how to evaluate usability during an acquisition project results in organizations doing the best they can and hoping for good outcomes.
APPROACH Because organizations may consider selection of a COTS product a project, they have project management constraints imposed by the Iron Triangle of project management: scope, cost, and schedule (Ebbesen & Hope, 2013;
Pinto, 2010; Pollack et al., 2018). This interaction describes the trade-offs between the constraints and the resulting effects on quality (as defined by the organization) (Microsoft, 2023). For example, with enough money and time, the organization can find resources to incorporate needed tasks such as a comprehensive evaluation. Or, with sufficient resources but limited time, an organization may need to reduce the scope of tasks they perform, including a comprehensive usability analysis. Thinking about and talking through trade-offs can provide guidance for organizations acquiring COTS medical software systems. Quality appears at the center of the triangle and reflects changes to the other segments. Figure 1 Part a shows the original balanced triangle. Part b shows how the quality segment remains balanced when the other segments are increased. Part c shows how quality increases if the schedule and cost segments remain the same because of the reduction in scope. The organization and project team define “Quality” prior to the project start, with measures and metrics to compute throughout the project.
With the Iron Triangle and acquisition process steps provided by CMMI- ACQ, ISO/IEC/IEEE 12207:2017 and DAU, we propose ways to incorporate usability evaluation into medical COTS product selection. The following incorporates usability research and practices into an organizational process of selection and acquisition of an EHR or related software system. The process presented has the following assumptions about the acquisition scenario:
Organizational and acquisition leadership recognize the importance of usability evaluation when comparing EHR and other healthcare-related products for use in their organization.
A sample of appropriate end users (clinical and other staff and system administrators) have availability to provide evaluation input for their respective functional goals.
Figure 1: The project management iron triangle of constraints (adapted from microsoft, 2023).
Highlighting Usability in Healthcare System Selection 247
In evaluation scenarios requiring hands-on interaction by the acquiring organization, usability experts have access to a representative system and test account for evaluations.
Vendors value the effort enough to allow the acquiring organization time and access to evaluate system usability and discuss findings.
Vendors provide the acquiring organization access to subject matter experts who can supply information about the system.
Vendors allow interviews of past users during the Market Research step for the acquiring organization to gain understanding of usability concerns.
The technical environment and data the vendor uses to demonstrate their system realistically represents the target production environment for all expected levels of complexity.
In the next sections, we suggest guidance following the DAU Contracting Process for Acquisitions, Pre-Solicitation and Solicitation-Award phases.
Pre-Solicitation Initial Planning / Form the Team – Include a Lead Usability team member who has the responsibility of incorporating usability into the discussion and evaluation throughout the acquisition project. The Lead may have additional staff assigned to assist in completing activities in subsequent steps. In preparation for Market Research, determine which COTS products are available that can perform the general functions identified for the project.
Market Research – Investigate usability concerns of COTS products when used to perform generally expected functions. A search of social media and U.S. Food and Drug Administration reports may identify usability issues (Fuller and Arnold, 2019). As these are commercial products, publications likely do not name the product, especially because some have clauses in their contracts that prohibit discussion of difficulties encountered with EHR use (Tahir, 2015).
Define Requirements – Define usability requirements based on commonly- used tools, such as Usability Heuristics (Nielsen, 1994), and Success Rate (Nielsen & Budiu, 2001), based on functional requirements established by acquisition team members. Request for Proposals typically contain a list of requirements the proposed system should meet. Stakeholders, such as clinicians, create this list, ideally following discussions with human factors staff to determine the evaluation criteria within the recommended evaluation process.
Acquisition Business Strategy – The DAU defines this step as determining the business, technical, product support, security, and supportability strategies that the project management team plans to employ to manage program risks and meet objectives. This strategy evolves over time, especially if the business environment changes. The strategy should address requirements for system performance as well as business risks. As this pertains to usability, it should include contractual provisions for changes if the team discovers design issues post-implementation or the clinical needs change. For example, in the COVID-19 pandemic, healthcare systems
implemented changes to track patient disease and vaccine status (Adams et al., 2021). Before reliable COVID-19 testing existed, if a patient had symptoms and exposure but a negative test result, the EHR needed to present this information in a way that supported clinical decision making. In addition, the logic required for determining the correct type and frequency for vaccine administration was more complicated than for previous vaccines.
Solicitation-Award Solicitation – Prior to the publication of the solicitation, the usability team should develop a list of questions for vendors, including contact information of organizations currently using the COTS product under consideration. They should also determine the evaluation criteria and establish evaluation methods to use during the evaluation step.
Evaluation – The CMMI and ISO guidance both suggest that the evaluation step should include a demonstration to confirm functionality. As usability contains the components of effectiveness, efficiency and satisfaction, the evaluation should cover all three. Ideally, the sample users would have the opportunity to try to complete functions hands-on, perhaps after a brief training program. If the vendor cannot provide a hands-on experience, the users should witness a demonstration. If the demonstration exists within the vendor environment (most likely), a risk exists that it will not perform the same way in the production environment in ways that will affect the evaluation. The authors suggest the following to evaluate each section of usability: Effectiveness – a sample of representative end users should either attempt to complete a set of functions or witness a demonstration of the functionality. After each function shown, each of the sample users should complete a form documenting the level of success (Complete, Success with a minor issue, Success with a major issue, Failure). This determines the extent to which the COTS product can perform needed functionality as well as if it may pose a risk to patient safety.
Efficiency – usability experts or those suitably trained can perform efficiency evaluations for each system demonstrated, including collecting task time and interface interactions.
Satisfaction – the System Usability Scale (SUS) (Brooke, 1996) provides a relatively easy method of documenting perceptions of usability. Each end user would complete a SUS for each system under consideration. Composite scoring would allow for comparisons between systems. Negotiation – typically negotiation takes place regarding price, scope, and schedule. This step allows the usability team to identify known issues and contracting agents to negotiate for resolution. Because the brief exposure the team has to the product during the demonstration phase likely will not provide adequate understanding, ideally the resulting contract would contain specifics related to training and responses to usability concerns after deployment. Because COTS products have design constraints, expect changes to cost more than accepting the off-the-shelf product.
Highlighting Usability in Healthcare System Selection 249
Negotiation also includes how to resolve usability concerns discovered after contract award. Identifying those concerns requires significant organizational commitment to monitoring and measuring patient safety metrics and alerting the contracting personnel with requested modifications.
Award – this step involves signing a contract with a COTS vendor for deployment in a healthcare organization.
DISCUSSION
Assessing healthcare system usability requires more than a simplified view of the user experience. Usability includes effectiveness, efficiency, and satisfaction, where in this context effectiveness includes both efficacy and safety, including patient, worker, and organizational safety. Design challenges may remain hidden until a user performs all functions and interactions with a system, usually after an organization signs a contract for the system. The process proposed in this paper accounts for project management tips and guidance from established standards organizations. It takes into consideration that usability, although deemed important, may not have a well-defined role in the acquisition process. Because reviews for usability considerations may take longer than expected, including usability criteria may confound the cost and schedule constraints of the Iron Triangle. In some cases, the requirements of a usability evaluation may reduce or eliminate it from the project evaluation process.
This leads back to the Iron Triangle of constraints in Figure 1. Rigorous evaluation of the three components of usability, effectiveness, efficiency, and satisfaction, takes planning, time, and detailed attention to execute. Regardless of the location and technical platform, this includes ensuring permission for and feasibility of recording and viewing demonstrations for playback to evaluate the effectiveness component (the extent to which the system performs specified functions).
Planning for a demonstration and/or training and hands-on interaction with the system depends on whether it will take place online, at the vendors or the healthcare organization location(s), or a combination of online and on-site.
Online preparation includes ensuring availability of evaluators and feasibility of viewing remotely. The process must include contingency plans if evaluators have limited or no availability or the technology fails.
Planning for on-site demonstrations includes travel to either the vendors’ locations or to one site, which complicates the availability of representative clinical end users for healthcare systems with multiple hospitals and clinics across a wide geographical area. Since two of the three evaluation methods rely on actual end users, such as clinicians, preparation for an extended leave requires advanced planning including re-scheduling patient appointments. The logistics of viewing demonstrations or taking part in training and hands-on system interaction involves finding an appropriate location that can accommodate the vendors, users, and human factors evaluators for an extended time. This planning must also include consideration
for separation of vendors to avoid identification of other systems in contention.
Preparation also includes making the evaluation instruments available. Tools exist for online survey access, with a print option in case evaluators prefer a paper copy.
Planning must also include training on how to complete the evaluation using the selected instrument.
Ideally, having a dry run of the process and expectations for the evaluation would reduce the risk of difficulties. Once acquisition leadership reviews the complexity and resources needed to perform a comprehensive usability evaluation, however, they may determine that the trade-off between schedule and scope warrants a less rigorous evaluation, even if it results in a reduction in quality.
LIMITATIONS The assumptions described previously may not always hold, and consideration for using this process to award a contract should include how to detect and account for when they do not. Most of the assumptions address actions of the proposing vendors while a few relate to those of the acquiring organization. Two of those assumptions require the organization to prioritize a culture of safety and end-user usability. The assumption that the demonstration environment and data provide a realistic representation of the target production environment may not hold true and may cause issues post award. This emphasizes the need for negotiation to resolve any issues.
While providing a productive and satisfying user experience appears high on the list of new system priorities, too frequently other constraints, such as time and money, overshadow that goal. This includes the common practice of only considering end user experience after system deployment, when users complain of difficult system interactions, possibly resulting in patient safety concerns. At that point, unless accounted for in the contract, resolution may have steep costs, making it infeasible.
NEXT STEPS Usability consideration for information system acquisition has historically focused on aspects of the task, the technology, and end users (Card, Moran, & Newell, 1983). This paper presents a process that incorporates usability and real-world organizational concerns to produce an evaluation method during the system acquisition phase.
The next steps consist of refining the process through testing it in organizations, potentially revealing additional constraints or other considerations. Testing this process has the goal of increasing awareness of usability concerns in a COTS product for selection and use. It may result in more informed negotiation concerning usability and user acceptance testing and future design changes prior to award.
Following a strictly sequential process as described in project management material may result in significant rework or renegotiation post-award.
Highlighting Usability in Healthcare System Selection 251
Investigating how to apply other project management frameworks such as Agile may act as another next step, although they likely have hidden complexities as well.
REFERENCES
Adams, K., McClure, M. L., & Fuller, H. J. (2021). A description of rapid design and implementation of new features in an electronic health record in the United States Department of Veterans Affairs, Veterans Health Administration response to the Covid-19 pandemic. Knowledge Management & E-Learning, 13(4), 438. Adler-Milstein, J., Raphael, K., Bonner, A., Pelton, L., & Fulmer, T. (2020).
Hospital adoption of electronic health record functions to support age-friendly care: results from a national survey. Journal of the American Medical Informatics Association, 27(8), 1206–1213. Alfaraj, H. and Qin, S., 2010. Deriving Software Acquisition Process from Maturity Models—An Experience Report. Journal of Software Engineering and Applications, 3(03), p. 280. Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in
industry, 189(194), 4–7. Carayon, P., & Hoonakker, P. (2019). Human factors and usability for health information technology: old and new challenges. Yearbook of medical informatics, 28(01), 071–077. Colicchio, T. K., Cimino, J. J., & Del Fiol, G. (2019). Unintended consequences of nationwide electronic health record adoption: challenges and opportunities in the post-meaningful use era. Journal of medical Internet research, 21(6), e13313. Defense Acquisition University (DAU). (2023). Software Acquisition. Retrieved from
https://aaf.dau.edu/aaf/software/. Dixit, R. A., Boxley, C. L., Samuel, S., Mohan, V., Ratwani, R. M. and Gold, J. A., 2023. Electronic Health Record Use Issues and Diagnostic Error: A Scoping Review and Framework. Journal of Patient Safety, 19(1), pp. e25–e30. Douglas, J. (2011). COTS project management strategy from a state government
PMO perspective. Paper presented at PMI® Global Congress 2011—North America, Dallas, TX. Newtown Square, PA: Project Management Institute. Ebbesen, J. B. and Hope, A. J. (2013) Re-Imagining the Iron Triangle: Embedding Sustainability into Project Constraints. PM World Journal, Volume II, Issue III. pp. 1–13. Edwards, P. J. Edwards, Moloney, K. P., Jacko, J. A. Jacko, Sainfort, F. (2008).
Evaluating usability of a commercial electronic health record: A case study. International Journal of Human-Computer Studies, 66 (10), Pages 718-728, ISSN 1071-5819, https://doi.org/10.1016/j.ijhcs.2008.06.002. Fuller, H. J. A., & Arnold, T. (2019). Identifying Medical Equipment Usability Issues from Social Media Reports. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, 8(1), 217–221. Sage CA: Los Angeles, CA: SAGE Publications. https://doi.org/10.1177/2327857919081055 Goldstein, J. E., Guo, X., Boland, M. V. and Smith, K. E., 2023. Visual Acuity:
Assessment of Data Quality and Usability in an Electronic Health Record System. Ophthalmology Science, 3(1), p.100215. Gould, J. D., & Lewis, C. (1985). Designing for usability: key principles and what designers think. Communications of the ACM, 28(3), 300–311. International Organization for Standardization (ISO). (2017). IEEE ISO/IEC/IEEE
12207:2017(E). Systems and software engineering — Software life cycle processes.
International Organization for Standardization (ISO). (2018). ISO 9241- 11:2018(en). Ergonomics of human-system interaction – Part 11: Usability: Definitions and concepts. Website: https://www.iso.org/obp/ui/#iso: std: iso:9241:-11: ed-2: v1: en. Jacko, J. A. ed., 2012. Human computer interaction handbook: Fundamentals, evolving technologies, and emerging applications. Kruse, C. S., Mileski, M., Dray, G., Johnson, Z., Shaw, C., & Shirodkar, H. (2022).
Physician Burnout and the Electronic Health Record Leading Up to and During the First Year of COVID-19: Systematic Review. Journal of Medical Internet Research, 24(3), e36200. Kutney-Lee, A., Sloane, D. M., Bowles, K. H., Burns, L. R., & Aiken, L. H. (2019).
Electronic health record adoption and nurse reports of usability and quality of care: the role of work environment. Applied clinical informatics, 10(01), 129–139. Maillet, É., Mathieu, L., & Sicotte, C. (2015). Modeling factors explaining the acceptance, actual use and satisfaction of nurses using an Electronic Patient Record in acute care settings: An extension of the UTAUT. International journal of medical informatics, 84(1), 36–47. Melnick, E. R., Dyrbye, L. N., Sinsky, C. A., Trockel, M., West, C. P., Nedelec, L.,... &
Shanafelt, T. (2020, March). The association between perceived electronic health record usability and professional burnout among US physicians. In Mayo Clinic Proceedings (Vol. 95, No. 3, pp. 476–487). Elsevier. Melnick, E. R., West, C. P., Nath, B., Cipriano, P. F., Peterson, C., Satele, D. V.,... & Dyrbye, L. N. (2021). The association between perceived electronic health record usability and professional burnout among US nurses. Journal of the American Medical Informatics Association, 28(8), 1632–1641. Microsoft (2023). The project triangle. Website: https://support.microsoft.co m/en-us/office/the-project-triangle-8c892e06-d761-4d40-8e1f-17b33fdcf810 #__toc322692214. Nielsen, J. (1994). Enhancing the explanatory power of usability heuristics. Proc.
ACM CHI’94 Conf. (Boston, MA, April 24-28), 152–158. Nielsen, J., Budiu, R. (2001). Success Rate: The Simplest Usability Metric. NN/g Nielsen. Office of the National Coordinator for Health Information Technology. (2022).
Adoption of Electronic Health Records by Hospital Service Type 2019- 2021, Health IT Quick Stat #60. Website: Adoption of Electronic Health Records by Hospital Service Type 2019–2021 | HealthIT.gov. Office of the National Coordinator for Health Information Technology. ‘National Trends in Hospital and Physician Adoption of Electronic Health Records,’ Health IT Quick-Stat #61. Website: https://www.healthit.gov/data/quickstats/adoption- electronic-health-records-hospital-service-type-2019-2021. Petersen, S and Holman, T. (n.d.). HITECH (Health Information Technology
for Economic and Clinical Health) Act of 2009. TechTarget Health IT. Website: https://www.techtarget.com/searchhealthit/definition/HITECH-Act#: \protect$\relax\sim$:text=The%20HITECH%20%28Health%20Information %20Technology%20for%20Economic%20and,Act%20of%202009%20% 28%20ARRA%29%20economic%20stimulus%20bill. Pew, MedStar Health, and the American Medical Association. (2018). Ways to Improve Electronic Health Record Safety: Rigorous testing and establishment of voluntary criteria can protect patients. The Pew Charitable Trusts, August 28, 2018.
Highlighting Usability in Healthcare System Selection 253
Pino, F. J., Baldassarre, M. T., Piattini, M., Visaggio, G., & Caivano, D.
(2010). Mapping software acquisition practices from ISO 12207 and CMMI. In Evaluation of Novel Approaches to Software Engineering: 3rd and 4th International Conferences, ENASE 2008/2009, Funchal, Madeira, Portugal, May 4-7, 2008/Milan, Italy, May 9-10, 2009. Revised Selected Papers 3 (pp. 234–247). Springer Berlin Heidelberg. Pinto, J. (2010). Project Management: Achieving Competitive Advantage. New Jersey: Pearson Education. Pollack, J. Helm and J. Adler, D. (2018) What is the Iron Triangle, and how has it changed? International Journal of Managing Projects in Business, Volume 11, Issue 2, pp. 527–547. Project Management Institute. (2023). What is Project Management. Website: https:
//www.pmi.org/about/learn-about-pmi/what-is-project-management. Richter, Karen. (2008). CMMI for Acquisition (CMMI-ACQ) Primer, Version 1.2 (CMU/SEI-2008-TR-010). Retrieved January 27, 2023, from the Software Engineering Institute, Carnegie Mellon University website: http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=8711. Schulte, F., & Fry, E. (2019). Death by 1, 000 clicks: Where electronic health records
went wrong. Kaiser Health News, 18. Tahir, D. (2015). Doctors barred from discussing safety glitches in U. S.-funded software. Politico. Website: https://www.politico.com/story/2015/09/doctors-ba rred-from-discussing-safety-glitches-in-us-funded-software-213553. United Nations General Assembly. (2015). Transforming our world: the 2030 Agenda for Sustainable Development. In: Seventieth United Nations General Assembly, New York, 25 September. New York: United Nations. Website: https://undocs.org/en/A/RES/70/1. van Velsen, L., Ludden, G. and Grünloh, C., 2022. The Limitations of User-
and Human-Centered Design in an eHealth Context and How to Move Beyond Them. Journal of Medical Internet Research, 24(10), p. e37341.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Healthcare and Medical Devices
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-55-1
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional