Caio Márcio Almeida e Silva, Alexandre Barros Neves, Stephania Padovani (2022). Criteria for Measuring the Efficiency of use of Appliances: Survey from Usability Experts and Designers Opinion. In: Advances in Ergonomics In Design, Usability & Special Populations: Part III. AHFE Open Access, vol. 20. https://doi.org/10.54941/ahfe1001317
This paper is part of a larger research project that aims to propose a methodological approach that aims to measure the efficiency of use of appliances in Usability Tests. This paper, in turn, aims to develop a list of criteria that should be considered in evaluation the efficiency of use of appliances in Usability Test. To achieve this goal, a research was conducted in the literature, with researchers in usability, with designers specialize in appliances and usability experts. The tools were considered: interview and five-point Likert scale. As a result, the following criteria have been selected as a recommendation to be considered in research that address the efficient use of appliances in Usability Test: efficiency perceived by the participant, the total time to perform the task, number of steps taken to implement the task, number of errors, cognitive effort, number of attempts and number of questions.
The interaction with home appliances presents specificities which justify a definition of specific criteria for measuring the efficiency in its context of use. Some of these peculiarities, which distinguish the interaction with appliances from the interaction with other objects and interfaces, are presented below.
A) Amplitude of user profiles. The massive presence of this type of products in people daily life, both in homes and in public spaces, such as offices, combined with habits of life increasingly dependent on them, expose the appliance to the interaction with a very large variety of user profiles.
This variety of user profiles takes place mainly in relation to: gender, age, educational level, socio-economic class. Regarding gender, a significant change has been observed in Brazilian society with regards to the increasing involvement of males in the household chores. This fact, which is well established in other sectors long ago, in Brazil has become remarkable in this century. The interaction of children with appliances, which until a few decades ago was something to be avoided
completely, currently, in some cases is encouraged. Naturally, this does not occur with all types of products, but is quite evident in the products that offer little physical risk, such as microwaves or refrigerators. Moreover, with the aging of the global population and the increase of quality of life and independence of elders, this group of users has also been interacting with appliances with not negligible frequency and intensity.
In the domestic environment is very common that the appliance is used in different times by people with very different education and socioeconomic levels. A common example of this situation is the case of products that during the week are operated by a maid with a low educational level and on weekends by the housewife with a very different socio-economic level.
B) The interaction time spend with the control interfaces. When compared to other products which are related to work (whether software or machines) or leisure (electronic games, for example), the duration of use of the appliance interfaces is rather brief. The time spent in programing a microwave oven, for example, is relatively very short when compared to the cooking time itself, or to an ordinary activity on a given software. This fact makes the time variable, one of the main parameters for measuring the efficiency reported in the literature, has to be considered carefully when we evaluate the usability of home appliances.
C) The existing relationship between digital interface (control) and physical function / product interface. Most of the data in a control panel of an appliance match a physical user action in another part of the product. For example, to use a washing machine is not enough to select the program and start the washer, it should obviously put the clothes in the washing drum and add the consumables used in the process. This fundamental relationship between digital and physical interfaces in appliances, sometimes not as obvious as in the example above, strongly influences the efficiency of the system.
In this sense, the present article aims to develop a list of criteria that should be considered in assessing the efficiency of use of an appliance in Usability Testing. The starting point was the following research question: what criteria to consider in the efficiency of use of an appliance in Usability Tests? This is one step of a larger research that aims to propose a methodological procedure for measuring the efficiency of use of appliances.
FUNDAMENTALS Along with effectiveness and satisfaction, efficiency is considered one of the three dimensions / measures of usability. There are some considerations in the literature that address the efficiency. However, these approaches do not consider efficiency in depth for the use of appliances. What follows are some definitions of efficiency that will prove important for the next stage of research.
The efficient in using a certain system is defined as "resources expended in relation to the accuracy and completeness with which users achieve goals" (ISO9241-11, 2010). The standard cites as relevant resources mental or physical effort, time and material or financial costs. Therefore, the efficiency would be the result of the ratio between effectiveness and these resources.
This definition, although quite clear, allows a range of interpretations and very wide applications, especially regarding the classification and way to measure the "resources expended ". If on one hand this fact is appropriate for allowing the application of the concept to different contexts, on the other, it imposes the need for further specification of its application in specific contexts.
Because of this vast range of applications, different approaches and ways of measuring efficiency can be found on the literature. Some of these interpretations are reported below. In 1997, the MUSiC1 project presented three metrics which can be related to efficiency. They are: time efficiency, relative efficiency and productive period. The authors consider the time efficiency as the ratio between the level of
1 MUSiC – Measuring Usability of Systems in Context (Macleod at al, 1 997).
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
efficiency and the total time that was spent to perform a given task. The relative efficiency is pointed out by the authors as a ratio between the efficiency of a common user and efficiency of an expert user 2. A productive period, according to the authors, refers to the time when unproductive actions were not taken3. According to them, the time of such actions can be subtracted from the total time for the task, then, be regarded as productive period.
In contrast with Macleod at al (1997), who related efficiency with time and mathematical formulas, Preece et al. (2005) understand efficiency as a goal that fits the set of usability goals. And in turn, presents it as "the way the system assists users in performing their tasks." Thus, efficiency is considered here from a more subjective criteria, which leads us to question the kind of help the system would provide: feedback, status, objective labels, interaction with good guidance, etc. Another question that is pertinent to this type of approach is how to translate this subjective metrics into an objective parameters..
With regards to the efficiency for the human-computer interfaces, Hornbaek (2006) adopts a more specific approach, indicating the metrics used. They are: total time on task or partly by sub-task, reaction time, dead time, rate of data input (eg, number of characters), mental effort (physiological measures or questionnaire), amount of clicks and actions, number of queries to the aid; deviation from the path or optimal solution, information accessed and used, and resources to be able to communicate. On the other hand, it is considered that as important as identifying metric for measuring the effectiveness, is to clarify the way to quantify them separately, while metrics, and together, forming a single dimension representing use efficiency.
Hekkert and Desmet (2007) were more general, if we compare to Hornbaek (2006). Based on ISO 9241-11, these authors relate efficiency, especially to time. They state that "efficiency is the amount of time it takes to meet the goal." Making a comparison between these authors, one can assimilate time as an important metric. However, it is important to point out additional data and metrics to time to obtain a more detailed evaluation, such as the number of trials suggested by Partala (2009).
However, Partala (2009) proved confusing when considering that efficiency should be measured by indicators such as task completion and number of attempts (learning times). Here, we understand that the number of attempts is directly related to the efficiency. However, completion of the task proves more associated with effectiveness. A justification for this is the fact that this completeness / accuracy be considered by the ISO9241-11 2010 in the definition of effectiveness of use.
Unlike the authors presented in this text, Kuniavsky (2010) proposes that the efficiency is also seen from the user perception. For the author, the efficiency is "how fast / cheap the result is." Therefore, it is considered the perception of speed (time) and the perception of material / financial resources spent. Since they treat efficiency from users’ perspective, it is understood that this data would be something complementary to other criteria presented, which allows the efficiency to be viewed not only from the look of the specialist. This approach adds more data to the analisys and discussion, enriching the research.
In the case of new elements to be considered in usability research, we have what has been proposed by Seva et. al. (2010), from the theoretical basis of Kashimura & Kurosu (1995). These authors worked with the concept of apparent usability. In their research, they identified that it depends directly upon two approaches to efficiency: cognitive efficiency and operational efficiency. The first is identified as being the types of strategies that will help one to understand the interface more easily. The latter, in turn, is considered as the strategies that prevent users from making mistakes. However, these approaches are still quite subjective since they do not clearly point to an objective measurement.
After identifying some authors on the efficiency, in the scope of usability, it was important to create a summary of the criteria considered by them. So the information was systematized and presented in the table below:
2 In this paper, we will consider a person who is very experienced in using a certain product. 3 Time of help, Time of search and Time of Snag; according to (Macleod at al, 1997).
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
For this article, we will consider as efficiency of use "the resources expended to perform a task."
METHOD OF RESEARCH The research described in this paper was divided into four stages, as shown in Figure 1:
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
Figure 1. Outline of the research method.
Step one corresponds to the criteria gathering which influence the measurement of the efficiency of use of appliances. This survey was made from three separate sources: literature, researchers in the field of usability and expert designers in the development of household appliances. The criteria arising from the literature were identified from a literature review of scientific articles and books. The other criteria were identified with the help of online questionnaires. In this research, three questions were asked: A. What do you understand by efficiency of use relative to an appliance? Give an example. B. Everyday people use appliances to assist them in performing household chores. Sometimes, during the first
use, they can experience some difficulty to use them in a proper way. Whereas a person could accomplish his / her goal, what factors should be considered in evaluating the efficiency of use of an appliance? C. Which adjectives could be related to the efficient use of an appliance? (Name at least five) Question A aimed at identifying the perception and understanding that the subject had on efficiency of use. So, were considered for this study, responses and / or examples that were within what we understand by use efficiency.
Questionnaires that did not have neither an answer nor an example, similar to the concept of efficiency applied in this study, were not considered. Question B, in turn, aims to present an overview of criteria that can be considered to measure efficiency. Finally, question C aimed to identify a group of adjectives that can be
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
applied in future researches for the development of a semantic panel in the field of User Experience.
The questionnaires were sent by email and participants were asked to respond in about 56 hours of deadline for submission of responses. The questionnaires were sent to 17 usability researchers in 12 home appliances designers. Eleven out of the17 usability researchers and all 12 appliances designers answered the questionnaires. However, one of the answers was disregarded due to the inconsistency.
Step two corresponded to the tabulation of the questionnaires. The answers to question B were separated and grouped with the use of an affinity diagram. Literature of the field, researchers in usability experts and designers specialized in appliances: the three sources were considered. As a result, it was obtained a set of criteria that influence the efficiency of using appliances.
Next step, number three, was convertion of the set of criteria in a list containing a Likert scale of five points. Here, the criteria was presented and relate to a scale ranging from very irrelevant to the very important. This list was sent by email, initially to researchers in usability, as a pilot, and then to the experts in usability4.
The fourth step presents the results of research criteria to be considered in the efficiency of use.
RESULTS
From the open questionnaire responses in step 1, the criteria presented below were ranked per frequency. Initially the criteria from the questionnaire to the researchers in usability will be presented and then the criteria from the responses of expert designers in home appliances.
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
Figure 03: Tabulation of criteria suggested by researchers and designers specialized in appliances.
Figure available in the original PDF paper.
As it can be observed, the criteria suggested by the home appliances designers have shown a low frequency. That happened because some responses were referring to the efficiency of the system performance, and not to efficiency of use. Thus, these responses were disregarded.
After listing the criteria proposed by these two groups, a parallel check in the literature was made in order to group by affinity these criteria. The affinity here was considered as aspects belonging to the same group and therefore they could be classified into a single criterion. Therefore, a table was developed containing three sources of criteria, and a fourth column with their respective categories, shown in figure 04.
After categorizing the criteria, a Likert scale of five points related to criteria was developed. This scale was tested as a pilot by six researchers. From the results, we chose not to name only the extremes (very irrelevant - very important), but to identify all columns. This decision was taken because the answers were considered only if they were marked in the columns of largely irrelevant or very relevant. The other columns that were not named were not recognized by the system Google Drive. Then, a scale with the description of all columns, as shown in Figure 0 5, was developed.
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
Figure 04: Relationship between the selected criteria and the initial sources of survey criteria.
Figure available in the original PDF paper.
This scale was sent by email for tens usability experts, from which nine answered. The strategy for tabulation of results followed Shackel and Richardson (2008) recommendation in which the criteria should have at least 80% of responses in the two rightmost columns (relevant and highly relevant) and no response in the two leftmost columns (very irrelevant and immaterial), as Figure 06.
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
Figure 05: Likert scale of five points for evaluation criteria.Figure 06: The Likert scale tabulation strategy.
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
The result of the selection process proposed by Shackel and Richardson (2008) is presented in Table 01.
Table 01: Listing the status of the criteria evaluation.
Thus, from the assessment, were considered the following criteria: efficiency perceived by the participant, the total time to complete the task, number of actions / steps taken for the task accomplishment, number of errors, cognitive effort, number of attempts and the number of doubts.
Comparing the criteria selected with three initial sources explored at the beginning of this research, it was identified that these criteria were the most repeated among the three sources. The highlighted fields represent the highest density of repetitions involving the three sources of research front of categorized criteria, now selected.
DISCUSSION
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
The efficiency identified in the literature was perceived in two different ways: too vague for the scope of this research, or too specific for GUI systems. In this sense, these developments as input data were used so that we could apply them to the evaluation context appliance. In the present research, the definition of the evaluation context was done with the help of the questionnaires, which always pointed its scope: assessment of the efficiency of use of appliances.
A relevant point to be discussed, initially, is the perception of efficiency by designers specialized in appliances. From their responses, it was identified that the majority related the efficient of use with the system efficiency. One possible reason for this misunderstanding may be detachment between the user and the development context of the appliances.
Regarding the usability researchers, it was found that some directed their responses to human-computer interaction. This could be identified from the examples mentioned, as well as the type of indicated criteria. These criteria were more contextualized within the scope of HCI. Furthermore, it was found that the sample of researchers in usability proved to be too broad and with extreme profiles. For example, in one group there were people working in the area for about a year without postgraduate course in the area, as well as people with postgraduate course in the area and active in the area for about fifteen years.
However, the composition of the group with different profiles within the usability is important for the id entification of the criteria. Nevertheless, it was felt that for the phase of criteria selection, it would be more consistent to restrict the sample to experts, with the minimal educational level being a master degree in the area and at least three years experience. This restriction resulted in reducing sample to nine participants. However, since the latter is a more homogeneous sample, this reduction does not appear as an influential factor for non- validation of the data collected.
The tool used in step 3 of this research proved to be objective and easily applied. However, it is not rich, considering the need for accuracy. As an example, comparing a five-point scale, where the research subject has only this number of points to choose from, and a 10 centimeters line segment, where the researched can opt for a much greater number of options to mark. Perhaps the five-point scale may prove more suitable to be applied to respondents with low repertoire in research methods (participants in a Usability Testing). Usability Specialist, in turn, could already be applied a broader tool, as the line segment itself.
Related to selected criteria, it was identified that the criterion "number of attempts" and "number of errors" are shown close, however, different. In this sense, it is understood that the number of trials proves more applicable to efficiency, since they are strategies, cycles to perform a task that does not have the intention to reflect completeness, but fatigue. The number of errors is understood as the number of complete cycles to perform a task. These cycles were completed with a failure of the task. In this case, as the cycles are directly related to the completeness / accuracy of the task, it is understood that this criterion is more related to the effectiveness that, properly with efficiency.
CONCLUSIONS
This article aimed to develop a list of criteria which should be considered in evaluating the efficiency of use of an appliance, in Usability Tests. To achieve this goal, a research in the literature, with researchers in usability, designers specialized in appliances and usability experts was conducted. Interview and Likert five- point scale were applied.
As the input of the research, there is the following list of criteria which is recommended to be considered in researches that address the efficient use of appliances, in Usability Tests. The criteria are: efficiency perceived by the participant, total time to perform a task, number of actions/steps to perform a task, number of errors, cognitive effort, number of attempts and number of doubts.
For future researches, it is suggested that the selection of criteria related to efficiency is taken from other tools, to compare with the criteria selected here. Another proposed way forward for continuity this research is to present, in an objective manner, a methodological procedure to measure the criteria selected in this research, in Usability Tests. Finally, it is suggested a deployment of criteria to be considered as measurement of the effectiveness and satisfaction of use, to be applied in appliances Usability Tests.
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
REFERENCES
Desmet, P. A. M.; Hekkert, P. (2007), Framework of product experience. In: International Journal of Design, 1, 57-66. Frokjaer, Hertzum, Hornbaek. Measuring usability: are effectiveness, efficiency and satisfaction really correlated? In: S IGCHI conference on Human Factors in Computing Systems, 345-352, 2000. Hornabaek, Law. (2007), Meta-ananlysis of correlations among usability measures. In: C o n f e r e n ce o n H u m a n F ac t or s i n C o m p u t in g S y s t e m s , Nova York, 617-626. Hornabaek, K. (2006), Current practice in measuring usability: challenges to usability studies and research. Int. J. Human -
Computer Studies, 64, p. 79-102. Kuniavsky, M. K. (2010), Smart Things: Ubiquitous Computing User Experience Design. Burlington, MA, USA. ISO 9241-11 – Ergonomic Requirements for office work with visual display terminals (VDT)s – Part 11: Guidance on usability,
2010. Kurusu, Kashimura. (1995), Apparent usability vs. inherent usability: experimental analysis on the determinants of the apparent usability. In: Conference Companion on Human Factors in Computing Systems. Nova York, 292-293. Macleod, Bowden, Bevan, Curson. (1997), The MUSiC performance measurement method. Behaviour & Information
Technology, vol. 16, nº 4/5,279-293. Preece, J.; Rogers, Y.; Sharp, H. (2005) Interaction design: beyond human-computer interaction. New York: John Wiley & Sons. Shackel, B.; Richardson, S. (2008) “Human Factors for Informatica Usability”, Cambridge University Press, New York.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Advances in Ergonomics In Design, Usability & Special Populations: Part III
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-4951-2108-1
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional