Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Design of a human factors questionnaire to evaluate digital solutions developed for industrial work

VTT Technical Research Centre of Finland Ltd, Visiokatu 4, 33101 Tampere, Finland ↗
Publication year
Digital Object Identifier10.54941/ahfe1001992

Abstract

Industrial work is changing due to the fourth industrial revolution (Kagermann, 2013). The work will become more mediated when new technologies emerge in factories (e.g., robotics, artificial intelligence and augmented reality). During this change, it is important to consider human factors (HF) issues holistically when developing and implementing novel digital solutions. Often, HF methods such as questionnaires are focusing on a certain topic related to the development of the solution (e.g., usability, acceptance). However, in industrial work context, there are many different HF issues that should be considered when designing a human-technology interaction. To address this lack of a quick and holistic questionnaire, a WorkerFeedback questionnaire was developed through a Design and Evaluation Framework (Kaasinen et al., 2018), literature review and expert validation. The questionnaire development was based on the Design and Evaluation Framework, which supports design and evaluation of digital solutions that will be used by future factory workers (Kaasinen et al., 2018). The framework is useful for providing an overview to important user experience related issues and it is a holistic approach to design and evaluation tasks. The literature review was based on the HF topics that are related to the success of implementing new technology on to the factory floor. Several questionnaires were reviewed related to following topics: user experience, usability, user acceptance, usefulness, ergonomics, safety and ethics. Questionnaires were used as a basis to generate a 14-item questionnaire – two questions for each topic. The questionnaire was validated by an expert panel, which included three human factors experts with over 15 years of experience of user studies, mainly in industrial contexts. They were asked to review, for example, the adequacy of the topics and the formulation of the questions.The expert panel agreed that their overall impression of the questionnaire was good. The suggested seven topics were seen to be relevant for the context. Majority of the feedback was related to the formulation of the questions and making them easier to understand. Based on the experts’ feedback, the questions were shaped to their final form. In future, the WorkerFeedback questionnaire will be piloted with the actual digital solution users. The WorkerFeedback questionnaire was developed to smoothen the digital transition in industry and to support human-technology interaction design. The questionnaire can be used by HF experts as well as practitioners who want to get a holistic overview of developed digital solutions or solutions under development for work context.

INTRODUCTION

Industrial work is changing due to the fourth industrial revolution (Kagermann et al., 2013; Kadir and Broberg, 2020). Workers are starting to use digital solutions that support them in their work (e.g., robotics, artificial intelligence and augmented reality) (Romero et al., 2016). When implemen- ting these technologies on a factory floor, it is important to consider human factors (HF) related issues comprehensively. There are frameworks that pro- pose systematic considerations of HF when designing and implementing Industry 4.0 technologies (Neumann et al., 2021; Kadir and Broberg, 2021). These frameworks form a good foundation for taking HF into account, but to support the easy collection of user feedback, they could be complemen- ted with quick to use questionnaires. Human factors related questionnaires often focus on a specific topic related to the development of the solution (e.g., usability, acceptance) (Stanton, 2017) rather than giving a holistic overview of HF issues. For digital solution developers it could be useful to have one simple method that could provide an overall view of HF and then further complement the findings with more detailed and specific HF questionnaires if needed.

This paper describes the development of a comprehensive WorkerFeedback questionnaire suitable for use in the design and evaluation of digital solutions for industrial work. Section two describes the design and review process of the

© 2022. Published by AHFE Open Access. All rights reserved. 635

636 Aromaa and Heikkilä

WorkerFeedback questionnaire. Feedback from the expert review is described in section three with the final questionnaire items. Then, the findings and considerations during the development are discussed, and conclusions are drawn.

METHODS

Design Process The questionnaire was designed iteratively. The first version of the questi- onnaire was developed based on the Design and Evaluation Framework for Operator 4.0 solutions in (Kaasinen et al., 2019). It included two questions for each HF topic (user experience, user acceptance, usability, safety, and ethics). The questionnaire was used in two studies: an industry 4.0 conce- pts evaluation (Aromaa et al., 2018) and a well-being solution development study for factory work (Heikkilä et al., 2021). These studies showed that the questionnaire supported the collection of the overall user experience of the solutions, but it could be complemented with two extra HF topics: ergono- mics and usefulness. Based on these findings, the whole design and evaluation framework was modified further to include seven HF topics: user experie- nce, usability, user acceptance, usefulness, ergonomics, safety, and ethics (see Figure 1).

Based on the extension to the framework, two HF experts reviewed que- stionnaires and methods related to usefulness and ergonomics to find and formulate two questions for each topic. Meanwhile, the original questions were iteratively developed further. When designing the questionnaire, it was agreed to use positive wording in the questions to avoid possible mistakes made by respondents. Sauro & Lewis (2011) agree that mistakes might occur when using both wording types (positive and negative).

Figure 1: Design and evaluation framework for operator 4.0 solutions. Modified from Kaasinen et al. (2019).
Original PDF Figure 1:

Design of a Human Factors Questionnaire to Evaluate Digital Solutions 637

Expert Review The WorkerFeedback questionnaire was reviewed by an expert panel, which included three human factors experts with over 15 years of experience of user studies, mainly in industrial contexts. They were asked to review, for example, the adequacy of the HF topics and the formulation of the questions. The questionnaire was sent to the experts to be reviewed and after individual reviews, two 1-hour discussion sessions were held to discuss the questionnaire and its questions. Comments were written down and analyzed later by two researchers.

RESULTS

Feedback from the Expert Review The expert panel agreed that overall the questionnaire covered relevant HF aspects and included a comprehensive set of questions. The 5-point Likert scale was seen as appropriate as it is a standard approach. Most of the review comments were related to how to clearly formulate the statements, but overall, no major changes were suggested.

In the first version of the WorkerFeedback questionnaire, the user expe- rience topic was addressed with a visual smiley faces question and with one question statement. For the expert review, an alternative question was sugge- sted to possibly replace the smiley face question. Based on the expert review, it was assumed that the format of the smiley face question could lead to chal- lenges later in the analysis of the results because its scale was not coherent with the other questions. Thus, two statements related to the user experience were preferred over smiley faces.

The second question on the usability theme raised some discussion about whether respondents would understand the meaning of “smooth work pra- ctices”. The expression can be related to a specific work task or to the level of the whole factory. In the end, the question was seen as relevant since it is important to evaluate the solution from the systematic perspective — not just as its own entity but its integration into the work practices.

Discussion related to user acceptance was mainly related to the issue of whether the topic was overlapping with the safety and ethics questions, in particular, whether the trust question should rather belong to the safety or ethics sections. However, the discussion concluded that trust is a valid topic within user acceptance, as it is a crucial factor in accepting a technical solu- tion and is included in commonly known acceptance models (Davis, 1989; Venkatesh et al., 2003).

Regarding the usefulness, there were concerns whether the first question was too nonspecific. It was agreed that it is not always known beforehand what the benefit for the user is and therefore a more general question is justi- fied. In addition, it was suggested that all statements should be expressed in a similar way to form a coherent questionnaire (e.g., both “work” and “job” were used in the questionnaire).

Even though all the questions were formatted such that the “strongly agree”selection was the most positive answer to the questions, some negative

638 Aromaa and Heikkilä

wording was used in the ergonomics statements (e.g., “I did not feel a physical load while using the solution). According to the expert review, these negati- vely phrased statements were complex and difficult to understand. Therefore, these questions were formulated in a positive way.

Both safety questions were also negatively phrased and it was suggested that they should be formulated differently. In addition, there was some discus- sion concerning whether the users would be able to evaluate safety issues by themselves or whether safety should be evaluated by safety experts. Howe- ver, the expert panel agreed that by formulating the questions in the correct way, questions on safety issues could be asked directly from the users. It was also discussed if the second safety related question was too narrow and focu- sed only on one area of safety. In the end, it was agreed that attention in a factory environment is an important aspect of safety and thus, the question was supported.

It was agreed that the ethics questions raised relevant ethical issues. How- ever, there was again the issue related to the negation format in one question (“the use of an application should not feel questionable”). To bring the que- stion into a consistent format with the other questions, an opposite wording for questionable was not easy to find. After considering several options, “eth- ically acceptable” was found to be the most appropriate and comprehensive solution.

WorkerFeedback Questionnaire to Evaluate Digital Solutions in Industrial Work Based on the expert review the final version of the WorkerFeedback que- stionnaire to evaluate digital solutions in industrial work was formulated (Table 1). It includes 14 statements in seven HF categories (user experience, usability, user acceptance, usefulness, ergonomics, safety, and ethics). All sta- tements are evaluated on a 5-point Likert scale (strongly disagree – disagree – neither agree nor disagree – agree – strongly agree). In addition, relevant background questions are suggested to be used.

DISCUSSION

The development of the WorkerFeedback questionnaire was based on the design and evaluation framework (Kaasinen et al., 2019) and existing HF questionnaires. The process was iterative and resulted in 14 question items which were stated the same way to minimize mistakes when filling in the questionnaire.

When developing a questionnaire including several viewpoints, it was noted that some of the HF topics are often evaluated by experts rather than solution users themselves (e.g., safety reviews and ergonomic posture analy- sis). Therefore, it was not always easy to formulate the questions in a way that the users could evaluate them easily by themselves.

To find the right wording and to avoid complexity in the questions was also a challenge for some topics. Safety and ergonomics questions are typi- cally described in a negative format, for example, “does not pose safety risks” or “does not increase the physical load”. However, this negative approach

Design of a Human Factors Questionnaire to Evaluate Digital Solutions 639

Table 1. WorkerFeedback questionnaire statements to evaluate digital solutions in industrial work.

Table available in the original PDF paper.

Questionnaire Statements My overall feeling of using the solution is positive I like working with the solution The solution is easy to use The solution supports smooth work practices I would like to use the solution in the future I can trust the solution I find the solution useful in my work Using the solution would improve my work performance The solution is physically pleasant to use (e.g., no discomfort in the hands, neck or eyes) The solution is mentally pleasant to use (e.g., it does not cause stress) I find the solution safe to use I can maintain the needed awareness of the environment while using the solution Using the solution at work is ethically acceptable (e.g., it does not feel questio- nable) The usage of the solution respects my privacy

was not suitable for this questionnaire because it made it difficult to under- stand which value to select on the Likert-scale. Therefore, all questions were formulated consistently in a positive format. In some questions, the negati- vely formulated statements were left in brackets after the new statements to clarify the meaning of the statement to the respondents.

This questionnaire was developed to be used as a quick and easy overview of HF issues related to digital solutions developed for industry work. Howe- ver, the purpose is not to substitute more detailed and specific questionnaires. The WorkerFeedback questionnaire could be used as a first step to under- stand possible HF issues with digital solutions, and if it reveals concerns with a certain topic, the topic could be evaluated in more detail by using more specific HF methods (e.g., Stanton (2017)).

The development of the questionnaire and expert evaluation were the first steps in the questionnaire design. Next, the questionnaire will be tested in real-life industrial use cases to be validated further and to test the usefulness of the overall feedback. In addition, it would be interesting to study whether two questions for each topic are sufficient to give the first impression of the use of the solution.

CONCLUSION

The purpose of this study was to design an HF questionnaire to get feedback regarding new digital solutions developed for industrial work context. The WorkerFeedback questionnaire was based on the Design and Evaluation Fra- mework for Operator 4.0 Solutions (Kaasinen et al., 2019) and was reviewed by an HF expert panel.

Based on the expert review, it was agreed that the developed questionnaire was comprehensive and provides a good overview of HF issues. In some

640 Aromaa and Heikkilä

cases, the design and formulation of the questionnaire items needed some extra consideration due to nature of certain HF topics that are usually eva- luated by experts rather than users themselves (e.g., safety, ergonomics). The questionnaire addresses seven HF topics: user experience, usability, user acce- ptance, usefulness, ergonomics, safety, and ethics. For each topic, there are two statements which are evaluated on a 5-point Likert scale.

The questionnaire provides a first impression of digital solutions under development and guides the design towards those HF topics that need more consideration. The WorkerFeedback questionnaire can be used by HF experts in research and industry, but it is easy to use also by other people such as digital solution developers.

ACKNOWLEDGMENT

This project has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 873087 (pro- ject SHOP4CF). This paper reflects only the authors’ view and the Commis- sion is not responsible for any use that may be made of the information it contains. The authors are grateful to all researchers, project partners and HF experts who have contributed to and supported the work presented in this publication.

REFERENCES

Aromaa, S., Liinasuo, M., Kaasinen, E., Bojko, M., Schmalfu, F., Apostolakis, K. C. Z., Dimitrios, Daras, P., Özturk, C., Boubekeuer, M. (2018) “User evaluation of Industry 4.0 concepts for worker engagement”, in International Conference on Human Systems Engineering and Design: Future Trends and Applications, Cham, Springer. Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of

information technology, Information Technology MIS QUARTERLY Volume 13, No. 3. Heikkilä, P., Honka, A., Kaasinen, E., Väänänen, K. (2021). Quantified factory worker: field study of a web application supporting work well-being and productivity, COGNITION TECHNOLOGY & WORK Volume 23, No. 4. Kaasinen, E., Liinasuo, M., Schmalfuß, F., Koskinen, H., Aromaa, S., Heikkilä, P.,

Honka, A., Mach, S., Malm, T. (2019) “A Worker-Centric Design and Evalu- ation Framework for Operator 4.0 Solutions that Support Work Well-Being”, in: Human Work Interaction Design, Designing Engaging Automation, Cham, Springer International Publishing. pp. 263–282. Kadir, B. A., Broberg, O. (2020). Human well-being and system performance in the transition to industry 4.0, INTERNATIONAL JOURNAL OF INDUSTRIAL ERGONOMICS Volume 76. Kadir, B. A., Broberg, O. (2021). Human-centered design of work systems in the

transition to industry 4.0, APPLIED ERGONOMICS Volume 92. Kagermann, H., Wahlster, W., Helbig, J. (2013). Recommendations for Implementing the Strategic Initiative INDUSTRIE 4.0: securing the future of German manufacturing industry. Neumann, W. P., Winkelhaus, S., Grosse, E. H., Glock, C. H. (2021). Industry

4.0 and the human factor – A systems framework and analysis methodology for successful development, INTERNATIONAL JOURNAL OF PRODUCTION ECONOMICS Volume 233.

Design of a Human Factors Questionnaire to Evaluate Digital Solutions 641

Romero, D., Stahre, J., Wuest, T., Noran, O., Bernus, P., Fast-Berglund, Å., Gorecky, D. (2016) “Towards an operator 4.0 typology: A human-centric per- spective on the fourth industrial revolution technologies”, proceedings of the 46th International Conferences on Computers and Industrial Engineering, Tianjin, China. Sauro, J., Lewis, J. R. (2011) “When Designing Usability Questionnaires, Does It

Hurt to Be Positive?”, SIGCHI Conference on Human Factors in Computing Systems, Vancouver, BC, Canada. Stanton, N. A. (2017). Human Factors Methods: A Practical Guide for Engineering and Design, CRC Press. Venkatesh, V., Morris, M. G., Davis, G. B., Davis, F. D. (2003). User Acceptance of

Information Technology: Toward a Unified View, MIS QUARTERLY Volume 27, No. 3.

Cover of the proceedings volume Ergonomics In Design

Explore the complete digital volume

Ergonomics In Design

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-23-0

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →