Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

A Collaborative KPI Framework for Evaluating a Digital Twin Demo Platform: Supporting Circular Economy Transformation in SMEs

Häme University of Applied Sciences, 13100 Hämeenlinna, Finland ↗
Publication year
Digital Object Identifier10.54941/ahfe1007585

Abstract

This study addresses the challenges faced by small and medium-sized enterprises (SMEs) when adopting circular economy (CE) practices, particularly when the SMEs are characterized by limited resources, lack of expertise, and constrained access to tailored digital tools. Within the Interreg Baltic Sea Region (BSR) CIRC-2-ZERO initiative, a digital twin demo platform (DTDP) is being developed to support SMEs in terms of data-driven circular product and value chain design. Given that the effectiveness of such platforms depends on systematic evaluation, this study proposes a collaborative key performance indicator (KPI) framework to enable systematic assessment and monitoring of the DTDP. The study applies a participatory and iterative methodology, employing the nominal group technique and multi-voting across three expert workshops involving twenty participants from academia, business support organizations, and the electronics and engineered wood products sectors. This stakeholder-driven approach ensures that the KPI framework reflects both operational realities and user needs. The study is structured into three phases—namely, KPI identification, refinement and validation, and development of data collection procedures. The resulting framework organizes the KPIs into three core dimensions—technical performance, usability, and value impact—while distinguishing between indicators relevant to the DTDP’s piloting phase and those important for its long-term development. The framework also includes structured data collection procedures intended to support consistent and reliable monitoring. This study contributes to the growing body of research on performance assessment of digital platforms supporting CE transitions, offering a practical, stakeholder-driven tool for assessing DTDPs and supporting SMEs’ circular transformation. Moreover, the study promotes sustainable industrial practices and supports regional development in the BSR.

INTRODUCTION

Small and medium-sized enterprises (SMEs) are fundamental contributors to industrial sustainability, innovation capacity, and regional development. However, the adoption of circular economy (CE) practices remains a complex challenge for many SMEs due to their limited expertise, scarce Received April 29, 2026; Revised May 31, 2026; Accepted June 16, 2026; Available online July 20, 2026

financial and human resources, and lack of appropriate data-driven tools to support decision-making (Zhu et al., 2025). Moreover, global environmental pressures, coupled with new regulations regarding sustainability and the CE, increase the need for digital solutions that can assist SMEs in this transition (Swarnakar et al., 2025). To address these gaps, the Interreg Baltic Sea Region (BSR) CIRC-2-ZERO initiative aims to develop and pilot a digital twin demo platform (DTDP) that enables simulation of circular product and value chain design.

A central outcome of the DTDP project will be the development of a digital environment that strengthens the capabilities of SMEs in the electronics and engineered wood products (EWPs) manufacturing sectors, supporting them in redesigning products to ensure their durability, repairability, and recyclability using simulation tools while also allowing them to map, analyze, and optimize their supply chains for circularity. Digital twins (DTs) have significant potential when it comes to industrial sustainability. However, their real impact is dependent on SMEs having the ability to systematically measure their performance, usability, and value contribution (Soori, Arezoo and Dastres, 2023). For this reason, developing a structured key performance indicator (KPI) framework is essential for assessing the DTDP during the pilot phase.

The present study aims to fill the identified research gap by outlining the conceptual and methodological foundations of a collaborative KPI framework designed to enable a systematic evaluation specifically tailored to the DTDP. To accomplish this, it presents the current stage of the project, which involves three workshops focused on defining, refining, and validating the KPIs, as well as on developing data collection procedures to facilitate monitoring of the KPIs. This paper details the design of the evaluation process, justifies the selection of the participatory methods, and highlights the key dimensions guiding the KPI identification and data collection to ensure an effective DTDP assessment.

From a methodological perspective, this study employs the nominal group technique (NGT) and multi-voting (Van de Ven and Delbecq, 1972), thereby adopting a participatory and iterative approach based on structured collaborative techniques to ensure consensus among participants. By outlining this methodology, the present paper contributes to the literature on DT evaluation, stakeholder-driven KPI development, and CE-oriented digital transformation. Moreover, it provides a transparent foundation for future workshops and subsequent integration of results into the DTDP’s monitoring and evaluation strategy.

The remainder of this paper is structured as follows. The next section describes the theoretical framework of the CE, DTs, and KPIs for the DTDP assessment. The third section details the methodology used, including the design of the workshops and the KPI definition process. Next, the fourth section outlines the expected results of the participatory analysis. Finally, the fifth section presents the main conclusions of the study.

BACKGROUND AND THEORETICAL FOUNDATIONS Digital Twins and Circular Economy Transformation

DTs are virtual representations of products, processes, or systems, and they have emerged as powerful enablers of digital transformation (Issa, Hamad and Abdel-Geliel, 2023; Tyagi and Tyagi, 2025). According to Souza et al. (2026), DTs facilitate the collection, consolidation, and provision of relevant information at critical moments in both the lifecycle and the value chain. In the context of the CE, DTs represent a powerful pathway to sustainable value creation due to offering detailed lifecycle data and facilitating the monitoring and implementation of CE strategies throughout the entire product lifecycle (Mügge et al., 2025).

For SMEs, DTs provide opportunities to transition to CE practices while maintaining both competitiveness and cost efficiency by optimizing resource flows, reducing waste, evaluating circular design options, and supporting data-driven strategies. Yet the adoption of such tools in SME contexts remains limited (de Larmelina, da Silva and Risso, 2025), largely because SMEs typically require solutions that are lightweight, user-friendly, and adaptable to varying degrees of technological maturity.

The DTDP addresses these needs on the part of SMEs by focusing on accessibility, simplified data integration, and scenario-based modeling tailored to SMEs’ CE decision-making. Nonetheless, evaluation of whether the DTDP meets user needs requires a robust, context-sensitive set of KPIs. This is very important, given that few prior studies have focused on the evaluation of such platforms from the user experience perspective, especially for users who lack technical expertise but need to interact with the system at an operational level (Barricelli and Fogli, 2024).

KPIs in Digital Twin Assessment KPI development is essential because such indicators play several crucial roles in technology piloting processes. Indeed, KPIs enable the evaluation of simulation efficiency by assessing the accuracy of the predictions generated by DT platforms in comparison to real-world data. In addition, they are vital when it comes to monitoring system performance in real time, ensuring that simulations are quickly adjusted to optimize both physical and virtual processes (Pehlken et al., 2024).

The existing literature (e.g., Bluvstein et al., 2025; Doukari, Bacnak and Mahamedi, 2026; Santos et al., 2024; Soori, Arezoo and Dastres, 2023) highlights perceived usefulness, ease of use, and overall acceptance as critical aspects when defining KPIs for evaluating DT platforms, indicating that particular emphasis should be placed on their value and practical relevance. The key aspects commonly considered in such studies include (1) technical performance (e.g., system stability, speed, reliability, interoperability, scalability); (2) usability (e.g., user experience, interaction quality, engagement); and (3) value impact (i.e., benefits generated for users, including decision-making improvements, cost/time savings, and contributions to sustainability strategies).

Most KPI frameworks (e.g., Bluvstein et al., 2025; Doukari et al., 2026) highlight the importance of participatory approaches in ensuring that indicators accurately reflect both stakeholder needs and operational realities. This is especially critical in relation to CE-oriented innovation, where the diversity of sectors and maturity levels can significantly shape the evaluation criteria.

METHODOLOGICAL APPROACH Participants

This study relied on the input of a panel of experts, recruited from partner universities and affiliated organizations involved in the DTDP project, including Business Support Organizations (BSOs). The selection considered their recognized expertise in areas aligned with the dimensions under analysis, ensuring that each dimension was informed by specialists with relevant knowledge and experience. The specialized literature (e.g., Bana e Costa and Silva, 2008; Eden and Ackermann, 2001, 2004) suggests that expert panels typically include a relatively small number of participants with diverse perspectives on the subject at hand. When forming the panel for this study, the following criteria were considered: (1) individuals with a significant amount of work experience (i.e., more than fifteen years of professional experience) in CE practices; (2) a balance in terms of backgrounds (i.e., academia and industry); (3) individuals from SMEs in the electronics and EWPs sectors, consistent with the Interreg BSR project’s focus on those two industries; (4) a gender balance; and (5) individuals’ availability to participate in three workshops.

The workshops were conducted online via Microsoft Teams, supported by dedicated Miro boards and Google Drive integration, and took place in April 2026. The workshops involved twenty participants from higher education and research institutions, business support organizations, and representatives of SMEs and large companies in the electronics and EWPs sectors. This composition ensured complementarity among the scientific, technical, and business perspectives.

Participatory and Iterative Design Given the interdisciplinary nature of the DTDP and the diversity of stakeholders involved in the project, the KPI development followed a participatory and iterative methodology. This methodological approach was underpinned by facilitated, structured activities, employing the NGT and multi-voting (Van de Ven and Delbecq, 1972), as recommended for structured expert consensus-building (Eden and Ackermann, 2001). The NGT is a structured approach that facilitates identification of problems and generation of information, which can subsequently be prioritized via group discussion (Harvey and Holmes, 2012; Potter, Gordon and Hamer, 2004). As a method for building consensus, it seeks to achieve general agreement through balanced participation and mutual respect among participants (McMillan, King and Tully, 2016). Moreover, the NGT requires minimal

A Collaborative KPI Framework for Evaluating a Digital Twin Demo Platform 55 preparation by experts, is guided by the concept of transparency, and provides a cost-effective and time-efficient means of engagement (Potter, Gordon and Hamer, 2004).

According to Van de Ven and Delbecq (1972), the NGT involves four to six steps: (1) Silent generation of ideas: Participants individually write down potential

KPIs for each evaluation dimension on separate sticky notes, focusing on measurable indicators. (2) Sharing ideas: Using the round-robin process, each participant shares one KPI at a time until all the generated ideas are presented. This ensures that all voices are heard. (3) Group discussion: A brief discussion follows to clarify and ensure a

common understanding of each KPI. The facilitator must ensure that all participants have equal opportunity to question or defend a decision while maintaining a balanced pace of discussion. Criticism should be avoided. (4) Elimination of redundancies: The group eliminates any redundant KPIs. (5) Multi-voting: A list is created encompassing all the KPIs considered by the experts. At this stage, each expert is tasked with ranking the KPIs they deem to be the most important. (6) Discussion of the vote: While many authors do not consider this step

(Harvey and Holmes, 2021; McMillan, King and Tully, 2016; Potter, Gordon and Hamer, 2004), Van de Ven and Delbecq (1972) do. They suggest discussing the results, allowing everyone to question the initial voting outcomes.

Upon completion of the NGT steps, the anticipated outcome is a refined set of results that leverage the invaluable tacit knowledge and expertise of the participating experts, thereby enhancing the quality and insightfulness of the results (Potter, Gordon and Hamer, 2004).

PRELIMINARY OUTCOMES At this stage of the project implementation, the value of the research lies in the design of a rigorous and participatory evaluation process and the associated data collection procedures, alongside the introduction of the preliminary KPIs.

Phase 1: KPI Identification The first workshop was dedicated to the exploration phase of the project, where participants identified and generated an initial list of KPIs for monitoring the DTDP, covering three evaluation dimensions—namely, technical performance, usability, and value impact. This session focused on idea generation, leveraging the experts’ contributions to identify relevant KPIs.

To guide this process, the following trigger question was posed: How can the effectiveness and value of the DTDP be measured in terms of its technical performance, usability, and value impact?

All the NGT steps during this workshop were conducted using the Miro platform. Each participant used Miro to interact, share, and refine the KPIs in real time. The following steps outline how Miro was utilized throughout the workshop:

  • Participants individually generated potential KPIs for each of the three

evaluation dimensions (i.e., technical performance, usability, and value impact). Each KPI was written on a separate sticky or virtual note. The Miro board was structured into three main sections for each evaluation dimension, and participants added their KPIs under the appropriate categories. This process was conducted as a silent activity, ensuring that all the participants could contribute without being influenced by others.

  • Each participant took a turn sharing one KPI at a time. Participants were allowed to ask clarifying questions but did not discuss the KPIs in detail at this stage.
  • After all the KPIs had been shared, the group worked together to

identify and eliminate any redundant KPIs. To identify redundant KPIs, participants used a sticker (virtual sticky note) to mark the sticky notes they considered to be similar or duplicated. These stickers were used to mark the KPIs for later discussion, allowing the group to review and decide whether to combine, reformulate, or eliminate the potentially redundant KPIs. This step ensured that only the most relevant KPIs remained.

During the final review process, participants identified the need to divide the proposed KPIs into two distinct categories: (1) those applicable to the pilot phase and (2) those relevant to the DTDP’s longer-term development. Accordingly, the final stage of the process was devoted to classifying the KPIs into those two groups.

Table 1 presents the preliminary grouping of the KPIs across the three key dimensions, namely: (1) Technical Performance – 47 KPIs (2) Usability – 55; and (3) Value Impact – 34.

Table 1: KPI grouping by dimension: Preliminary overview.
Original PDF Table 1:

A Collaborative KPI Framework for Evaluating a Digital Twin Demo Platform 57 Phase 2: KPI Refinement and Validation

The second workshop focused on refining the list of KPIs identified during Workshop 1. The objective was to prioritize the KPIs and ensure that they were clearly defined, measurable, and actionable for implementation in the DTDP monitoring system.

Participants engaged in a multi-voting process to prioritize the KPIs based on their relevance, importance, and feasibility. From each dimension, the participants selected the three most important for both the piloting phase and the longer term development phase separately. These KPIs are outlined in Table 2.

Table 2: KPI grouping by dimension and phase: Preliminary overview.
Original PDF Table 2:

Next, participants refined the KPIs that were prioritized during the voting process. This involved clarifying the definitions, ensuring that each KPI was measurable, and structuring the KPIs to ensure they were actionable. Finally, in a group plenary, the KPIs were collectively reviewed. Any final adjustments or clarifications were made, and the KPIs were consolidated for the next phase.

Phase 3: Data Collection Procedures During the third workshop, participants developed data collection procedures for the KPIs, ensuring that each procedure clearly specified the data source(s), the collection method(s), the frequency of data collection, the measurement process, and the responsible party. More specifically, for each KPI, they indicated the data source(s), such as Microsoft Forms or system logs, and described the tools or methods used to gather the data, whether via automated tools or manual input for survey responses. They also defined how often data would be collected during the DTDP piloting period and after its deployment (e.g., daily, monthly, after each platform update), and they outlined the steps involved in measuring the data (e.g., survey completion, system log tracking, task completion during user tests). Finally, participants identified the individuals responsible for collecting the data and verifying its accuracy.

CONCLUSION

This paper has presented the conceptual and methodological foundations for developing a collaborative KPI framework to evaluate the DTDP designed and built as part of the CIRC-2-ZERO initiative. It primarily focused on the rationale, design, and structure of the participatory process, although it also presented preliminary results concerning the defined KPIs. The proposed methodology is intended to ensure stakeholder alignment, contextual relevance, clear criteria for future measurement, and flexibility for adaptation during DTDP piloting. Due to the structured approach, the initial KPI framework was not imposed in a top-down manner but instead emerged from collaborative reflection among experts familiar with both the technological and industrial aspects of the DTDP. The relevant steps included conducting Workshop 1 to co-create candidate KPIs and holding Workshop 2 to validate and prioritize those KPIs, before integrating the finalized KPI framework into the DTDP pilot evaluation plan. In Workshop 3, the experts defined detailed data collection procedures for each KPI, including the data sources, collection methods, frequency, measurement steps, and responsible parties. This methodological approach was found to be effective, and the final outcomes of the three workshops will be documented in future publications. The methodological foundations outlined here will contribute to ongoing research on CE-oriented digital tools and participatory evaluation frameworks.

ACKNOWLEDGMENT

This work was funded by the CIRC-2-ZERO Interreg BSR (EU) Project – Driving Collaborative Innovation towards Decarbonization and Advanced Manufacturing in SMEs across the Baltic Sea Region (https://interreg-baltic. eu/project/circ-2-zero/). The study reported in this article was conducted by researchers from the project’s Lead Partner, Häme University of Applied Sciences (Finland). The authors gratefully acknowledge the excellent contributions and insights shared by the panel members representing all 13 project partner organisations across seven Baltic Sea regions.

REFERENCES

Bana e Costa, C., Corrêa, E., De Corte, J. and Vansnick, J. (2002) ‘Facilitating bid evaluation in public call for tenders: a socio-technical approach’, Omega – The International Journal of Management Sciences, 30(3), pp. 227–242. DOI:10.1016/ S0305-0483(02)00029-4. Barricelli, B. and Fogli, D. (2024) ‘Digital twins in human-computer interaction: a

systematic review’, International Journal of Human–Computer Interaction, 40(2), pp. 79–97. DOI:10.1080/10447318.2022.2118189. Bluvstein, G., Kurscheid, S., Reinbold, N., Vorraber, W., Url, P., Orgler, M., Noori, S., Yaghmaei, E., Larsen, R., Thevenin, S., Rezaei, H., Wöss, S. and Daub, R. (2025) ‘Assembly lines in circulation – towards a holistic framework to enable the reuse of assembly resources’, 2025 IEEE 21st International Conference on Automation Science and Engineering (CASE). Los Angeles, CA, August 2025. Los Alamitos, CA: IEEE Computer Society. pp. 1213–1218. DOI:10.1109/ CASE58245.2025.11163991. de Larmelina, S., da Silva, A. and Risso, L. (2025) ‘A technology readiness assessment

approach for digital twin implementation in SMEs’, International Journal of Advanced Manufacturing Technology, 140(5-6), pp. 2777–2796. DOI:10.1007/ s00170-025-16370-5. Doukari, O., Bacnak, T. and Mahamedi, E. (2026) ‘Stakeholder-informed renovation digital twin for multi-domain monitoring: pilot implementation and acceptance evaluation’, Energy & Buildings, 354, pp. 1–13. DOI:10.1016/j. enbuild.2026.116979. Eden, C. and Ackermann, F. (2001) ‘SODA - the principles’, in J. Rosenhead and

J. Mingers (eds.) Rational analysis for a problematic world revisited: problem structuring methods for complexity, uncertainty and conflict. John Wiley & Sons, pp. 20–41. Eden, C. and Ackermann, F. (2004) ‘Cognitive mapping expert views for policy analysis in the public sector’, European Journal of Operational Research, 152(3), pp. 615–630. DOI:10.1016/S0377-2217(03)00061-4. Harvey, N. and Holmes, C. (2012) ‘Nominal group technique: an effective method

for obtaining group consensus’, International Journal of Nursing Practice, 18(2), pp. 188–194. DOI:10.1111/j.1440-172X.2012. 02017.x. Issa, R., Hamad, M. and Abdel-Geliel, M. (2023) ‘Digital twin of wind turbine based on Microsoft® Azure IoT platform’, 2023 IEEE Conference on Power Electronics and Renewable Energy (CPERE). Luxor, Egypt, February 2023. Los Alamitos, CA: IEEE Computer Society. pp. 1–8. DOI:10.1109/CPERE56564.2023.10119576. McMillan, S., King, M. and Tully, M. (2016) ‘How to use the nominal group and

Delphi techniques’, International Journal of Clinical Pharmacy, 38(3), pp. 655– 662. DOI:10.1007/s11096-016-0257-x. Mügge, J., Seegrün, A., Hoyer, T.-K., Riedelsheimer, T. and Lindow, K. (2024) ‘Digital twins within the circular economy: literature review and concept presentation’, Sustainability, 16(7), p. 2748. DOI:10.3390/su16072748. Pehlken, A., Davila, M., Dawel, L. and Meyer, O. (2024) ‘Digital twins: enhancing

circular economy through digital tools’, Procedia CIRP, 122, pp. 563–568. DOI:10.1016/j.procir.2024.01.082. Potter, M., Gordon, S. and Hamer, P. (2004) ‘The nominal group technique: a useful consensus methodology in physiotherapy research’, NZ Journal of Physiotherapy, 32(2), pp. 70–75. Santos, R., Piqueiro, H., Dias, R. and Rocha, C. (2024) ‘Transitioning trends into

action: A simulation-based digital twin architecture for enhanced strategic and operational decision-making’, Computers & Industrial Engineering, 198, pp. 1–12. DOI:10.1016/j.cie.2024.110616.

Soori, M., Arezoo, B. and Dastres, R. (2023) ‘Digital twin for smart manufacturing: a review’, Sustainable Manufacturing and Service Economics, 2, pp. 1–15. DOI:10.1016/j.smse.2023.100017. Souza, J., Mendonça, F., Soares, L. and Gomes Jr., J. (2026) ‘From cognitive to

circular: a digital twin systematic review’, Journal of Industrial Information Integration, 50, pp. 1–16. DOI:10.1016/j.jii.2025.101051. Swarnakar, V., Eltoukhy, A., Omar, M., Acquaye, A., Alawadi, K. and Khalfan, M.

(2025) ‘Unlocking circular economy potential in construction and demolition: an empirical analysis of success factors, failure drivers, benefits, and emerging technologies’, Smart and Sustainable Built Environment, pp. 1–21. DOI:10.1108/ SASBE-01-2025-0044. Van de Ven, A. and Delbecq, A. (1972) ‘The nominal group as a research instrument for exploratory health studies’, American Journal of Public Health, 62(3), pp. 337–342. DOI:10.2105/ajph.62.3.337. Zhu, B., Ahmadov, T., Kuba, O., Gerstlberger, W. and Prokop, V. (2026) ‘Bridging

inter-organizational collaboration, circular economy practices, and firm performance: the role of dynamic capabilities within Baltic states’, Corporate Social Responsibility and Environmental Management, 33(2), pp. 3037–3063. DOI:10.1002/csr.70314.

Cover of the proceedings volume Human Factors, Business Management and Society

Explore the complete digital volume

Human Factors, Business Management and Society

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-91-5

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →