Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Microbial Resource Research Infrastructure for the 21st Century: usability analysis of the Collaborative Working Environment

1 Centro de Computação Gráfica CCG/ZDGV, Guimarães, 4800, Portugal ↗
2 Department of Microbiology and Ecology, University of Valencia, Valencia 46100, Spain ↗
3 Independent Researcher, Guimarães, 4800, Portugal ↗
4 University of Torino, Torino, 10124, Italy ↗
5 Biological Resource Center of the Institut Pasteur, Institut Pasteur, Université Paris Cité, Paris 75006, France ↗
6 MIRRI-ERIC Central Coordinating Unit, MIRRI-ERIC, Braga 4710, Portugal ↗
Publication year
Digital Object Identifier10.54941/ahfe1003176

Abstract

Ease-of-use is of paramount importance to ensure the acceptability of a virtual platform. With that in mind, we sought to evaluate the usability of Microbial Resource Research Infrastructure (MIRRI)’s Collaborative Working Environment (CWE). Usability tests intend to determine whether an interface facilitates a user’s ability to complete tasks in the platform analyzed. For the MIRRI’s CWE user tests, 29 participants, aged between 25 and 65 years old, from 10 countries (Brazil, Chile, Colombia, France, Italy, Latvia, Netherlands, Peru, Portugal, Spain), with different professional backgrounds (i.e. Research/ Education, industry, Officers, among others) were invited to carry out activities within 3 modules of the platform, namely: (1) Microbial Resources Catalog (MRC), (2) Services and Workflows Catalog (SWC) and the (3) Transnational Access information and application submission (TNA). At the end of the experience, participants were asked to fill out the System Usability Scale (SUS) - a standardized questionnaire widely used to measure usability. Based on the SUS analysis, participants had a good perception of the usability of the MIRRI platform. MIRRI's CWE platform innovates in the unification of data and information from the microbiological universe. The usability analysis validates the user experience, thus ensuring that acceptance by the users is not hindered by poor ease-of-use.

INTRODUCTION

Microorganisms are crucial for research in fields such as environmental microbiology, biotechnology and medicine (Kee et al., 2021, O’Connor,

© 2023. Published by AHFE Open Access. All rights reserved. 42

Microbial Resource Research Infrastructure for the 21st Century 43

2021, Schneider 2021). With humanity facing several challenges such as climate change, pandemics and global energy crisis, developing a sustai- nable bio-based economy paves the way for new planet-friendly soluti- ons, and microbial resources are among the pillars of this economy. The Work Package 6 (WP6) of the project “Implementation and Sustainability of the Microbial Resource Research Infrastructure for the 21st Century” (IS_MIRRI21), funded by the European Union (EU) under the Horizon 2020 framework program, intends to develop and implement a Collabo- rative Working Environment (CWE) (www.mirri.org) that aims to offer its users and stakeholders, in a centralized way, microbial strain data, activities and services, provided by multiple pan-european institutions. To ensure the information available on the platform is easy to find and to understand, an usability test study was carried out as part of the WP6 activities.

Usability is an attribute that analyzes the ease of use of a product, or an interface, by the user (Nielsen, 1993), considering aspects such as efficiency, effectiveness, and satisfaction, within a specific and a social context of use (ISO 9241-210, 2019). For Nielsen (1993), usability is traditionally compo- sed of 5 dimensions, namely: Learnability, Efficiency of use, Memorability, Errors and Satisfaction.

The results from the study will allow developers to improve the design and/or contents of the CWE platform, and thus meet users’ expectations.

MIRRI CWE PLATFORM MIRRI builds the CWE as a single point of access to a broad range of high-quality microorganisms (including their derivatives and associated data) and services, ranging from general to more application-specific ones (inclu- ding pipelines, made available as tailor-made, turnkey solutions); expert advice on microbial resources, related aspects such as taxonomy, legal use or biosafety aspects, among other; and training programmes on the use and preservation of microorganisms to promote professional development of Microbiological Resources Centres (mBRCs) and Culture Collection person- nel. Interactions facilitated by the CWE will enable microbe-based innovation in fields such as health, food, and environment.

The MIRRI CWE contains three modules offering key information: the Services and Workflows Catalogue (SWC), the Microbial Resources Catalogue (MRC) and the Transnational Access programme platform (TNA) (The platform as described next corresponds to the implementation at the time of the research and has since been updated).

The MIRRI SWC (https://www.mirri.org/services/) offers more than 90 general services and more than 30 application-specific services and work- flows. High-level experts and advanced equipment are at the service of the scientific community to provide solutions in the fields of Biodiversity conse- rvation, Health and Food, Agrofood, Environment and Energy (i.e., MIRRI Strategic Areas). The SWC is divided into two catalogues: General Services catalogue, which includes services with broader purposes (e.g. Identification by gene sequencing), and Application-Specific services catalogue, aimed at

Figure 1: SWC General services catalogue (Left); main page of the TNA (Right).
Original PDF Figure 1:
Figure 2: Main page of the MRC.
Original PDF Figure 2:

presenting oriented solutions (e.g., “Screening of organic pollutants’ biotran- sformation activities”) towards the different strategic areas. Users can visit the publicly available catalogues, ask for information and request services by the tools provided (Figure 1, Left).

The TNA platform (https://www.mirri.org/services/transnational-access- tna/) (Figure 1, Right) gathers the information concerning the Transnational Access program, which provides free-of-charge access to MIRRI microbial resources, services and facilities. All the information about the calls and procedures to apply (including the link to the platform dedicated to the submission of applications) is organised into different tabs. Additionally, it contains key documents in PDF format to be downloaded.

Finally, the MIRRI MRC (https://www.mirri.org/microbial-resources- data/) is an European database of microbial resources. It hosts top-quality and curated data from more than 400.000 microbial resources held in more than 50 biorepositories in Europe. The MRC is publicly available and users can find information of microbial resources by using the search engine which queries different database fields using logical combinations (Figure 2).

METHODOLOGY

To collect feedback on the CWE and to analyse its functionalities, moderated usability tests with end-users were planned under task 6.5 of the WP6: “Task 6.5: Evaluation of MIRRI CWE by the end-users”. In this perspective, the

Microbial Resource Research Infrastructure for the 21st Century 45

three modules of the CWE described above, SWC, MRC, TNA platform, were tested.

The sessions were conducted online, registering each participant’s navi- gation choices, task completion rates, SUS (System Usability Scale) and feedback from the think aloud protocol and a short interview at the end. However, in this paper only the SUS and the task evaluation analysis were reported, due to WP6 confidentiality issues. Five moderators, who were part of three project partner organizations, conducted the sessions and data analysis.

The tasks were defined according to different search options, fields and tools in the MRC and SWC, and to the information displayed in the TNA platform. Six pilot tests were undertaken to ensure the feasibility of the tasks and the protocol.

Participants For the MIRRI’s CWE user tests, 29 participants (13 females and 16 males), aged between 25 and 65 years old (Mean 41.7, SD = 11.6), from 10 countries (Brazil, Chile, Colombia, France, Italy, Latvia, Netherlands, Peru, Portugal, Spain) with different professional backgrounds, no previous contact with the CWE, were recruited. No compensation was provided for participating. The selection of users was based on target professional profiles, mainly related to microbiology research and biotechnological companies, being distributed as follows:

  • Research/Education Institutions + microbial Biological Resource Centres (mBRC)/ Culture Collections (CC): 4 Junior researchers (MRC + 1 SWC + 2 TNA); 4 Senior researchers (1 MRC + 1 SWC + 2 TNA); 2 Research technicians (1 MRC + 1 SWC); 2 Curators (1 MRC + 1 SWC); 1 Teacher (non-researcher) (1 MRC); 1 Postgraduate student (1 MRC);
  • Industry (selected from start-ups, Small and Medium Enterprises (SMEs) and large companies - NOTE: TNA is for start-ups + SMEs only): 1 Operations Officers (1 SWC); 1 Outsourcing/ Procurement/Partnership managers (1 MRC); 3 Chief Scientific Officers (2 MRC + 1 SWC); 4 Staff researchers (1 MRC + 1 SWC + 2 TNA); 2 Staff technicians (1 MRC + 1 SWC);
  • Other: 3 Access/Operations Officers from another Research Infrastructure (e.g. BBMRI, EMBRC, EVAg, DSMZ, …) (1 MRC + 1 SWC + 1 TNA); 1 Science communication professional (1 MRC) This totaled 13 participants for MRC, 7 for TNA and 9 for SWC. Sixteen participants worked in public sectors, 11 in private and 2 in private but non- profit foundations. The online sessions were scheduled between May and July 2022.

Usability Test One script per feature was designed and proposed. The structure was the same for all the three modules:

  • Introduction: a recording consent request, an explanation of the test procedure and a background question were included.
  • Scenarios and tasks: an introductory scenario contextualized the user for completing a group of tasks. The MRC script contained seven scenarios and ten tasks; the SWC was designed with four scenarios and seven tasks and the TNA script was composed of one scenario and ten tasks.
  • Assessment: an open question about the overall experience, and the SUS questionnaire, composed of 10 statements for measuring usability (i.e. effectiveness, efficiency, and overall ease of use) allowed for the module final assessment.

The tasks were defined to encourage the user to explore the contents of each platform through search tools (fields, options); examine the informa- tion provided; visualize and interpret the interface; place orders, and ask for information. For each CWE module chosen, a script was prepared with a set of tasks (Table 1).

Task Evaluation The tasks were evaluated by objective and subjective metrics, namely:

  • Success/failure in each task: to be successful the user must complete the tasks in time (3-5 minutes). A task failure is considered when the participant takes more time or abandons the task.
  • Errors: number of path errors while solving the task;
  • Comments: users’ comments, suggestions and think-aloud activity.

The results were analyzed and the identified issues were classified accor- ding to their criticality:

  • Critical: If we do not fix this, users will not be able to complete the scenario. This issue needs to be fixed by all means necessary before a release;
  • Serious: Many users will be frustrated or give up if this isn’t fixed. It has to be fixed as soon as possible;
  • Minor: Users are annoyed, but this does not keep them from completing the task (e.g., cosmetic issues that are necessary to fix, but don’t hinder the user experience as much as critical or serious issues). This should be revisited later.

SUS The SUS questionnaire gives a global view of subjective perception of usa- bility (Brooke, 1996). It has become a popular usability questionnaire for end-of-test (Sauros & Lewis, 2016, p. 198). It is composed of ten items with five Likert scale response options (Brooke, 1996).

Data Analysis This study followed the SUS calculations by Brooke (1996), the analysis metrics by Bangor et al. (2009, Figure 3, Left), which points out whether the result is non-acceptable, marginal or acceptable, and the percentile analysis result by Sauro and Lewis (2016, Figure 3, Right).

Microbial Resource Research Infrastructure for the 21st Century 47

Table 1. Set of tasks.

Table available in the original PDF paper.

MRC 1. Using the search tool of the MIRRI Strains Catalogue, look for the Type strains of species of genus Trichoderma 2. Using the search tool of the MIRRI Strains Catalogue, look for Listeria monocytogenes strains isolated from cheese 3. Using the search tool of the MIRRI Strains Catalogue, look for strains of Saccharomyces cerevisiae isolated in France between 2017 and 2022 4. Using the search tool of the MIRRI Strains Catalogue, look for strains of Escherichia coli isolated from humans and available at the CECT 5. Open the first of the list and by inspecting the relative data-sheet find the CECT accession number 6. Using the search tool of the MIRRI Strains Catalogue, look for all Cyanobacteria isolated in Spain with no restriction under Nagoya protocol 7. Find the CBS strains Ilyonectria europaea isolated in Chile 8. By inspecting the relative data-sheet, find out which medium is recommended for culturing this fungus 9. Using the search tool of the MIRRI Strains Catalogue, find the total number of the preserved bacterial strains 10. Using the search tool of the MIRRI Strains Catalogue find the total number of Filamentous Fungi with the exception of Penicillium chrysogenum TNA 1. Find if the TNA covers travelling expenses 2. Find the TNA eligibility criteria 3. Find the TNA catalogue (the PDF document) 4. Find if there is any TNA offer about dermatophytes, and/or identification of fungal strains and /or typing of fungal strains 5. Find if you can apply for the TNA offers found in the last task (if any) based on your country of origin and your position (in public health services) 6. Find the information on application procedure steps 7. Find the guidelines for applicants 8. Find the portal for proposal submission 9. Find how the application is evaluated: what is the USP (users section panel), the criteria, and the scores 10. Find the deadline of the applications

SWC 1. Find the service genome sequencing and add it to the cart 2. Find if the service includes “gene annotation”. If not, go back to the catalogue, find this service and add it to the cart 3. Order both services (genome sequencing and gene annotation); 4. Find the screening service “Analysis of the resistance/sensitivity of strains to physical and chemical stressors”. Search by using the categories and subcategories 5. Find the service to get the strain as active culture and through it, connect to the MIRRI strains catalogue to search for Pseudomonas strains 6. Find the service “Analysis of the microbiome of metal contaminated samples” and check if it fits your requirements 7. Regarding this service, contact MIRRI and ask for details about the techniques behind the analysis by sending the request

Boucinha and Tarouco (2013) stratified the 10 SUS’ questions within Nielsen’s 5 attributes of usability (1993), called here as: (1) Satisfaction, questions 1, 4 and 9; (2) Ease of memorization, question 2; (3) Ease of lear- ning, questions 3, 4, 7 and 10; (4) Efficiency, questions 5, 6 and 8; and (5) Minimization of errors, question 6.

Figure 3: SUS Parameters (Bangor et al., 2009), Left; curved grading scale interpreta- tion of SUS scores (Sauro & Lewis, 2016), Right.
Original PDF Figure 3:

RESULTS AND DISCUSSION

Task Evaluation Results The success and errors metrics for each task were analyzed (Figure 4). In SWC, five of seven tasks were 100% successful, with failures in tasks 1 and 4. In the two tasks that failed, the participants mentioned that the titles of the categories and subcategories were sometimes not easy to understand (e.g. in task 4, they usually confused between the categories “Phenotypic characterisation” and “Screening, tests and bioassays”) or have difficul- ties in discriminating services labels (e.g. in task 1, between “Genome Sequencing…” and “Gene sequencing…”). Regarding the errors, there was a great variability among the participants. In addition, six critical issues and eight serious were detected, basically related to the graphical location of some items, missing information or information that confuses the user.

In the case of TNA, eight of ten tasks were 100% successful. The failure rate in Task 4 was very high and undoubtedly needs attention for revision, suggesting the lack of a search tool or more details on the web page to per- form the task. In task 7, the problem focused on locating the information and where the information was provided. Participants did not make any mistakes in seven out of the ten tasks, but errors were committed by almost all parti- cipants in Task 4 and Task 10. Regarding the last, users were not able to find

Figure 4: Task evaluation results of the usability tests.
Original PDF Figure 4:

Microbial Resource Research Infrastructure for the 21st Century 49

Figure 5: SUS score: MRC, TNA, SWC (Left), SUS percentiles for the overall study. See Figure 3 right for the correspondence between grades (A-F) and SUS scores (Right).
Original PDF Figure 5:

the information about the deadline because of its location. Furthermore, no critical or serious issues were identified in performing the tasks.

Regarding MRC, two participants (out of 13) had difficulties in carrying out the tasks. Their personal profile was examined to detect a hint that could explain their behavior: they were both 50/55 years old, female and male, Chief Scientific Officers (CSO) from different countries. As they were the only CSOs in this MRC test, perhaps this professional dedication could be related to this response. This observation would however need further analysis with a larger study population (the research had another CSO participant, with the same profile, but 10 years younger, who did well in the SWC). However, even excluding these two participants, only 3 out of 10 tasks were 100% successful with the other participants. Together with the high error rate, it shows that the MRC has potential to be improved. Lastly, two critical issues, related to the graphical design part of the menu, and eight serious issues, related to the graphical visibility of some icons, were found.

SUS Results The final SUS Score was 81 for TNA, 77.8 for SWC and 75 for MRC. All of them are above the acceptability index of Bangor et al. (2009). On the other hand, only TNA has a score over 80, as shown in Figure 5, Left.

Regarding the percentile ranks (Figure 5, Right)., almost half of the parti- cipants evaluated the MIRRI CWE modules within the 95% percentile of the usability benchmark proposed by Sauro and Lewis (2016). It is important to emphasize that at SWC none of the participants scored the module under non-acceptable rates. Only three users scored the test below 68 (Brooke, 1996, average score). In TNA, only 1 person had a SUS score in the lower marginal area. All others had a score above 77.5. In the MRC, the two CSO scored in the unacceptable area (20 and 32.5). All other participants had a score equal to or above acceptability.

Figure 6: SUS stratification.
Original PDF Figure 6:

The analysis of the SUS under the perspective of the Nielsen attributes revealed scores above the threshold of acceptability for all parameters, except ease memorization in the SWC (Figure 6).

DISCUSSION

In general, participants were satisfied with the tested MIRRI CWE modules. Considering SUS scores, and Nielsen attributes, the functionalities provided by the software modules were easy to use and learn, memorisable (with the exception of the ease of memorisation of the SWC, which was marginal accor- ding to Bangor et al., 2009, with minimal errors, and the navigation was considered efficient. Most of the participants (48%) were concentrated in A+, the higher grading scale according to Sauros and Lewis (2016), with 17% of users between D and F. On the other hand, the SUS score of two modules were below 80, which is becoming a common industrial goal of excellence, corresponding to an A- in the Sauro-Lewis curved grading scale (Sauros & Lewis, 2016).

For all three modules the perceived usability was good as indicated by the SUS results. For SWC and TNA, the participants did not have too many difficulties to follow the scenarios and tasks, which were performed quite well according to the number of successes and errors.

Like any starting platform, there is great potential for improvement. In the SWC and TNA the main observations and issues derived were gathered and some recommendations were added to facilitate finding the important infor- mation through search tools, links and user interface design improvements. In the MRC, the main difficulty was a layout confusion, where most users did not find the correct space for the actions. Still regarding the MRC, the participants suggested the addition of a glossary with the description of all terms in the searchable fields, at the beginning of the MRC website.

All this information has been prioritized and the most significant impro- vements have already been implemented.

Microbial Resource Research Infrastructure for the 21st Century 51

CONCLUSION

Faced with a globalized world and the fact that microorganisms are crucial for society, the CWE is an innovative platform that allows the sharing and unification of information internationally. The concern with the user expe- rience, more specifically with the usability of the interface, demonstrates, in addition to care with the project, the search for a more user friendly, adequate and satisfactory operation in accordance with the expectations of the users.

This research demonstrated engagement of the microbiology community, which can be evidenced with the participation of people from 10 different countries, the majority of them coming from 4 different countries (Spain, France, Italy and Portugal) of the MIRRI Consortium. Based on the SUS analysis, participants had a good perception of the usability of the MIRRI CWE module platform. Regarding the performance of the tasks, in general the participants performed well, completing most of the tasks. Some issues were identified with improvements were already implemented.

Overall, the three CWE modules were well-received and accepted among participants in terms of usability, but changes in the design and tools can still be foreseen for future releases, regarding the interaction of the users with the CWE.

ACKNOWLEDGMENT

This work was financially supported by project “Implementation and Sustainability of Microbial Resource Research Infrastructure for the 21st Century”(IS_MIRRI21), from the European Union’s Horizon 2020 Research and Innovation programme, under the Grant Agreement no 871129. The work presented here would not have been possible without the dedicated efforts of all members of the MIRRI consortium, particularly the teams in Work Package 6.

This publication reflects only the author’s view and the European Commission is not responsible for any use that may be made of the infor- mation it contains.

REFERENCES

Bangor, A., Kortum, P., Miller, J. (2009) Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. JOURNAL OF USABILITY STUDIES Volume 4 No. 3. Boucinha, R. M., Tarouco, L. M. R. (2013) Avaliação de Ambiente Virtual de

Aprendizagem com o uso do SUS - System Usability Scale. RENOTE Volume 11 No. 3. Brooke, J. (1996) “SUS: A ‘quick and dirty’ usability scale”, in: Usability Evaluation in Industry, Jordan, P., Thomas, B., Weerdmeester, B. (Eds.), London: Taylor & Francis. ISO 9241–210 (2019) Ergonomics of human-system interaction. Kee, S. H., Chiongson J. B. V., Saludes J. P., Vigneswari S., Ramakrishna S., Bhubalan

K. (2021) Bioconversion of agro-industry sourced biowaste into biomaterials via microbial factories - A viable domain of circular economy. ENVIRONMENTAL POLLUTION Volume 271.

Nielsen J. (1999). Designing Web Usability: The Practice of Simplicity. California: New Riders Publishing Thousand Oaks. Nielsen, J. (1993) Usability Engineering. San Diego: Academic Press. O’Connor KE. (2021) Microbiology challenges and opportunities in the circular

economy. MICROBIOLOGY Volume 167 No. 1. Sauro, J., Lewis, J. R. (2016) Quantifying the user experience: Practical statistics for user research. Morgan Kaufmann. Schneider Y. K. (2021) Bacterial Natural Product Drug Discovery for New Antibi-

otics: Strategies for Tackling the Problem of Antibiotic Resistance by Efficient Bioprospecting. ANTIBIOTICS Volume 10 No. 7.

Cover of the proceedings volume Usability and User Experience

Explore the complete digital volume

Usability and User Experience

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-86-5

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →