Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Measuring Mental Workload of Medical Physicists, Radiation Therapists and Dosimetrists at the Five Stages of Radiation Treatment Planning in External-Beam Radiotherapy using NASA-TLX

1 Irish Centre for Patient Safety, National University of Ireland Galway (NUIG), Galway, Ireland ↗
2 Department of Medical Physics and Bioengineering, Galway University Hospitals, Galway, Ireland ↗
Publication year
Digital Object Identifier10.54941/ahfe100537

Abstract

Mental Workload of Medical Physicists, Radiation Therapists and Dosimetrists was measured at five stages of Radiation Treatment Planning in External-Beam Radiation Therapy using NASA-TLX, a self-reported subjective technique for assessing mental workload developed by NASA (Hart and Staveland (1988). The five stages of the Treatment Planning process which were considered were; Image Import and Prescription Review, Target Contouring, Beam Manipulation and Calculation, Physician Approval, and Export of Plan to Pre-Treatment Review. Besides the assessed workload, the discrepancies and deviations from protocol (defined here as errors) reported during the five stages of the Treatment Planning process were also recorded using two independent internal reporting systems. The work was carried out over a two week period during September and October 2013 and 21 treatment cases were assessed in total. Medium to high perceived mental workload was reported in all five stages of the Treatment Planning process. Nine cases of errors were rectified and recorded in 7 out of the 21 treatments. The workload and error values recorded are discussed, as well as the association between errors and mental workload.

INTRODUCTION

Radiotherapy is a well established cancer treatment modality. External-Beam Radiation Treatment (EBRT) consists of the following main activities; 3D Imaging and Contouring, Treatment Planning, and Treatment Delivery. It involves a range of disciplines including Radiation Oncologists, Medical Physicists, Radiation Therapists, Dosimetrists and Nurses. It is a high risk process in which errors can occur at any point. Treatment Planning is concerned with the preparation of plans which ensure that a radiation dose specified by the Radiation Oncologist is delivered safely to patients. Errors in this activity can have a devastating effect on patients (WHO, 2008 & Bogdanich, 2010).

Treatment Planning is an intensive computer based activity. It involves significant 3D modelling of topographical Human Aspects of Healthcare (2021) aspects of organs of the body to be irradiated and the radiation dose to be administered. The Medical Physicists, Radiation Therapists, Dosimetrists involved in the planning process use specialist software which places significant demands on their cognitive resources. Typically, this involves a high information load due to the complexity of the Treatment Planning software, the complicated nature of communicating with other software devices such as Computed Tomography scanners, the risk associated with the decisions to be made, and the limited time frame within which the work has to be completed. The level of error which can be tolerated is very low due to the potentially catastrophic nature of the associated outcomes. Consequently, controls are in place to detect and recover from errors, and also to learn from them. As a result, significant emphasis is placed on quality assurance (QA) and error reporting in radiotherapy.

The five main stages of treatment planning relevant to the current case are presented in Figure 1. They were adapted from the stages of EBRT Process Maps presented by Ford et al. (2012) and best practice in the institution studied.

Figure 1. Five stages of the Treatment Planning process
Original PDF Figure 1.

Across various industries, levels of workload have been identified as major influences on human performance. Typically, it is standard practice to try to ensure that workload is neither too high nor too low. When workload is too high, operators generally become stressed, are unable to complete their relevant tasks and often fail to complete their work to the required quality standards. When workload is too low, operators can become bored, lose interest, and fail to respond properly to task dynamics (Jex, 1988). It is important to be aware of workload levels in treatment planning in order to understand the potential for error in performance and to take remedial action when required.

Within Human Factors, a number of methods are available to assess workload, including the Cooper & Harper Scale (Cooper & Harper, 1969), SWAT (Reid et al., 1981), NASA-TLX (Hart & Staveland, 1988), SWORD (Vidullch et al., 1991), Simplified SWAT (Luximon & Goonetilleke, 2001). NASA-TLX has been used extensively across a broad range of applications, and has been found to be one of the most, “effective and widely used measures of perceived workload currently available for examining transitions in task load” (Satterfield, 2012). It has been used to analyse workload across a broad range of Radiotherapy cancer treatment processes (Mosaly et al., 2011, Mazur et al., 2012, Mazur et al., 2014). However, in the case of Treatment Planning, the only reported analysis was carried out using a single stage Treatment Planning process which is not reflective of the five distinct processes within Treatment Planning referred to in Figure 1 (Mosaly et al., 2011, Mazur et al., 2012).

In the study reported in this paper, NASA-TLX was applied to Treatment Planning using the five stage model presented in Figure 1. Also, the analysis was carried out in the actual working environment utilising a fully staffed team in a major teaching hospital in Ireland. Medical Physicists, Radiation Therapists and Dosimetrists were involved at the various stages of the Treatment Planning process. Following the application of NASA-TLX, errors associated with the treatment plans analysed were identified from internal error reporting systems. The relationships between these errors and workload were investigated.

METHODS

Research implementation The data collection phase of this study was carried out at Galway University Hospital over a two week period during September and October 2013. Ethical approval for the study was obtained from the Clinical Research Committee of Galway University Hospitals; Reference number “C.A. 808 – Data Collection for the Radiation Oncology Systems Safety Analysis”. Volunteer participants were provided with an information sheet which outlined the purpose of the study. It explained that all of the data collected would be anonymous with respect to both the participants and the patient data concerned and that it would not be used for any other purpose. Consent was obtained from the participants and they were informed that they were free to withdraw from the study at any time.

NASA TLX In the research reported in this paper, NASA-TLX was used to assess the mental workload of the professionals involved in the five stages of Treatment Planning; Image Import and Prescription Review, Target Contouring, Beam Manipulation and Calculation, Physician Approval, and Export of Plan to Treatment Delivery as presented in Figure 1. NASA-TLX is a self-reported subjective technique for assessing mental workload that was developed by NASA (Hart & Staveland, 1988). Using NASA-TLX, workload is assessed on six dimensions (Mental Demand (MD), Physical Demand (PD), Temporal Demand (TD), Performance (PE), Effort (EF), and Frustration (FR)) which are compared against each other and recorded on a 21 point scale. The workload score is calculated on a scale between 0 and 100 with individual points defined by increments of 5. The scores are combined and the proportion of the effect is determined. Each dimension is defined individually, as presented in Figure 2.

Seven team members who carry out treatment planning as a normal part of their duties participated in the study. Basic training lasting approximately one hour was provided to them on the topic of Workload and the use of the NASA-TLX tool. Twenty one Treatment Planning cases were evaluated using NASA-TLX. In total 105 NASA- TLX worksheets were administered (21 Cases x 5 Stages of Treatment Planning). Each of the seven participants completed three cases. The worksheets were self-reported and filled-in immediately after each of the five stages was concluded.

Error Reporting System For the purpose of this study an error was defined as a deviation from standard protocol that has the potential to affect patient treatment. This is similar to the definition by Ford et al. (2012): “Failure to complete a planned action as intended, or the use of an incorrect plan of action to achieve a given aim.” During this study all errors identified were corrected prior to treatment. The errors reported during the five stages of the Treatment Planning process were recorded using two internal reporting systems: Physics Check (checking carried out by Physicists) and Therapist Check (checking carried out by Radiation Therapists). They are both pre-treatment checking systems and well established in Radiotherapy. They are required to be independent of the planner (other person check).

The reported errors are quantified according to the Dosimetric Severity Scale discussed by Ford et al (2012). It is a 1-10 point scale describing dose deviation per treatment; scores 1 & 2 (being defined on the same level) having <5% absolute dose variation and score 9 & 10 having >100% absolute dose variation. The Dosimetric Severity was analyzed and applied to the scale accordingly. E.g., the score 1 & 2 is defined as a 0-5% dose deviation. Therefore, if the value was less than 2.5% it was taken as a severity of 1. If the value was between 2.5 and 5%, it was assigned a Severity Scale of 2.

Figure 2. Workload definitions

Figure available in the original PDF paper.

The errors associated with 21 Treatment Plans undertaken in this study were analysed after the administration of NASA-TLX. The errors were then scored using the Dosimetric Severity Scale.

RESULTS

NASA-TLX Ninety eight out of the 105 NASA-TLX worksheets administered were completed satisfactorily, representing a response rate of 93%. The main reasons for incorrect or non-completion of worksheets were errors of commission (filling out) on 5 worksheets, and also errors of omission (failing to fill in) relating to 2 worksheets of a particular plan and the need to repeat the entire planning process. All data analysis was carried out using SPSS v21 statistical software.

The results for the following data are presented in Tables 1-2 based on the NASA-TLX guidelines developed by Hart and Staveland (1988).

1. Overall NASA-TLX scores for each of the five stages of Treatment Planning: Image Import and prescription review, Target Contouring, Planning / Beam manipulation and calculation, Doctor Approval, and Plan export / DRRs).

a. Minimum (Min.), Maximum (Max.), Mean and Standard Deviation (SD) for each stage (see Table 1); b. Workload Rating Description based on the Workload Rating Scale for each of the five stages (see Tables 1 and 3). 2. NASA-TLX workload scores for each TLX dimension expressed as a percentage contribution to the total score (see Table 2). The dimensions and their contributions were: EF (27%), MD (24%), and TD (22%), followed by FR (16%), PE (10%) and PD (1%).

The TLX values were rounded to the nearest decimal number. A simple workload rating scale presented in Table 3 was applied to the results. The cut-off point for high workload was selected as 55 or greater, taking into account the workload score variability and following from the work of Mazur et al. (2012).

Table 1: TLX workload scores of the assessed stages Min. Max. Mean SD Worklo
Original PDF Table 1:
Table 2: Mean values of the TLX workload scores of each TLX dimension (in %)
Original PDF Table 2:
Table 3: Workload rating scale (Galičič, Fallon, van der Putten, Sands, 2013) Workload level Workload score level
Original PDF Table 3:

Reported errors Nine errors were identified in a total of 7 treatment plans out of a total of 21. The results for the data with respect to the errors identified are presented in Tables 4-6 under the following headings;

1. The corresponding Treatment plan case number, Error description, Dosimetric Severity Score based on Ford et al. (2012). Each Severity Score is assigned an error number for identification purposes (see Table 4); 2. NASA-TLX workload scores for each of the five Treatment Planning stages for the seven cases in which errors occur (see Tables 5-6).

The Dosimetric Severity classification was assessed based on the evidence given in the description in the Physics Check and Therapist Check. Errors number 6 and 8 did not provide enough information to apply an accurate classification so one was estimated on the evidence provided. These are represented by 1*. A delay in treatment was rated as a Dosimetric Severity equivalent of 1. This is based on the assumption of a minimal effect to the radiobiological due to the delay. This is a subjective rating and is open to interpretation. A recurring error involved the selection of Diodes for treatment. Diodes are used to ensure the correct dose delivery. This in itself would not result in any harm to the patient, but does have the potential of preventing the identification of further errors. There is also a high probability that errors associated with “Diodes not correctly applied” would be detected later in the treatment process and corrected. On this premise these errors were included but rated as 0.

The association between workload and reported errors A list of Treatment Plan cases and their associated error description and Severity Scores are presented in Table 4. The workload for each Treatment plan case was assessed individually. A summary of the TLX workload scores of the Treatment Plan cases where the errors reported occurred is presented in Table 5. The mean values for each TLX dimension of the Treatment Plan cases where the errors reported occurred are presented in Table 6. The missing values were due to errors of commission and errors of omission, as described above.

Table 4: Table of reported errors with severity ratings Treatmen t plan Error description Severity Error number score case
Original PDF Table 4:
Table 5: TLX workload scores of the Treatment Plan cases where errors reported occurred (in %) Min. Max. Mean SD Workloa Workloa d score
Original PDF Table 5:
Table 6: Mean values of the TLX workload scores of treatment plan cases where errors reported occurred (in %)
Original PDF Table 6:

DISCUSSION

Workload Moderate to high perceived mental workload was reported at all five stages of the Treatment Planning process (see Table 3). The highest workload range was at the Planning / Beam Manipulation and Calculation stage, followed by the moderate workload range at the Doctor Approval, Target Contouring, Image Import and Prescription Review, and Plan Export stages. The workload scores in this study include a more diverse range of Treatment Planning processes than in the studies by Mosaly et al. (2011), and Mazur et al. (2012). The workload scores reported in this study had a mean score of 43 for five stages of Treatment Planning, whereas Mosaly et al. (2011) reported a workload score of 56 for a single stage. However, in Mazur et al. (2014), the workload scores reported for Treatment Planning considered as a single stage, in a simulated environment, had a mean score of 46, which is similar to the results in this study.

TLX dimensions The dimensions that contributed the most to the five stages evaluated were EF (27%), MD (24%), and TD (22%), followed by FR (16%), PE (10%) and PD (1%). The results reflect the nature of the job which is demanding in terms of mental effort. The work requires significant attention and constant vigilance to ensure optimal performance. Often there are tight deadlines in order to complete a specified work schedule within a specified time frame, irrespective of the demands of the individual case. At the same time, there is a dependence on the performance of the relevant computer hardware and software. It is not uncommon for computer software to freeze or crash during its operation. This can contribute to the loss of data, as the task may have to be completed twice, e.g. Replanning. One of the most common comments reported on the TLX Worksheets as causes of higher workload were, Manual Contouring (6 cases), Replanning (5 cases), and computer crashes (2 cases). There may be a need to develop rules for scheduling different kinds of cases in order to smooth and optimise the workload, and to investigate manual versus automation concerns. For example, in some cases boluses had to be contoured manually rather than automatically, and this could have influenced the workload levels and/or the distribution of TLX dimensions.

Errors Errors were reported in 7 out of 21 cases (33.33%) that occurred during the assessment. Physics Check and Therapist Check were used to track errors in the various stages of the treatment process. Most of the errors happened at the 3rd stage, Planning / Beam Manipulation and Calculation (errors 2, 3, 4, 5, 7, 8 and 9 from Table 4), and the 5th

stage, Plan export / DRRs (error 6 from Table 4). However, error 1 was initiated at the Pre-Planning stage, and was only discovered after the Planning stage. In Error 1, the original dose prescription and the amended dose prescription were regarded by medical staff as clinically equivalent to each other from a risk point of view. In practice, the original prescription would not have resulted in a radiation overdose, but would have resulted in an underdose. This was regarded by medical staff as a lower category error as it was possible to recover from it, whereas it would be impossible to recover from an overdose.

The association between errors and workload The reasons for the difference in the number of cases (N) reported in Tables 5 and 6 were because some of the worksheets were not filled out correctly, or the evaluated stages were left out on purpose because of a replan, as it was in one of the examples in this case.

The dimensions that made the greatest contribution to workload of the five stages evaluated were TD (27%), EF (24%), MD (21%), FR (16.5%), PE (10.5%), and PD (1%). Compared to the overall workload scores, the values of the dimensions changed slightly in Treatment Plans where the errors occurred. There was an increase in TD (5%), and EF (3%), while there was no difference in PD. There were very small differences in FR and PE (0.5% in each dimension), however, there was a decrease in MD (3%). The way the dimension scores were distributed was different in the cases where errors occurred than in the other cases.

The following results were obtained as a result of the comparison between the workload scores for the tasks in which the errors occurred and the overall workload scores: The workload was higher at the Image Import and Prescription Review stage (45.52 compared to 40.44), Target Contouring stage (45.66 compared to 41.29), and Doctor Approval stage (49.94 compared to 41.86). The workload was lower at the Planning / Beam Manipulation and Calculation stage (48.18 compared to 53.60) and at the Plan Export / DRRs stage (34.99 compared to 38.80).

The compared workload dimensions between the workload scores where the errors occurred and the average workload scores brought the following results:

1. MD was slightly higher at the Planning / Beam Manipulation and Calculation stage (25.80 compared to

25.81), and lower at the Image Import and Prescription Review stage (19.35 compared to 26.33), Target Contouring stage (15.35 compared to 19.78), Doctor Approval stage (24.16 compared to 28.22), and Plan Export / DRRs stage (18.64 compared to 21.23). 2. PD was higher at the Image Import and Prescription Review stage (3.71 compared to 1.38), and lower at the Target Contouring stage (0.00 compared to 0.57), Planning / Beam Manipulation and Calculation stage (0.00 compared to 0.52), Doctor Approval stage (1.16 compared to 1.47), and Plan Export / DRRs stage (0.00 compared to 0.15). 3. TD was higher at the Image Import and Prescription Review stage (26.42 compared to 22.52), Target Contouring stage (32.42 compared to 23.33), Doctor Approval stage (25.16 compared to 19.62), and Plan Export / DRRs stage (39.50 compared to 29.00). TD was lower at the Planning / Beam Manipulation and Calculation stage (12.40 compared to 14.73). 4. PE was higher at the Image Import and Prescription Review stage (12.64 compared to 11.35), Target Contouring stage (15.21 compared to 12.50), and Planning / Beam Manipulation and Calculation stage (13.00 compared to 10.44). PE was lower at the Doctor Approval stage (9.41 compared to 10.01), and Plan Export / DRRs stage (2.78 compared to 5.78). 5. EF was higher at the Planning / Beam Manipulation and Calculation stage (38.00 compared to 32.39), and lower at the Image Import and Prescription Review stage (24.42 compared to 25.19), Target Contouring stage (18.57 compared to 24.35), Doctor Approval stage (19.58 compared to 22.67) and Plan Export / DRRs stage (23.00 compared to 30.36). 6. FR was higher at the Image Import and Prescription Review stage (13.28 compared to 13.16), Doctor Approval stage (20.50 compared to 17.85) and Plan Export / DRRs stage (16.07 compared to 13.44). FR was lower at the Target Contouring stage (18.42 compared to 19.45), and Planning / Beam Manipulation and Calculation stage (16.10 compared to 17.07).

The results reflect the performance in these cases. Because of the time pressure, the work pace was most probably rushed (and therefore resulted in a higher TD). The cases where the workload dimensions occurred reported decreased MD, while FR and PE stayed at almost the same level. This could explain that the quality of work performed was still at the level of the regular working standard.

The change in the distribution of workload dimension scores may be attributable to the errors that occurred. Most of the errors (77.78%) occurred at the Planning / Beam Manipulation and Calculation stage. The workload dimensions at this stage were higher for MD, PE, EF, and lower for PD, TD, and FR when compared to the average workload levels of these dimensions. The remaining errors (22.22%) occurred at the Plan Export / DRRs and at the Pre- Planning stages. In this case, the workload dimensions were higher for TD and lower for MD, PD, PE, EF, and FR when compared to the average workload levels of these dimensions.

The change in the distribution of workload dimensions could also be attributable to the level of automation deployed by staff at different stages. For example, it was possible to choose between manual and automated contouring, however, automation issues were not investigated in this study.

The results obtained in this study for the TLX workload scores for Treatment Planning are lower when compared to the results obtained by Mosaly et al. (2011). The mean score obtained for Treatment Planning in this study was 43, which is categorised as “Moderate”, while the mean score obtained for Treatment Planning in that of Mosaly et al. (2011) was 56, which is categorised as “High”. It is possible that the different results can be explained by the greater number of stages included in this study. In other research, Mazur et al. (2014) reported that MD as well as TD and FR were the main contributors to the workload score of 46 in Treatment Planning. The score obtained in that study is similar to what is reported in this study. Differences may reflect the conditions of the simulated environment of that research, and not the actual Radiotherapy environment which was reported on here.

CONCLUSIONS

The work completed should facilitate targeted improvements and workload reductions within the Treatment

Planning activity, especially at the Planning / Beam Manipulation and Calculation stage, at which the workload was reported as the highest. Based on the collected data, improvements should be implemented which result in the reduction of EF, MD as well as the levels of FR. Higher levels of FR were mostly due to crashes of computer software, therefore an updated computer with well performing software could reduce these levels. There may be some software usability issues, which were not addressed in this research. However, all five stages at some point reported very high workload levels (>65). A larger sample of both; workload and associated errors would clarify the relationship between workload and error rates over a longer period of time.

ACKNOWLEDGMENTS

The research was supported by a grant from the Health Research Board (HRB) of Ireland, reference “C.A. 808 – Data Collection for the Radiation Oncology Systems Safety Analysis”. The authors would like to acknowledge the support of the Radiotherapy and Medical Physics staff at the Galway University Hospitals (GUH).

REFERENCES

Bogdanich, W. (2010). “Radiation offers new cures, and ways to do harm”. The Radiation Boom. The New York Times, accessed on 11 February 2014, available at: http://www.nytimes.com/2010/01/24/health/24radiation.html?_r=0 Cooper, G. E., Harper, R.P. (1969), “The use of pilot rating in the evaluation of aircraft handling qualities”. No. AGARD-567.

Advisory Group for Aerospace Research and Development Nueilly-Sue-Seine (France), 1969. Ford, E.C., Fong de Los Santos, L., Pawlicki, T., Sutlief, S., & Dunscombe, P. (2012), “Consensus recommendations for incident learning database structures in radiation oncology”. Medical physics, 39(12), 7272 Galičič, M, Fallon, E, van der Putten, W, Sands, G. (2013), “Applying NASA-TLX to Prostate Seeds Brachytherapy”, Patient &

Healthcare Provider Safety, London, November 2013 Hart, S.G. & Staveland, L.E. (1988), “Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research”, in: Hancock, P.A. and Meshkati, N. (Eds.) Human Mental Workload (pp.139-183). Amsterdam: North-Holland Jex, H.R. (1988), “Measuring mental workload: Problems, progress, and promises”. In Hancock, P.A. and Meshkati, N. (Eds.)

Human Mental Workload (pp.5-39). Amsterdam: North-Holland Luximon A., Goonetilleke, R.S. (2001), “Simplified subjective workload assessment technique”. Ergonomics. 2001 Feb 20;44(3):229-43. Mosaly, P.R., Mazur, L.M., Jackson, M., Chang, S.X., Deschesne Burkhardt, K., Jones, E.L., Xu, J., Rockwell, J., Marks, B.L.

(2011), “Empirical Evaluation of Workload of the Radiation Oncology Physicist During Radiation Treatment Planning and Delivery”. Proceedings of the Human Factors and Ergonomics Society Annual Meeting September 2011 vol. 55: 753- 757 Mazur, L.M., Mosaly, P.R., Jackson, M., Chang, S. X., Deschesne Burkhardt, K.D., Adams, R. D., Jones, E. L., Hoyle, L., Xu, J., Rockwell, J., Marks, B. M. (2012), “Quantitative Assessment of Workload and Stressors in Clinical Radiation Oncology”, International Journal of Radiation Oncology*Biology*Physics, Volume 83, Issue 5, 1 August 2012, Pages e571–e576 Mazur, L M., Mosaly, P. R., Hoyle, L.M., Jones, E.L., Chera, B.S., Marks, L.B. (2014), "Relating physician’s workload with

errors during radiation therapy planning". Practical Radiation Oncology Volume 4, Issue 2, March–April 2014, Pages 71– 75 Reid, G. B., Shingledecker, C.A., Eggemeier, T. (1981), "Application of conjoint measurement to workload scale development”.

Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Vol. 25. No. 1. Sage Publications, 1981. Satterfield, K. et al. (2012), "Measuring workload during a dynamic supervisory control task using cerebral blood flow velocity and the NASA-TLX.", Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Vol. 56. No. 1. Sage Publications, 2012. Vidullch, M. A., Ward, F .G., Schueren, J. (1991), “Using the Subjective Workload Dominance (SWORD) Technique for

projective workload assessment”. Human Factors, 22, 677-691 World Health Organization (WHO) (2008), “Radiotherapy risk profile: Technical manual”. Geneva, Switzerland: WHO publishing; 2008.

Cover of the proceedings volume Advances in Human Aspects of Healthcare

Explore the complete digital volume

Advances in Human Aspects of Healthcare

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-4951-2093-0

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →