Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Expanding Technology and Human Readiness Levels: Measuring the Maturity of New Learning Technology and Content

1 Applied Research Laboratories: The University of Texas at Austin, Austin, TX 78758 USA ↗
2 U.S. Army Combat Capabilities Development Command-Soldier Center, Simulation & Training Technology Center, Orlando, FL 32816, USA ↗
3 Alaka’ina Foundation, Orlando, FL 32826, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1006271

Abstract

According to the most recent research on the effectiveness of the US Army's institutional training and education, only fifty percent of what is taught is actually transferred to the learner's later real-work activities and competence. This finding is consistent with other independent research conducted for the US Navy which begs the question of not only whether the methods we use in our institutions are effective but whether the technology and content designed to support learning is a root-cause of this low transfer in general. Today the means of measuring and accepting military training simulations, stimulations and learning systems, as well as other forms of learning content, is purely functional. During current learning system and/or content verification and validation, there is no focus or time spent on testing whether or not the target learning outcomes actually occur. This requires a form of measurement standard that is informed with instrumented, objective, and inspectable data that Army acquisition professionals, training and education developers and decision-makers can use to discern and measure a system's learning impact before making final procurement, design or implementation milestone decisions. This standard and its associated instruments could also help learning researchers and engineering teams gauge the maturity progress of their results during formative testing.This paper will describe and discuss what we term Learning Readiness Measures (LRM) that could inform such a standard, and the required minimum associated data instruments needed to provide the data evidence required to support such a standard. The LRMs are designed to expand on the already ubiquitous Technology Readiness Levels used by DoD and government, and the recent published Human Readiness Levels expansion-standard to assess acquisition and implementation risk. In addition, this paper will discuss the larger learning engineering process within which this standard and its instruments could be used to help make data-informed investigations, as part of creating learning technology and content, that would enable more informed training/education implementation decisions. The learning engineering approach uses a systemic process that iterates three distinct phases (1) the creation of learning stimulation engines, content and/or feedback, (2) the implementation that produces the inspectable data-evidence via naturalistic measures of human psychomotor and/or cognitive indicators associated with a competency being learned, and (3) investigating and analyzing the collected data to determine how well target users can not only intuitively use or understand the learning technology or content - with minimal preliminary instruction - but actually develops the targeted long-term behavior or affective changes the products are intended to produce.We will also briefly discuss the recommended minimal instruments needed to collect data on user naturalistic and involuntary responses to learning stimulus. These include measurement capability that measures factors such as: cognitive recognition speed and accuracy, cognitive attention processes, cognitive load (too many sensory inputs to track or correlate), stress from not understanding stimulus and finding navigation paths to a feature or function, gross and fine motor activity while performing a task, internal user temperature within the environment performed in, verbal responses and/or inquiry during technology operation and emotional state or intensity.Our premise is that the LRMs and associated instruments will improve learning system evaluation and quality, help improve the low transfer of institutional training and education, but also reduce DoD or Government related project risks when making investments in researching and/or procuring technology-based learning systems and content. Such capability will also assist teams involved in a learning engineering process to gauge their progress toward an effective learning solution and to address challenges faced by DoD or Government in producing more ready forces and/or a more productive future workforce.

INTRODUCTION

According to research on the effectiveness of the US Army’s institutional training and education, only fifty percent of what is taught in learning institutions is actually transferred or reflected in the learner’s later real-work

© 2025. Published by AHFE Open Access. All rights reserved. 68

Expanding Technology and Human Readiness Levels 69

activities and abilities (Bickley et al., 2010). This begs the questions of not only whether methods used in Army institutions are effective but whether systems and/or technologies used to sustain such training is a root-cause of this low transfer. Today there are many training technologies and systems to conduct Soldier training in the field. How these training technologies and systems are tested for how well they connect to and support training, and produce maximum learning outcomes, is not well examined. Based on experience, we argue the means of measuring the usability, maturity, and effectiveness of US Army training technologies, learning systems, and other forms of learning content during and post development are mainly functional. While measures such as user touch-points and other forms of Soldier inputs are sought, what isn’t determined is how well the training technology or system is supposed to support learning. In addition, new training technologies rarely provide options to collect data produced by learners showing how well the Soldier’s knowledge or behaviors are affected. Such data is critical to show how much learners improved compared to standards as a result of using the product, and to measure transfer in later measures.

We propose that much of these fits within a larger recognized practice of fielding products without sufficient maturity as cited in a recent US Government Accountability Office (GAO) report, where it noted “...that while the Army has applied some leading practices for technological development in its equipment modernisation efforts, its weapon systems development is at a lower level of maturity than the GAO had advised.” (GAO, 2019 p.17). A more recent GAO report on Actions Needed to Support Fielding New Equipment noted, “The Army has fielded new equipment to the first units before completing some key planning element requirements...” (GAO, 2024 p. 20). While these occurrences may be based on real-time necessity, it’s not a best practice to ensure maximum learning occurs consistently and based on a standard.

Here-in we propose a rubric for the US Army in support of its acquisition programs, technology research, and its vendors of simulation and training technology, that extends the existing technology readiness level (Manning, 2023) and human-factors readiness levels (HFES, 2021), to include the maturity of new learning technology and systems called Learning Readiness Measures (LRMs).

BACKGROUND

The primary purpose of using readiness levels is to help engineering project teams in making decisions concerning the development and transitioning of technology. In the vernacular of the US defense department adaptive acquisition framework (DOD, 2020), these levels are used in the technology maturation and risk reduction phase, which culminates in a decision to move forward in what it calls “Milestone B” or a readiness to develop. Readiness levels are one of several tools and methods used to manage the progress of research and development maturation activities within the DOD. At the same time, we propose that learning technology and systems need to be continuously monitored even after development and fielding; to ensure,

over time, the products remain capable of meeting requirements as new environments, scenarios, or tasks not considered during the product’s critical design process. For learning technology, this means how well the transfer of learning occurs to a real job-site, which for the US Army could mean combat.

The logic of LRMs is that they work as an extension to the other existing technology and human readiness instruments. In order for a learning technology to be useful, it must first work reliably and perform functionally on-demand and as required; this is the role of the technology readiness levels. Next, human readiness levels ensure that learning technology must be usable and compatible with the learner and their work environment, meaning how they interface with the technology or how its information is consumed must be intuitive to the targeted user profile(s). A key outcome for learning that human readiness brings is that it ensures learners spend their limited cognitive resources on learning, and not trying to translate presented information, navigating where to go or determining what they need to do next. The interaction of these two types of readiness levels is a synergy where technology reliably facilitates learning, rather than hindering it with usability challenges, and adaptively enhances the outcomes of the individual or the team of Soldiers, as well as facilitates training decisions.

Finally, learning principles and practices must be implemented to maximize the learning outcome. This falls in line with the developing practice of learning engineering (Goodell & Kolodner, 2022), which uses a systemic process that iterates through the same methods used in other system engineering activities. The LRMs can be thought of as a way to integrate learning engineering into the defense training technology and system engineering process.

To determine what should be included in the LRMs, we considered the learning science that provides the elements to create an effective learning experience (e.g., National Research Council, 2000; The National Academies of Sciences, Engineering, Medicine, 2018). Some core requirements that learning technology or systems must have capability to support include: Promote integrating prior knowledge and skill levels. Technology integrates and supports developing new knowledge, skills and abilities on top of past learned knowledge, skills and abilities. Integration is when a new training technology is adapting to a learner’s new KSAs that were acquired from past training during institutional learning or on the job experience. Adaptive learning technology is critical in this respect. This is especially critical for improving the learning transfer problem cited above.

Promote demonstration of knowledge, skill, attitude, or ability to learn. Learners need examples to reference when actively attempting to replicate what is to be learned. We also learn through stories. This means providing vicarious demonstrations of tasks to learn, which can be live or recorded but should be done by a credible person who can competently and confidently demonstrate in a real-world context, and in real-time. Key is that the demonstration of the concept or procedure applies or fits within the work context and environment to which it will be transferred.

Expanding Technology and Human Readiness Levels 71

Promote and support active performance of targeted tasks, skills, knowledge and/or abilities. Instead of providing content to read, listen to or rotely consume, technology and systems must immerse learners within real problem-centered stimulus relevant to a real-world task or problem the learner will or may face in their future assignment. They must be able to explore the problem in their own state of expertise. This could be real case- study review and application, simulation content that emulates real systems and environments they may perform in, and should use terms, symbology, tools and practices that are currently in use in the actual job environment. The training stimuli should match the duties and activities to which learning will be transferred.

Promote and support spaced recall and practice to ensure the memory schema’s pathways from previous learning experiences are strengthened. Spacing the recall / application of learning at points in time when memory recall naturally degrades actually makes it stronger. In addition to this reinforcement over time, by changing the conditions and/or context of a learning stimulus from how it was previously experienced, will also enable more tacit or generalizable recall of the learned topic or procedure. In this way, spaced learning could include requiring learners to recall what was learned in previous training but under different conditions or context, to promote learning transfer; the key is an awareness of the learner’s prior training.

Record and produce collectable data. Another capability that modern learning technology and systems need is to produce evidence-based feedback, to include past performance, preferably through live recordings, so learners can understand when, where, and why they did or did not improve. This data-informed feedback can also support later reflection on performance, more objective and unbiased assessment, and can support objective feedback to the learning engineering process on improvements needed in the content or technology. This data-informed feedback can also be accessed for near real time individual or team feedback in an after-action review (AAR) format, where instructors, observer controllers, or assessors can guide the team in a discussion of detection (referencing the recorded data points), reflection, and self-correction (Smith-Jentsch et al., 2008; Townsend et al., 2018).

DISCUSSION

Learning Readiness Measures From the requirements noted above, and as a means of integrating LRMs within the general framework of the existing 9 levels of the technology and human readiness levels, the following maturity rubrics are proposed in Table 1 below.

Table 1: Recommended learning readiness measurements.

Table available in the original PDF paper.

9 Learning system is actively used within institutions, training

facilities or on-the-job activity. Produces data that enables long-term learning transfer evaluations. System and anonymous learner data is available to refine standards, methods, content, features. 8 Learning system is pilot tested in target learning environments with short-term learning data available to show satisfactory attainment of learning standards, across several target learner profiles. Data is made available to refine standards, methods, content, features prior to fielding. 7 Learning system developed and beta tested in simulated

target learning environments with short-term learning data available to show satisfactory attainment of learning standards, across several target learner profiles. Data is made available to refine standards, methods, content, features prior to pilot testing. 6 Learning system developed and alpha tested in laboratory with short-term learning data available to show incorporation of learning standards, concept and principles. Data is made available to refine standards, methods, content, features prior to beta testing. 5 Learning concept demonstrated in a laboratory environment

with representative target standards, methods, content, features and learner profile. Data shows how the in-field measures of short-term learning are objective, unbiased and with inspectable, collectable data output that is made meaningful to the learner. 4 Learning concept demonstrated in a laboratory environment with a selected set of standards, methods, content, features and a targeted learner profile. Data collected from instruments measure satisfactory physical and cognitive factors that enable maximum learning. 3 Concept of adaptive learning developed based on Level 2

learning science- based principles and best practices. Targeted subject matter topics and standards defined from Level 1 analysis. Target learner profiles and their learning environment/context captured. Long-term learning evaluation defined. 2 Learning science principles, practices and long-term learning standards captured and defined from data gathered from an on-the-job performance environment. Data is critically analyzed and labeled by certified competent practitioners and/or analysis information available in publications. 1 Target learning tasks, skills, knowledge and abilities are

observed, measured, and raw inspectable data collected in an on-the-job performance environment.

Expanding Technology and Human Readiness Levels 73

Figure 1: Technology, human and learning readiness levels.
Original PDF Figure 1:

When combined with the Technology and Human Readiness Levels, a learning technology or system’s maturity would need to be assessed through as is, an example of the overall evaluation matrix is shown in Figure 1.

Learning Measurement Instrumentation A key element of the LRMs is that in level 4 and optimally in level 5, we introduce the idea of employing learning measurement instruments to measure a technology-based learning experience, specifically its instructional efficiency (Van Gog & Paas, 2008).

Previous research has shown that various learning physiology attributes can be measured using modern instrumentation (Giannakos et al., 2020). Such measurements can reveal important learning information beyond the common measures of outcome such as accuracy, precision, latency, or errors made. Combinations of behavioral, cognitive, and emotional measures can provide information concerning the relative learning process, and both the effect and affect of a learning experience.

Based on this idea, studies have focused on maximizing cognitive load factors competing with the human learning processes (Van Merrienboer & Sweller, 2005) as well as “orchestration load”, which is the coordinating of technology-based, classroom learning activities and learning processes (Prieto et al., 2017). The idea is to measure the degree of mental effort invested in learning and test performance by both the learner and the expert instructor. By using instruments, learning scientists and engineers can determine what stimulus formats minimize a learner’s non-effective cognitive effort called extraneous cognitive load and which formats maximize the cognitive effort that fosters learning called germane cognitive load. Examples of germane cognitive load could be allowing learners to focus more attention on the learning problem state than navigating the technologies in a user interface; or providing learners useful solution steps or on-demand tutoring based on their need or competence profile. Other examples of extraneous cognitive load may be providing a limited source of information media and

using terms or examples that are less intuitive or well understood by the target learner profiles (e.g., a generation or culture with different experiences or vernacular). These are factors that would benefit from being measured.

Research has also revealed physiological data that can detect the quality of a learning experience with good accuracy - e.g., heart-rate, electrodermal activity, skin temperature, and blood volume pressure (Giannakos et al., 2020). In addition, research has found that by using pupillometry through eye-tracking instruments, both pupil response and fixation patterns during a learning experience can predict later recognition memory strength (Kafkas & Montaldi, 2011). Finally, research on the effect of performance degradation under pressure shows that by increasing and tracking the experience a learner gains when performing specific tasks, their cognitive load and anxiety will be reduced through explicit and implicit (tacit) learning (Wilson et al., 2009). All of these forms of data can inform an optimum learning experience.

We recommend the inclusion of instruments in LRM assessments, especially during testing maturity levels, to determine how learning-efficient (the process) and effective (the outcome) a technology’s learning experience is. This should be based on a targeted learner’s involuntary physiological responses to learning stimulus before investing in developing or fielding that technology. In this way, the LRMs and associated instruments will improve learning system evaluation and quality and help improve the transfer of institutional training by providing learners with self-management and metacognitive data.

Figure 2: Examples of learning measurement instruments.
Original PDF Figure 2:

When selecting and employing instrumentation in the LRMs they should not interfere with the learning process. Based on cited research, we recommend instruments like those shown in Figure 2 to be considered but designed such that the learner is barely aware they exist (other than awareness and/or consent they are to be measured). Additionally, instruments should not produce additional physical discomfort such as adding considerable

Expanding Technology and Human Readiness Levels 75

weight or inducing cognitive loads that may distract the learner’s focus. At the same time, instruments should be selected that can reliably provide the physiological data that reveals the learning factors discussed above. Table 2 shows an example of such an instrument analysis.

Table 2: Learning measurement instrument analysis.

Table available in the original PDF paper.

Instrument Measurement Capability Eye tracking / Pupillometry

- Ability to measure detections of persons or objects of interest in the battlespace.

device ((eye tracker - NEW)

- Measure pupil involuntary response to stimulus Heart-Rate Variability

Ability to measure stress from workload or not having

sensor (Emotibit - NEW)

needed knowledge or skill Galvanic Skin Response

Ability to measure stress from workload or not having

sensor (Emotibit - NEW)

- Ability to measure the classification of objects and

needed knowledge or skill First-person video / audio

personnel as threats or as mission-objective related

recorders (eye tracker - NEW)

- Ability to measure communications and reports to leadership and peers - Ability to provide rapid

contextual information in context to a specific cognitive or behavioral activity Second/third-person video /

- Ability to measure cognitive decisions and resulting

audio recorders (GoPro)

behaviors - Ability to measure behavioral data using markerless motion capture. - Ability to measure

coordinated interactions with team members GPS receiver and recorder Positional / coordinated movement in a live environment Software to synchronize / Ability to provide data-informed feedback on previous

analyze data (NEW)

performance and determine the cognitive and behavioral response to experiential stimulus.

Learning Readiness Measurement Implementation In addition to incorporating data from hardware measurement instrumentation, we propose the need for sufficient software and data architecture tools and storage to properly implement the LRMs in a systematic verification and validation process. This includes being able to

control learner stimulus during testing, adapting stimulus based on a learner’s affective measures, assessing and collecting the learner’s activity data. and calculating and producing the learning outcome data in a standardized format. Each of these capabilities can support a phase in the overarching learning engineering process.

Figure 3: Learning readiness measurement architecture and tools.
Original PDF Figure 3:

As shown in Figure 3, what we suggest using is an adaptive learning system (ALS) that is capable of consuming and hosting occupational expert competencies and standards that are to be learned through a new learning technology. The ALS can also ingest the learner physiologic data as well as video, audio and other behavioral data (e.g., GPS), and can use that data to classify a learner’s physical, affective and cognitive state, as well as show the measured results to engineers in real-time or as part of play-back analysis after a recorded learning experience. We feel this type of architecture is critical, especially during pre-decisional phases of a project’s development cycle, such as during learning science and technology research, as well as during early learning technology production testing.

CONCLUSION

Readiness measurements allow project managers, designers and engineers to understand and assess how much more development a particular technology needs before being effectively and efficiently useful for the user. A primary objective of system and learning engineering is to gain sufficient technical, behavioral, physiological and cognitive ability efficiency, and to develop a program’s System Requirements Document (SRD) that is based on inspectable and referrable data. LRMs will help to verify that the learning solution(s) required technology and methods are sufficiently mature at a Readiness level 6 or above before proceeding into an end-item design and

Expanding Technology and Human Readiness Levels 77

development or what is formally called “Milestone B”in the DoD acquisition vernacular. LRMs are intended to add an additional dimension to the existing Technology Readiness Levels and Human Readiness Levels when a project is focused on learning technology and systems.

In addition to LRMs, in order to provide the learning engineering team evidence that specific levels are achieved, instrumentation is required to demonstrate a technology’s maturity in producing or facilitating applicable and useful human learning responses, and that learning transfer to Army field activities is maximized. Finally, we proposed employing an LMS architecture that will facilitate a systematic learning engineering process that not only helps assess if target competency standards are attainable but will produce the learning process and outcome data as evidence of learning efficiency and effectiveness for decision makers to consider.

REFERENCES

Bickley, W., Pleban, R., Diedrich, F., Sidman, J., Semmens, R., Geyer, A. (2010). Army Institutional Training: Current Status and Future Research. US Army Research Institute. Giannakos, M., Sharma, K., Papavlasopoulou, S., Pappas, I. (2020). Fitbit for

learning: Towards capturing the learning experience using wearable sensing. International Journal of Human-Computer Studies, 136, 102384. Goodell, J., & Kolodner, J. (Eds.). (2022). Learning Engineering Toolkit: Evidence- Based Practices from the Learning Sciences, Instructional Design, and Beyond (1st ed.). Routledge. https://doi.org/10.4324/9781003276579 Government Accounting Office (GAO) (2019). Army Readiness: Progress and

Challenges in Rebuilding Personnel, Equipping, and Training (GAO-19-367T). US Government Accounting Office. https://www.gao.gov/products/gao-19-367t Government Accounting Office (GAO) (2024). ARMY MODERNIZATION: Actions Needed to Support Fielding New Equipment (GAO-24-107566). US Government Accounting Office. https://www.gao.gov/products/gao-24-107566 HFES (2021). Human Readiness Level Scale in the System Development Process (ANSI/HFES 400 2021). https://my.hfes.org/online store/publications Kafkas, A.; Montaldi, D. (2011). “Recognition memory strength is predicted by

pupillary responses at encoding while fixation patterns distinguish re collection from familiarity”. The Quarterly Journal of Experimental Psychology. 64 (10): 1971 1989. Kafkas, A.; Montaldi, D. (2012). Familiarity and recollection produce distinct eye movement and pupil and medial temporal lobe responses when memory strength is matched. Neuropsychologia. 50 (13): 3080 93. Manning, C. G. (2023). Technology Readiness Levels - NASA. NASA. https://

www.nasa.gov/directorates/somd/space-communications-navigation-program/ technology-readiness-levels/ Merrill, M. D. (2009). First Principles of Instruction. In C. M. Reigeluth & A. Carr (Eds.), Instructional Design Theories and Models: Building a Common Knowledge Base (Vol. III). New York: Routledge Publishers. National Research Council (2000). How People Learn: Brain, Mind, Experience,

and School: Expanded Edition. Washington, DC: The National Academies Press. https://doi.org/10.17226/9853

National Academies of Sciences, Engineering, and Medicine (2018). How People Learn II: Learners, Contexts, and Cultures. Washington, DC: The National Academies Press. https://doi.org/10.17226/24783. Department of Defense (2020). DOD Instruction 5000.02 Operation of the Adaptive

Acquisition Framework. Office of the Under Secretary of Defense for Acquisition and Sustainment. Prieto, L. P., Sharma, K., Kidzinski, Ł., Dillenbourg, P., (2017). Orchestration load indicators and patterns: In-the-wild studies using mobile eye-tracking. IEEE Transactions on Learning Technologies 11 (2), 216–229. Schmidt, M., Tawfik, A., Earnshaw, Y., and Jahnke, I. (2019). Learner and user

experience research: An introduction for the field of learning design & technology. Available at: https://edtechbooks.org/ux. Smith-Jentsch, K. A., Cannon-Bowers, J. A., Tannenbaum, S. I., & Salas, E. (2008).

Guided team self-correction impacts on team mental models, processes, and effectiveness. Small Group Research, 39(3), 303–327. Townsend, L., Johnston, J., Ross, W. A., Milham, L., Riddle, D., and Phillips, H.

(2018). An integrated after-action review (AAR) approach: Conducting AARs for scenario-based training across multiple and distinct skill areas In S. Harris (ed.), Engineering Psychology and Cognitive Ergonomics (pp 230-240, Vol. 10906). Cham: Springer International Publishing. doi: 10.1007/978-3-319-91122-9_20. Van Merrienboer, J. & Sweller, J. (2005). Cognitive Load Theory and Complex Learning: Recent Developments and Future Directions. Educational Psychology Review, Vol. 17, No. 2, June 2005. Van Gog, T., Paas, F. (2008). Instructional efficiency: Revisiting the original construct

in educational research. Educ, Psychol, 43, 16–26. Wilson, N., Sun, R., Mathews, R. (2009). A motivationally-based simulation of performance degradation under pressure. Neural Networks, 22, 50250.

Cover of the proceedings volume Human Dynamics, Product Evaluation and Quality

Explore the complete digital volume

Human Dynamics, Product Evaluation and Quality

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-48-9

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →