Anne Sinatra, Lisa Townsend, Benjamin Goldberg, Paige Lawton (2025). Examples and Lessons Learned Utilizing the Generalized Intelligent Framework for Tutoring (GIFT). In: Human Factors and Simulation. AHFE Open Access, vol. 180, pp. 43–51. https://doi.org/10.54941/ahfe1006387
The Generalized Intelligent Framework for Tutoring (GIFT) is an open-source intelligent tutoring system (ITS) framework that can be used to create ITSs in any topic area (Goldberg & Sinatra, 2023). Flexibility is a large part of GIFT’s design, and there are many different ways that it can and has been used. In our discussion as part of the “SIG: Intelligent Tutoring Systems and Generalized Intelligent Framework for Tutoring-GIFT: Applicability to Industry, Academia, and Government” panel, and in the current paper, we focus on two different examples of using GIFT, and the lessons learned. These examples and lessons learned provide insight into how GIFT has been used, and how others could use it for their own work. The two use cases to be discussed are relevant to Human Factors professionals and include using GIFT to create and provide online adaptive lessons prior to the beginning of a class and using GIFT for real-time assessment during a data collection. In the first example, GIFT was used to create adaptive courses which were provided to students online prior to an in-person class. The discussion focuses on the features of GIFT that were used, and the type of adaptations that GIFT made. Further, lessons learned from defining the associated GIFT course concepts, and designing the GIFT courses are discussed. In the second example, GIFT was used for real time assessment during a data collection effort utilizing a handheld tablet to readily and easily track team performance over time. The data outputs were then utilized to guide After Action Reviews following the scenarios. Our panel discussion will showcase different ways that GIFT can and has been used, and provide SIG participants the opportunity to ask questions that are relevant to their own work.
The Generalized Intelligent Framework for Tutoring (GIFT) is a service- oriented architecture for creating intelligent tutoring systems (ITSs). ITSs provide opportunities to tailor learning experiences in real-time to an individual learner and adapt content based on learner performance. GIFT is an open-source ITS architecture that has been developed with multiple goals in mind including supporting ITS research, making it easier to author ITSs,
and facilitating guided tutoring in experiential learning. The first version of GIFT was released to the public in May of 2012, and over the last decade the goals and functionality of GIFT have continued to evolve (Goldberg & Sinatra, 2023; Sottilare et al., 2017). GIFT’s authoring tools are flexible, and allow for an adaptive tutor to be constructed from existing materials (e.g., PowerPoints, question banks, surveys), or to interface with simulation- based training environments (e.g., Virtual Battle Space, Unity, etc.). Beyond delivery of adaptive learning materials, ITS frameworks such as GIFT are highly relevant to the Human Factors community as they can be leveraged for research in the fields of cognitive modeling, augmented cognition, human robot interaction, and intelligent interaction design.
The SIG Session and Panel The current paper and discussion are associated with a panel aligned to the Special Interest Group (SIG) titled “Intelligent Tutoring Systems and Generalized Intelligent Framework for Tutoring-GIFT: Applicability to Industry, Academia, and Government”session at the Applied Human Factors and Ergonomics 2025 conference. This panel discussion is designed to inform and engage conference participants in exploring GIFT’s tools and methods, with goals of demonstrating and discussing ways that the GIFT software has been used and providing a forum for others in the research community to ask questions about how they themselves could utilize it.
The panel of current GIFT users spans academia, industry, and government. The panel members will discuss a domain use case, the unique ways they utilized GIFT, and the associated lessons learned from their projects. Use cases will spotlight GIFT applied for online and real time data collection, applying GIFT for experiential learning, integrating an external simulation with GIFT, and enhancing STEM education through GIFT. The SIG session will culminate in a demonstration within GIFT that is selected based on the input of the session participants.
The Generalized Intelligent Framework for Tutoring GIFT is a highly modular architecture that provides users an ability to create a GIFT course that can be configured to operate linearly or to be adaptive. A linear course always presents the same information in exactly the same way to each learner, and can be leveraged for traditional experiments (Sinatra, 2014; Sinatra, 2018). Adaptive courses can adapt based on learner performance. There are two approaches to creating an adaptive course in GIFT.The first approach is to use the Adaptive Courseflow Object as part of a GIFT course. The Adaptive Courseflow Object follows Merrill’s Component Display Theory (1987), and the student goes through Rules, Example, Recall, and Practice phases on defined course concepts (Goldberg, Hoffman & Tarr, 2015). The second approach is to configure a GIFT domain knowledge file (DKF) which connects to an external training application (e.g., a simulation), and can assess learner performance in real-time. Performance ratings can also be added by an Observer using the GIFT Game Master interface (Goldberg, Hoffman & Graesser, 2020).
Examples and Lessons Learned Utilizing the Generalized Intelligent Framework 45
The Current Paper In the current paper we discuss two panel topics that represent examples of ways that GIFT has been leveraged for research: the first example is GIFT providing adaptive tutoring in an online data collection environment, and the second example is GIFT’s tools being leveraged to support data collection at an in-person event. These examples provide opportunities for researchers to see how GIFT has been used.
EXAMPLE 1: USING GIFT FOR AN ONLINE DATA COLLECTION GIFT was used to create adaptive courses about the Detect, Engage, and Assess elements of the DIDEA (Detect, Identify, Decide, Engage, Assess) direct fire engagement process. These GIFT courses were made available to students online prior to the start of an in-person MCOE (Maneuver Center of Excellence) Master Gunner class (Sinatra et al., 2023), and using the GIFT courses was optional. There was a pre-test and post-test administered by GIFT, and seven individual adaptive GIFT courses. These GIFT courses were accessed through logging into GIFT using a provided username. The use of usernames occurred so that the accounts would include all of the relevant GIFT courses in the correct order, and so that user provided information could be stored in the system.
The adaptivity within the GIFT courses used the Adaptive Courseflow Object. GIFT course concepts were identified for each of the lessons, and concepts were assigned to specific adaptive courseflows based on the specific content. When in the adaptive courseflows, materials that were originally in PowerPoint slide form were converted to GIFT Slide Shows for use as presented content, and multiple choice questions were included for the recall phase to determine if the learner was proficient on the concept. If the student did not meet the threshold for proficiency on the recall phase, he or she was presented with additional content about the concepts missed, and then went through the recall phase again. The students would go through this process until they had reached proficiency on all of the concepts within the adaptive courseflow. The students had the ability to select the courses that they would like to complete, and after they completed all the courses they were asked to take a post-test. The data from the students were extracted, and information such as the number of times remediation was provided, and if proficiency was met on concepts was determined.
This provides a use case of how GIFT can be used to provide personalized multi-media learning interactions. All data collection was remote through GIFT’s Cloud instance, and students interacted with course materials on their own computers or devices. This did not require a download, and the GIFT course author was able to use existing materials such as PowerPoints to populate the adaptive courseflow.
Lessons Learned One of the most important lessons learned from this experience was regarding approaches to extracting data from adaptive courses and how to understand it. This required extracting the data from GIFT, examining
the data and the outputs to understand the formatting, and understanding how the data outputs related to the adaptive courseflow structure. After examining the data outputs, the best way to organize the data for clarity and to be human readable was determined. A similar illustrative example of this type of adaptive courseflow data organization is further discussed in Sinatra (2023).
The second lesson learned from this experience was that it is important to consider the user experience when determining the number of course concepts included in the adaptive courseflows. Some of the GIFT courses included single adaptive courseflows that covered many course concepts (e.g., 1 adaptive courseflow with 4 or more concepts), while others included more adaptive courseflows that covered less topics (e.g., 2 adaptive courseflows with 2 concepts each). From examining the user data, it became clear that in some cases the learners never met the GIFT defined proficiency on a concept, which indicates that they left the GIFT course without fully completing the adaptive courseflow object. This may have happened because there were a large number of concepts associated with a single adaptive courseflow which could have resulted in the student going through remediation many times. By having more adaptive courseflow objects that cover less concepts it reduces the chances of a student essentially getting “stuck” in a single adaptive courseflow, which could become frustrating from a student perspective. Therefore, it is important to consider the structure of the GIFT course and the impact the structure may have on the student completing the course.
EXAMPLE 2: USING GIFT GAME MASTER FOR REAL-TIME DATA COLLECTION AND ASSESSMENT In the second example, GIFT was used for real time assessment during a data collection effort utilizing a handheld tablet for Observers to readily and easily track and assess team performance over time. Attendees at the 2023 and 2024 Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) participated in a workshop entitled “From the Last of Us to the First of Us: Rebuilding After a Zombie Crisis” (Townsend et al., 2024). The workshop consisted of a crisis-management game that was a hybrid of a Tabletop Role Playing Game (TTRPG) with LARP (Live Action Role-Playing) elements using Zombies as the antagonist during an infrastructure collapse. The game objectives were for participants to select one of four locations and work as a team to meet multiple objectives while Observers assessed their performance.
One challenge the Observers encountered during 2023 was the ability to track behaviors effectively during gameplay. The observational checklists used were paper based and using the sheets became confusing as participants moved around, and anticipated behaviors occurred at different times than planned. In 2024, a computer tablet utilizing GIFT Game Master was used to input whether expected behaviors were observed or not and how well
Examples and Lessons Learned Utilizing the Generalized Intelligent Framework 47
they were performed. Using a tablet and collecting data electronically is a method that has proven more efficient in observable behavior data collection efforts (Townsend et al., 2017). GIFT Game Master was used to track, assess, and organize the data, and the data collected was easily pulled into a statistical program for analyses and interpretation. Expected behaviors were aligned to specific events and triggers at each of four locations – simplifying the process of what and when to observe. These expected behaviors followed the Targeted Acceptable Responses to Generated Events or Tasks (TARGETs) framework (Fowlkes et al., 1994). TARGETs is an event-based measurement technique that tracks performance linked to specific trigger events within training scenarios, providing opportunities for observing behaviors of interest. When triggers are linked to measures in this way, it provides a more objective approach to observing and assessing performance.
The GIFT Game Master interface was used to observe and assess observable expected behaviors aligned to one of four specific competencies:
Leadership
Supporting Behavior
Information Exchange
Mutual Trust Each of the four competencies included three expected behaviors across four different scenarios/weeks, with a specific task assigned each week:
Moving the Tree
Building Local Alliances and Sustenance/Connecting with Others
Hurricane
Sickness Prevails Each behavior was assessed as Poor, Satisfactory, or Exemplary, which aligned with Below Expectation, At Expectation, and Above Expectation in GIFT. Notes could be added using a “Create Note” button. A “Gesture Mode” could be used to make observations verbally using the tablet’s microphone. Notes taken were linked to behaviors that were traced on a timeline.
It is important to note that there has recently been a major redesign of the Game Master interface, which occurred after the data collection. See Figure 1 for what the Observer Assessment interaction screen interface looked like at the time of the data collection. In this previous interface, after signing in as a user with Observer permissions, the Observer was immediately taken to the Game Master and shown a “Create Session” button. This moved them to the Game Master assessment screen, where they could start to fill out assessments. The “Assignments” section on the left in Figure 1 was useful for participant IDs that could be assigned to Roles/Call Signs in the scenarios. Call signs identified the participant, while Role was an optional field that could be used to indicate what position was played (e.g. leader).
Figure 1: Observer assessment screen configured for the data collection in the previous game master interface.
To advance to the summary of assessments, after clicking on the Game Master and then Past Sessions, a list of all the sessions that had been assessed as the Observer (ordered from newest at the top to oldest at the bottom) appeared in the interface. The sessions each represented a single run through which was associated with a DKF execution. Clicking on any of them brought the user to an interface with a timeline. The timeline showed green and red indicators of how the students performed overall, specifically their “summative” or “final” assessment scores. Clicking the names of the competencies expanded them, and allowed the scores of the lower-level competencies to be seen. Red represented Below Expectation (or Poor), light green was At Expectation (or Satisfactory), and dark green with a star was Above Expectation (or Exemplary).
Seeing how assessments progress over time rather than just final scores simply required selecting the button to view the “formative” assessments. In the playback interface, pressing the play button or clicking in the timeline provided a playback of how the scores changed over time.
The playback interface included icons representing bookmarks (e.g., observed behaviors) that were recorded by the Observer. A bookmark was always produced at the beginning to track start time. More of these icons were also created when the Observer modified the Roles/Call Signs, which allowed for a view of how team composition changed over time.
Examples of the new Game Master interface are in Figures 2 and 3. The new design is intended to improve usability, and ease of use of the Game Master. Much of the functionality utilized for the described data collection is present in the new interface. Figure 2 shows an example of the Assessments and timeline in the new interface. Figure 3 shows an expanded view that breaks the Assessments down by their star ratings in the new interface.
Examples and Lessons Learned Utilizing the Generalized Intelligent Framework 49
Figure 2: Example of the new game master interface that shows assessments and the timeline.Figure 3: Example of the new game master interface that shows the breakdown of ratings for assessments.
While the specific example discussed in this section included Observers adding assessments directly into the Game Master interface, it is also important to note that automated assessments from an external training application can also be read into GIFT.
The above describes why and how GIFT Game Master was used for real time data collection. Using GIFT Game Master allowed the Observers to easily track behaviors over time, even if they occurred out of sequence. The Game Master dashboard provided insight on performance that was used to
inform an After Action Review (AAR). The level of effort in developing and building the observation and assessment in GIFT resulted in an impactful and near immediate return on investment.
Lessons Learned From Example 2 As noted above, the Game Master interface has recently been significantly updated. Therefore, some of the lessons learned from the example have already been addressed, or will be addressed soon.
The specific lessons learned/challenges associated with the second example of using GIFT are below:
1) Internet connectivity issues with accessing the Game Master for observation and assessment sometimes caused active sessions to disconnect mid assessing. It was easy to reestablish connectivity and navigate back to previous location, but it was tedious to have do so on a somewhat consistent basis. This issue was identified and is being addressed in updated versions of the Game Master. 2) Some of the buttons on the Game Master display were very small, making it difficult to make an intended first observation or assessment as an unintended button could inadvertently be selected by mistake. This is being addressed by increasing the size of those buttons and/or having them be customizable for user adaptability.
DEMONSTRATION As part of the SIG session, there will be a demonstration of GIFT. This will be audience driven and interactive. The audience will be able to select from different options that will represent different ways that GIFT can be used. The options will include those that align to the examples that have been described in the current paper, as well as other functionalities that exist in GIFT that are relevant to human factors professionals and their research.
CONCLUSION
GIFT provides an adaptable framework for creating ITSs that can be utilized for a variety of training needs across multiple domains by facilitating research, providing an easier ITS authoring experience, and facilitating tutoring. The SIG session presents an opportunity to demonstrate the flexibility of GIFT to human factors practitioners. The current paper represents one of the SIG panel topics by providing two use case examples and the lessons learned by utilizing GIFT for those use cases. The use cases include both online data collection and real-time data collection and assessment during an in-person training event. The demonstration of GIFT’s capabilities provides additional opportunities to showcase the different ways that GIFT can be utilized across those domains. Human factors professionals who attend the SIG session are anticipated to leave with an enhanced understanding of GIFT’s capabilities and how those capabilities might be applied in their own work.
Examples and Lessons Learned Utilizing the Generalized Intelligent Framework 51
ACKNOWLEDGMENT
The research described herein has been sponsored by the US Army Combat Capabilities Development Command – Soldier Center – Simulation and Training Technology Center. The statements and opinions expressed in this paper do not necessarily reflect the position or the policy of the United States Government, and no official endorsement should be inferred.
REFERENCES
Fowlkes, J. E., Lane, N. E., Salas, E., Franz, T., & Oser, R. (1994). Improving the measurement of team performance: The TARGETs methodology. Military Psychology, 6, 47–61. Goldberg, B., Hoffman, M., & Graesser, A. C. (2020). Adding a Human to the
Instructional System Loop: Integrating GIFT and Battle Space Visualization. Design Recommendations for Intelligent Tutoring Systems: Volume 8-Data Visualization, 191. Goldberg, B., Hoffman, M., & Tarr, R. (2015). Authoring Instructional Management Logic in GIFT Using the Engine for Management of Adaptive Pedagogy (EMAP). Design Recommendations for Intelligent Tutoring Systems: Authoring Tools and Expert Modeling Techniques, 319. Goldberg, B., & Sinatra, A. M. (2023). Generalized Intelligent Framework
for Tutoring (GIFT) SWOT Analysis. Design Recommendations for Intelligent Tutoring Systems: Volume 10-Strengths, Weaknesses, Opportunities and Threats (SWOT) Analysis of Intelligent Tutoring Systems, 9. Merrill, M. D. (1987). The new component design theory: Instructional design for courseware authoring. Instructional Science, 16(1), 19–34. Sinatra, A. M. (2014). The research psychologist’s guide to GIFT. In Proceedings of
the 2nd Annual GIFT Users Symposium (pp. 85–92). Sinatra, A. M. (2018, May). The 2018 research psychologist’s guide to GIFT. In Proceedings of the Sixth Annual GIFT Users Symposium (Vol. 6, p. 259). US Army Research Laboratory. Sinatra, A. M. (2023, July). Considerations for Developing a Generalized Gradebook
for an Open-Source Intelligent Tutoring System Framework. In International Conference on Human-Computer Interaction (pp. 162–173). Cham: Springer Nature Switzerland. Sinatra, A. M., Robinson, R. L., Goldberg, B., & Goodwin, G. (2023, September).
Impact of Engaging with Intelligent Tutoring System Lessons Prior to Class Start. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 67, No. 1, pp. 2262–2266). Sage CA: Los Angeles, CA: SAGE Publications. Sottilare, R. A., Brawner, K. W., Sinatra, A. M., & Johnston, J. H. (2017).
An updated concept for a Generalized Intelligent Framework for Tutoring (GIFT). GIFTtutoring. org, 1–19. Townsend, L. N., Johnston, G., Johnston, J. H., Mize, J, Griffith, T., & Engel, J.
(2024). Expanding Access to Learning Decision-Making and Teamwork Skills Using Low Fidelity, Tabletop Games: A Measurement Approach. Interservice/ Industry Training, Simulation, and Education Conference (I/ITSEC). Townsend, L., Johnston, J., Ross, B., Milham, L., Riddle, D., Phillips, H., & Woodhouse, B. (2017). Development of a mobile tool for dismounted squad team performance observations. In Virtual, Augmented and Mixed Reality: 9th International Conference, Proceedings 9 (pp. 312–321). Cham: Springer.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors and Simulation
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-56-4
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional