Arno Hartholt, Sharon Mozgai (2023). Creating Virtual Worlds with the Virtual Human Toolkit and the Rapid Integration & Development Environment. In: Intelligent Human Systems Integration (IHSI 2023): Integrating People and Intelligent Systems. AHFE Open Access, vol. 69, pp. 367–378. https://doi.org/10.54941/ahfe1002856
The research and development of virtual humans, and the virtual worlds they inhabit, is inherently complex, requiring interdisciplinary approaches that combine social sciences, computer science, design, art, production, and domain expertise. Our previous work in managing this complexity has resulted in the release of the Virtual Human Toolkit (VHToolkit), aimed at lowering the burden of creating embodied conversational agents. In our current efforts, we are integrating the VHToolkit with the Rapid Integration & Development Environment (RIDE), a rapid prototyping modeling and simulation middleware platform that leverages real-time game engines. This integration results in the ability to mix and match commercial AI services from AWS, Azure, and Google, as well as leverage novel 3D geospatial terrain creation pipelines. Combined with dedicated authoring tools that have been developed through human-centered design processes, the platform enables researchers, developers, and domain experts to rapidly create digital worlds with virtual humans for both military and civilian contexts. Our approach is highly interdisciplinary, including academia, government, and industry collaborators. The demonstration shows a user interacting with an embodied conversational agent embedded within real-world captured and virtualized terrain. Further research and development features of the platform are shown, including scripted agent behaviors, networked team play, and machine learning interfaces.
The ever-increasing pace of technological advances has led to a fascinating paradox. While the growing quantity of technologies increases the potential for creating more relevant, effective, and efficient solutions to today’s challen- ges, it also increases the required effort of realizing that potential due to the inherent combinatorial complexity of system integration. Furthermore, both the growing diversity and level of sophistication of technologies lead to an increase in specializations, which in turn leads to growing team sizes in order to provide broad coverages of all necessary disciplines, each of which with their own specialized background, language, and culture. Finally, in order to appropriately apply technologies to human needs, cognitive, physical,
Figure 1: The USC ICT Virtual Human Toolkit with example characters.
social, and organizational sciences are required, further exacerbating this interdisciplinary challenge.
Our work aims to address this complexity by creating approaches and platforms that combine novel research with robust development to enable interdisciplinary teams to design, develop, and deploy engaging, efficient, effective, human-centered, integrated systems.
In this demonstration, we provide an overview of our work-in-progress integration of two separate platforms – the Virtual Human Toolkit (VHToolkit)1 and the Rapid Integration & Development Environment (RIDE)2 – which enable researchers and developers to create virtual humans in virtual worlds.
VIRTUAL HUMAN TOOLKIT Virtual humans (VHs) are interactive, digital, embodied characters that perceive real humans and respond appropriately, both verbally and nonver- bally. They act as social interface agents that add a social component to the environments in which they are embedded. They provide a standardi- zed experience across users and can be omnipresent and indefatigable in their roles. ECAs have been shown to improve user’s perception of their environment (Johnson, Rickel, and Lester 2000), increase interaction time (Lane et al. 2011), and improve learning outcomes (Schroeder, Adesope, and Gilbert 2013). VH tools and platforms continue to grow in both number and capability (Hartholt and Mozgai 2022).
USC ICT has widely been recognized as one of the leaders in virtual human research and development, including basic research in cognitive architectures 1https://vhtoolkit.ict.usc.edu 2https://ride.ict.usc.edu
Creating Virtual Worlds With the Virtual Human Toolkit 369
Figure 2: RIDE showing terrain captured from Catalina Island.
(Rosenbloom, Demski, and Ustun 2016), audio-visual sensing (Scherer et al. 2012), and character animation simulation (Shapiro 2011), as well as applied prototypes for leadership development (Campbell et al. 2011), information dissemination (Rizzo et al. 2016), job interview training (Hartholt, Moz- gai, and Rizzo 2019), and life-long learning (Swartout et al. 2016). Virtual humans are excellent tools for exploring social sciences, including mora- lity, negotiation, and emotions (Neubauer et al. 2017) (Mozgai et al. 2017) (Neubauer et al. 2018) (Mell et al. 2018) (Chu et al. 2019) (Mell et al. 2020).
Our approach is highly interdisciplinary with a strong focus on integra- ting both theory and technology into common frameworks (Hartholt et al. 2009) (Devault et al. 2014). This has resulted in the Virtual Human Tool- kit (Figure 1), a collection of modules, tools, and libraries designed to aid and support researchers and developers with the creation of virtual human conversational characters (Hartholt et al. 2013). Recent efforts have resulted in the ability to develop virtual humans for a range of hardware platforms (Hartholt, et al. 2019; Hartholt et al. 2020), including web, mobile (Mozgai, Hartholt, and Rizzo 2020) (Mozgai, Akinyemi, et al. 2020), AR (Hartholt, Fast, et al. 2019), and VR (Gordon et al. 2019).
RAPID INTEGRATION & DEVELOPMENT ENVIRONMENT (RIDE) RIDE is a research & development platform that initially grew out of the US Army’s desire to prototype the next generation training and simulation system (Hartholt, Mccullough, et al. 2021a). As such, RIDE combines many simu- lation capabilities into a single framework, including synthetic real-world terrain, support for artificial intelligence (AI) and machine learning (ML) frameworks, networked multiplayer, Experience Application Programming Interface (xAPI) logging, Distributed Interactive Simulation (DIS) messa- ging, a unified web service interface, and multi-platform support. Many
370 Hartholt and Mozgai
Figure 3: The Virtual Human Toolkit architecture.
of these capabilities support rapid prototyping needs beyond the military, in particular for creating scenarios with multiple scripted agents embedded in synthesized terrain as a starting point to train more advanced behavior models. See Figure 2.
RIDE has been developed from the ground up to facilitate rapid prototy- ping specifically for simulation researchers and developers, following these guidelines:
Leverage real-time game engines for core capabilities (e.g., rendering).
Abstract away from specific game engines in order to provide simulation researchers and developers with the concepts they are most familiar with.
Provide a drag-and-drop development environment that offers reusable blueprints of commonly used functionalities.
Integrate into a common framework to add combinatorial value.
Offer all through a principled Application Programming Interface (API).
INTEGRATION & ARCHITECTURE Ongoing work integrates the VHToolkit with RIDE, revolving around arch- itecture, capabilities, and content. The VHToolkit architecture is modular, using a combination of message passing and direct data streams to facilitate inter-module communication, augmented with a microservices architecture, see Figure 3.
In order to support a large ecosystem that contains many different deve- lopers, researchers, technologies, applications, and organizations, RIDE follows a layered architecture (Figure 4).
The Engine Layer allows RIDE to leverage robust gaming technologies that provide common capabilities, including rendering, physics, animation, path- finding, User Interface (UI), audio, and network protocols. RIDE is agnostic to any specific game or simulation engine, which avoids vendor lock-in and enables researchers and developers to create simulation scenarios without the
Creating Virtual Worlds With the Virtual Human Toolkit 371
Figure 4: The RIDE architecture.
need to have experience with a specific engine. Currently, the main target is Unity, with key capabilities being ported to Unreal Engine.
The Middleware Layer abstracts and augments the Engine Layer with simulation-specific capabilities, including 3D geospatial terrain, AI agent behaviors, combat, scenarios, machine learning (ML) interfaces, and netw- orked multi-user. RIDE is designed to provide architectural flexibility in order to facilitate R&D in support of systems design, technical performa- nce assessment, and scalability. It contains and is extendable with permissive 3rd party assets and libraries, has native support for Amazon Web Servi- ces (AWS), Google, and Microsoft Azure cloud web services, and can itself act as a web service for any of its capabilities. The RIDE API encapsulates these available capabilities in a well-designed suite of systems and services. The API follows interface-based design, which allows for principled imple- mentations of new technologies; new concrete implementations simply need to implement the appropriate interface in order to be used in RIDE appli- cations. This not only provides the ability to extend RIDE with relatively little effort, it also enables multiple technical implementations of a single feature under unified interfaces, making it possible to contrast and com- pare, simultaneously or independently, separate approaches within a single platform.
The Project Layer allows researchers and developers to leverage RIDE as a foundation for their own projects. RIDE provides common functionality through a drag-and-drop interface in combination with the API. This enables researchers and developers to rapidly create new scenarios as a starting point for their specific needs. The Project Layer acts as an incubation area, where new technologies and approaches can be explored safely, with mature results moving back to the Middleware Layer in order to advance RIDE and benefit all of its users.
372 Hartholt and Mozgai
Figure 5: Military virtual human in 3D geospatial captured terrain.
Taken together, RIDE leverages robust industry capabilities, provides added value with each additional integrated capability, avoids reinventing the wheel, and offers researchers and developers a unified way to rapidly create, validate, and assess novel technologies and approaches, while contributing to a growing community of collaborators.
RIDE has integrated the VHToolkit messaging protocol (VHMsg) built on top of ActiveMQ. This allows interfacing with any of the main Toolkit modu- les and the capabilities they represent, including natural language processing (NLP) and nonverbal behavior generation. In addition, by leveraging RIDE’s dedicated web services system, commodity AI-related services are provided, including audio-visual sensing, speech recognition, NLP, and text-to-speech generation. Nonverbal behavior realization (e.g., lip-sync, facial expressions, conversational gestures) is directly integrated within RIDE. These capabilities allow agents to observe real human users, create its communicative intent, and realize that intent verbally and nonverbally, synchronized in real-time. See Figure 5 for a military and Figure 6 for a civilian virtual human example. Figure 7 shows a real-time comparison of question-answering NLP with the Microsoft QnAMaker, AWS Lex, and Google Dialogflow services in RIDE. Future work will focus on advancing the VH related API and creating a more modular and flexible VHToolkit 2.0. See (Hartholt et al. 2022) for details.
CREATING VIRTUAL HUMANS IN VIRTUAL WORLDS Within RIDE, creators can leverage existing 3rd party environmental assets (e.g., Unity Asset Store) or select from an existing suite of 3D geospatial ter- rains. This real-world data has either been captured using Small Unmanned Aerial System (sUAS) (Chen et al. 2020a) (Hou et al. 2022) or processed from Bing data (Chen et al. 2020b).
Creating Virtual Worlds With the Virtual Human Toolkit 373
Figure 6: Civilian virtual human in 3D geospatial captured terrain (USC campus).Figure 7: Contrast and compare NLP solutions in real-time in a single runtime.
Within the environment, agents can be placed and scripted with either state machines or behavior trees in order to bootstrap experiments. Provided behaviors include movement, attacking, and formations, tied to dedicated combat and health systems. Alternatively, networked multiplayer allows multiple teams of users to control avatars in a common synthetic environ- ment, each from their own location and type of hardware (e.g., desktop or mobile device). This allows bootstrapping scenarios and data collection in preparation for learning behaviors through machine learning platforms (Hartholt, Mccullough, et al. 2021b).
Virtual human authoring tools are currently being developed based on pre- vious work detailed in (Hartholt, et al. 2020) and (Hartholt, Fast, et al. 2021). An example can be seen in Figure 8, where a visual editor enables domain experts to create NLP content. The tool is integrated with AWS Lex, MS QnAMaker, and Google Dialogflow, allowing import and export of data from their respective web authoring tools. The authoring tool is developed natively in Unity, facilitating real-time debugging in the same virtual envi- ronment and ensuring the ability to support multiple hardware targets (e.g., desktop application vs. web app).
The development of these tools follows a user-centered design approach with particular emphases on embedding target users in the development team (S. A. Mozgai et al. 2021), domain transitions (Mozgai, Hartholt, Leeds, et al. 2020), and focus groups (S. Mozgai et al. 2021).
CONCLUSION
We presented an overview of the combined capabilities of the Virtual Human Toolkit and RIDE, demonstrating the ability to create interactive virtual humans within virtual worlds. The modular architecture, human-centered development approach, visual authoring tools, and interfaces to commodity AI web services enable interdisciplinary teams to design, develop, and deploy engaging, meaningful, efficient, and effective integrated systems.
ACKNOWLEDGMENT
The authors would like to acknowledge their many VHToolkit and RIDE collaborators: Ed Fast, Andrew Leeds, Kevin Kim, Edwin Sookiassian, Kyle McCullough, Andrew Feng, Austin Drexler, Chinmay Chinara, Jon Hawes, Noah Nam, Rajay Kumar, Ray New, Robert Finney, Zongjian Li, Andrew
Creating Virtual Worlds With the Virtual Human Toolkit 375
Gordon, Volkan Ustun, Meida Chen, Jon Gratch, David Traum, Anton Leuski, Stefan Scherer, Stacy Marsella, Chris McGroarty, and Keith Brawner. Part of the efforts depicted were sponsored by the US Army under contract number W911NF-14-D-0005. The content of the information does not neces- sarily reflect the position or the policy of the Government, and no official endorsement should be inferred.
REFERENCES
Campbell, Julia, Mark Core, Ron Artstein, Lindsay Armstrong, Arno Hartholt, Cyrus Wilson, Kallirroi Georgila, Fabrizio Morbini, Edward Haynes, Dave Gomboc, Mike Birch, Jonathan Bobrow, H. Chad Lane, Jillian Gerten, Anton Leuski, David Traum, Matthew Trimmer, Rich DiNinni, Matthew Bosack, Timothy Jones, Richard E. Clark, and Kenneth A. Yates. 2011. “Developing INOTS to Support Interpersonal Skills Practice.” IEEE Aerospace Conference Proceedings. doi: 10.1109/AERO.2011.5747535. Chen, Meida, Andrew Feng, Kyle McCullough, Pratusha Bhuvana Prasad, Ryan
McAlinden, and Lucio Soibelman. 2020a. “3D Photogrammetry Point Cloud Segmentation Using a Model Ensembling Framework.” Journal of Computing in Civil Engineering 34(6). Chen, Meida, Andrew Feng, Kyle McCullough, Pratusha Bhuvana Prasad, Ryan McAlinden, and Lucio Soibelman. 2020b. “Semantic Segmentation and Data Fusion of Microsoft Bing 3D Cities and Small UAV-Based Photogrammetric Data.” in Interservice/Industry Training, Simulation, and Education Conference. Chu, Veronica C., Gale M. Lucas, Su Lei, Sharon Mozgai, Peter Khooshabeh,
and Jonathan Gratch. 2019. “Emotion Regulation in the Prisoner’s Dilemma: Effects of Reappraisal on Behavioral Measures and Cardiovascular Measures of Challenge and Threat.” Frontiers in Human Neuroscience 13:50. Devault, David, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, Gale Lucas, Stacy Marsella, Fabrizio Morbini, Angela Nazarian, Stefan Scherer, Giota Stratou, Apar Suri, David Traum, Rachel Wood, Yuyu Xu, Albert Rizzo, and Louis- philippe Morency. 2014. “SimSensei Kiosk : A Virtual Human Interviewer for Healthcare Decision Support.” 2014 International Conference on Autonomous Agents and Multi-Agent Systems. International Foundation for Autonomous Agents and Multiagent Systems (1):1061–68. Gordon, Carla, Anton Leuski, Grace Benn, Eric Klassen, Edward Fast, Matt Liewer,
Arno Hartholt, and David Traum. 2019. “PRIMER: An Emotionally Aware Virtual Agent.” in Intelligent User Interfaces (IUI). Hartholt, Arno, Ed Fast, Andrew Leeds, and Sharon Mozgai. 2021. “Introducing VHMason: A Visual, Integrated, Multimodal Virtual Human Authoring Tool.” Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents, IVA 2021, 109–11. doi: 10.1145/3472306.3478363. Hartholt, Arno, Ed Fast, Zongjian Li, Kevin Kim, Andrew Leeds, and Sharon
Mozgai. 2022. “Re-Architecting the Virtual Human Toolkit: Towards an Inte- roperable Platform for Embodied Conversational Agent Research and Develo- pment.” IVA 2022 - Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents. doi: 10.1145/3514197.3549671. Hartholt, Arno, Ed Fast, Adam Reilly, Wendy Whitcup, Matt Liewer, and Sharon Mozgai. 2019. “Ubiquitous Virtual Humans: A Multi-Platform Framework for Embodied Ai Agents in Xr.” Proceedings - 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2019, 308–12.
376 Hartholt and Mozgai
Hartholt, Arno, Ed Fast, Adam Reilly, Wendy Whitcup, Matt Liewer, and Sharon Mozgai. 2020. “Multi-Platform Expansion of the Virtual Human Toolkit: Ubi- quitous Conversational Agents.” International Journal of Semantic Computing 14(3):315–32. doi: 10.1142/S1793351X20400127. Hartholt, Arno, Jonathan Gratch, Lori Weiss, and The Gunslinger Team. 2009. “At
the Virtual Frontier: Introducing Gunslinger, a Multi-Character, Mixed-Reality, Story-Driven Experience.” Lecture Notes in Computer Science (Including Subse- ries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 5773 LNAI:500–501. doi: 10.1007/978-3-642-04380-2_62. Hartholt, Arno, Kyle Mccullough, Ed Fast, Adam Reilly, Andrew Leeds, Sharon Mozgai, Volkan Ustun, and Andrew S. Gordon. 2021a. “Introducing RIDE: Lowering the Barrier of Entry to Simulation and Training through the Rapid Inte- gration & Development Environment.” in Simulation Interoperability Standards Organization (SISO) Simulation Innovation Workshop (SIW). Hartholt, Arno, Kyle Mccullough, Ed Fast, Adam Reilly, Andrew Leeds, Sharon
Mozgai, Volkan Ustun, Andrew S. Gordon, and Chris McGroarty. 2021b. “Rapid Prototyping for Simulation and Training with the Rapid Integration & Develo- pment Environment (RIDE) The Sigma Cognitive Architecture and System View Project RIDE (Rapid Integration & Development Environment) View Project.” in Interservice/Industry Training, Simulation and Education Conference (I/ITSEC). Hartholt, Arno, and Mozgai, Sharon. 2022. “Platforms and Tools for SIA Research and Development.” pp. 261–304 in The Handbook on Socially Interactive Agents: 20 years of Research on Embodied Conversational Agents, Intelligent Vir- tual Agents, and Social Robotics Volume 2: Interactivity, Platforms, Application. ACM. Hartholt, Arno, Sharon Mozgai, Ed Fast, Matt Liewer, Adam Reilly, Wendy Whitcup,
and Albert Skip Rizzo. 2019. “Virtual Humans in Augmented Reality: A First Step towards Real-World Embedded Virtual Roleplayers.” HAI 2019 - Proceedings of the 7th International Conference on Human-Agent Interaction. Hartholt, Arno, Sharon Mozgai, and Albert “Skip” Rizzo. 2019. “Virtual Job Interviewing Practice for High-Anxiety Populations.” Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents (IVA) 238–40. doi: 10.1145/3308532.3329417. Hartholt, Arno, Adam Reilly, Ed Fast, and Sharon Mozgai. 2020. “Introducing
Canvas: Combining Nonverbal Behavior Generation with User-Generated Con- tent to Rapidly Create Educational Videos.” Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents, IVA 2020. Hartholt, Arno, David Traum, Stacy C. Marsella, Ari Shapiro, Giota Stratou, Anton Leuski, Louis-Philippe Morency, and Jonathan Gratch. 2013. “All Together Now: Introducing the Virtual Human Toolkit.” Intelligent Virtual Agents; IVA 2013 368–81. doi: 10.1007/978-3-642-40415-3_33. Hou, Yu, Meida Chen, Rebekka Volk, and Lucio Soibelman. 2022. “Investiga-
tion on Performance of RGB Point Cloud and Thermal Information Data Fusion for 3D Building Thermal Map Modeling Using Aerial Images under Different Experimental Conditions.” Journal of Building Engineering 45. Johnson, W. Lewis, Jee W. Rickel, and James C. Lester. 2000. Animated Pedagogical Agents: Face-to-Face Interaction in Interactive Learning Environments. Lane, H. Chad, Dan Noren, Daniel Auerbach, Mike Birch, and William Swartout.
2011. “Intelligent Tutoring Goes to the Museum in the Big City: A Pedagogi- cal Agent for Informal Science Education.” Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 6738 LNAI:155–62. doi: 10.1007/978-3-642-21869-9_22.
Creating Virtual Worlds With the Virtual Human Toolkit 377
Mell, Johnathan, Gale M. Lucas, Sharon Mozgai, and Jonathan Gratch. 2020. “The Effects of Experience on Deception in Human-Agent Negotiation.” Journal of Artificial Intelligence Research 68:633–60. Mell, Johnathan, Gale Lucas, Sharon Mozgai, Jill Boberg, Ron Artstein, and
Jonathan Gratch. 2018. “Towards a Repeated Negotiating Agent That Treats Peo- ple Individually: Cooperation, Social Value Orientation, & Machiavellianism.” Proceedings of the 18th International Conference on Intelligent Virtual Agents, IVA 2018 125–32. doi: 10.1145/3267851.3267910. Mozgai, Sharon A., Brian Femminella, Arno Hartholt, and Albert Rizzo. 2021.
“User-Centered Design Model for Mobile Health (MHealth) Applications: A Mili- tary Case Study in Rapid Assessment Process (RAP).” Conference on Human Factors in Computing Systems - Proceedings. Mozgai, Sharon, Arno Hartholt, Dayo Akinyemi, Katarina Kubicek, Albert (Skip) Rizzo, and Michele Kipke. 2020. “Development and Initial Feasibility Testing of the Virtual Research Navigator (VRN): A Public-Facing Agent-Based Educational System for Clinical Research Participation.” pp. 304–7 in Internatio- nal Conference on Human-Computer Interaction (HCI). Mozgai, Sharon, Arno Hartholt, Andrew Leeds, and Albert Skip Rizzo. 2020. “Itera-
tive Participatory Design for VRET Domain Transfer: From Combat Exposure to Military Sexual Trauma.” Conference on Human Factors in Computing Systems - Proceedings. doi: 10.1145/3334480.3375219. Mozgai, Sharon, Arno Hartholt, and Albert Rizzo. 2020. “An Adaptive Agent- Based Interface for Personalized Health Interventions.” International Confe- rence on Intelligent User Interfaces, Proceedings IUI 37(111):118–19. doi: 10.1145/3379336.3381467. Mozgai, Sharon, Andrew Leeds, David Kwok, Ed Fast, Albert “Skip” Rizzo, and
Arno Hartholt. 2021. “Building BRAVEMIND Vietnam: User-Centered Design for Virtual Reality Exposure Therapy.” 2021 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR) 247–50. Mozgai, Sharon, Gale Lucas, Jonathan Gratch, Sharon Mozgai, Gale Lucas, and Jonathan Gratch. 2017. “To Tell the Truth: Virtual Agents and Morning Morality.” pp. 283–86 in Proceedings of the 17th International Conference on Intelligent Virtual Agents. Springer International Publishing. Neubauer, Catherine, Mathieu Chollet, Sharon Mozgai, Mark Dennison, Peter Kho-
oshabeh, and Stefan Scherer. 2017. “The Relationship between Task-Induced Stress, Vocal Changes, and Physiological State during a Dyadic Team Task.” ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction 2017-January:426–32. doi: 10.1145/3136755.3136804. Neubauer, Catherine, Sharon Mozgai, Brandon Chuang, Joshua Woolley, and Stefan Scherer. 2018. “Manual and Automatic Measures Confirm-Intranasal Oxytocin Increases Facial Expressivity.” 2017 7th International Conference on Affective Computing and Intelligent Interaction, ACII 2017 2018-January. Rizzo, Albert, Russell Shilling, Eric Forbell, Stefan Scherer, Jonathan Gratch, and
Louis-Philippe Morency. 2016. “Autonomous Virtual Human Agents for Heal- thcare Information Support and Clinical Interviewing.” Artificial Intelligence in Behavioral and Mental Health Care 53–79. Rosenbloom, Paul S., Abram Demski, and Volkan Ustun. 2016. “The Sigma Cognitive Architecture and System: Towards Functionally Elegant Grand Unification.” Journal of Artificial General Intelligence.
378 Hartholt and Mozgai
Scherer, Stefan, Stacy Marsella, Giota Stratou, Yuyu Xu, Fabrizio Morbini, Alesia Egan, Albert Rizzo, and Louis-Philippe Morency. 2012. “Perception Markup Language: Towards a Standardized Representation of Perceived Nonverbal Beh- aviors.” pp. 455–63 in. Springer, Berlin, Heidelberg. Schroeder, NL, Olusola O. Adesope, and RB Gilbert. 2013. “How Effective Are Peda-
gogical Agents for Learning? A Meta-Analytic Review.” Journals.Sagepub.Com 49(1):1–39. doi: 10.2190/EC.49.1.a. Shapiro, Ari. 2011. “Building a Character Animation System.” pp. 98–109 in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Swartout, William R., Benjamin D. Nye, Arno Hartholt, Adam Reilly, Arthur C.
Graesser, Kurt VanLehn, Jon Wetzel, Matt Liewer, Fabrizio Morbini, Brent Morgan, Lijia Wang, Grace Benn, and Milton Rosenberg. 2016. “Designing a Per- sonal Assistant for Life-Long Learning (PAL3).” The Twenty-Ninth International Flairs Conference.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Intelligent Human Systems Integration (IHSI 2023): Integrating People and Intelligent Systems
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-45-2
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional