Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Natural Language Processing to Assess Communication Dynamics between Cooperating Dyads during Video Gameplay

1 School of Biomedical Engineering, Science and Health Systems, Drexel University, USA ↗
2 LeBow College of Business, Drexel University, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1001827

Abstract

Latent Dirichlet Allocation (LDA) and Sentiment Analysis have become prominent tools in natural language processing applications for both research and industry. While LDA is a generative probabilistic modeling methodology that is widely used in Topic Modeling to extract underlying themes and topics from a collection of words, Sentiment Analysis is defined as identifying the hedonic tone of a corpus of text. Here, supervised Sentiment Analysis is used to classify conversations between team gaming dyads in terms of valence. Additionally, LDA is utilized to label segments of cooperative conversation between dyads as topics. Fourteen participants were paired as dyads (7 teams) and instructed to complete thirty-two 150 second gaming scenarios (trials) in the first-person shooter (FPS) video game Overwatch. While completing the scenarios, participants were instructed to communicate with their respective teammate via a voice communication headset. The conversations from each scenario were transcribed from recorded voice channels before analysis was performed. Our approach examines the relationship between perceived task difficulty and both conversation sentiment scores and topic frequency in both novice experienced skill groups. Preliminary results indicate evidence that conversation topic, sentiment and perception dynamics are consistent with an encouragement and frustration sentiment paradigm.

INTRODUCTION

The objective measurement of group cooperation and communication along with the assessment of team based outcomes has attracted increased attention

© 2022. Published by AHFE Open Access. All rights reserved. 129

in both research and industry (Jones and George, 1998) with team-based video games ideally suited to serve as a platform for teamwork and com- munication research as players work together to reach a common outcomes (Greitemeyer and Cox, 2013).

Asverbal communication is a primary signal for collaboration and coope- ration, natural language processing (NLP) tools have developed to categorize and classify conversational features that can serve as metrics for teamwork with the two prominent methods of Sentiment Analysis and Latent Dirichlet Allocation (LDA) primarily utilized. Through Sentiment Analysis, the feeling (or connotation) of a corpus of text can be observed for both positive and negative valence (Yadav and Vishwakarma, 2020) while LDA utilizes unsupe- rvised machine learning to label segments of text as topics (Blei et al., 2003).

This study seeks to compare the relationship between subjects’ speech lan- guage and self-perceptions of in-game performance by examining the inter- play between self-reported measures, sentiment scores and conversational topic modelling.

METHODS

Participants A total of 14 healthy adult participants (6 female, mean age = 22) volunteered for this study. All participants fulfilled the requirements of being right-handed with vision correctable to 20/20. Before the study, participants filled and signed the consent form approved by the Institutional Review Board of Drexel University.

Experiment Protocol Participants completed a single one-hour study where participants sat in front of a computer and played team-based task scenarios in Overwatch, a first- person shooter (FPS) game designed by Activision-Blizzard (Santa Monica, CA, www.activisionblizzard.com). Participants filled out background forms to classify whether they were a novice or experienced FPS video game players before playing two 16 trial sessions of the game with an assigned teammate of comparable skill level. Teams played in ‘Escort Mode’, an Overwatch game mode consisting of two teams with the opponent (two artificial intelligence (AI) bots set to a medium difficulty). Within a fixed duration the attacking team was instructed to escort a ‘payload’ to checkpoint in order to win the game, while the defending team was instructed to prevent the payload from being advanced to its destination. Each trial had a time limit of 150 seconds with 30 second interstitial rest periods. A single trial ended either when the timer expired, or the payload had been successfully escorted to the final che- ckpoint. At the end of each trial, participants were instructed to rate their perception of gameplay difficulty on a Likert scale (1 to 7). See Fig. 1 below for a visual depiction of the task and protocol diagram.

Data Acquisition, Processing and Analysis Each participant dyad produced a total of four audio recordings initially transcribed by Google Cloud’s Text-to-Speech API (Bisong, 2019) before

Natural Language Processing to Assess Communication Dynamics 131

Figure 1: (a) An example snapshot of the Escort Mode, where the attacking team tran- sports the ‘payload’ by standing close to it while defending team tries to prevent the attacking team escort the ‘payload’. (b) The map for the game scenarios. The green line represents the payload’s path, green dot represents the destination. (c) Diagrammatic representation experimental procedure.
Original PDF Figure 1:

recordings were manually corrected to correct machine error when conver- ting conversations to text. All data were preprocessed using MATLAB’s Text Analytics toolbox. Each transcript (document) was separated into 8 bins that corresponded to a single experimental trial (~180 seconds). Prior to LDA analysis sentiment analysis was performed to produce Vader sentiment sco- res ranging from −1 to 1, with the lower end reflecting negative sentiment and upper end reflecting positive sentiment.

For LDA modelling, each document bin was tokenized prior to stop word removal, punction erasure and lemmatization and converted into a single “bags-of-words” term frequency counter (a string array of unique words and a matrix of word counts) for fitting an LDA model with three topics which allowed for the calculation of the probability that a single topic was present as well as identifying the most frequent topic within a participants document. Additionally, each of three three topics generated were subjectively interpre- ted prior to statistical analysis producing the three topic labels “Directions”, “Frustration” and “Encouragement.

Results

LDA results are summarized in Figure 2 comparing both novice and expe- rienced FPS game players’ most frequently discussed topic during a single gaming session in terms of self-reported perception of the sessions diffi- culty (mean rating across session trials). A main effect is observed for Most Frequent Topic (F2,17.8=3.8, p = 0.04) and post-hoc analysis reveals a signi- ficant interaction between the Directions Topic and Level of Experience (F1,11.1=8.56, p = 0.04).

Figure 2 shows a comparison of novice and experienced players most frequently discussed session topic and their mean sentiment score (for all session trials) for gameplay during a single session. Here, there is a

Figure 2: LDA results: Comparison between novice and experienced players mean self- reported trial ratings for game difficulty and the most frequently discussed topic within the session. Whiskers are standard error of the mean (SEM).
Original PDF Figure 2:
Figure 3: LDA and sentiment analysis results: Comparison between novice and expe- rienced players mean trial sentiment score and the most frequently discussed topic within the session. Whiskers are SEM.
Original PDF Figure 3:

main interaction between Level of Experience and Most Frequent Topic (F2,17.9=4.51, p = 0.03) and post-hoc analysis indicates significant intera- ctions for the Novice Group between Direction and Encouragement Topics (F1,18.5=8.47, p = 0.05) as well as for Frustration and Encouragement Topics (F1,19.9=10.24, p = 0.03).

DISCUSSION

Just as the perception of task difficulty is closely tied to an individual’s skill level, both the nuances of spoken words and the tone surrounding them may very well be subject to the effect of one’s perception of expertise (or lack

Natural Language Processing to Assess Communication Dynamics 133

of expertise). In this study, we examine the relationship between perceived task difficulty and both conversation sentiment scores and topic frequency in both novice and experienced skill groups. Preliminary results indicate that when perceived difficulty is high, both novices and experts tend tow- ards words of encouragement when collaborating to achieve a common goal. However, experienced game players exhibit greater frequencies of frustra- ted language when task difficulty is low, highlighting that a lack of challenge catalyzes an intensified criticism of otherwise negatable behaviors. More inte- resting though, is an observable relationship between the prominent topics of conversation and sentiment scores related to these topics in the context of competitive tasking. It is apparent that for novices, despite being frustrated more often, these participants possessed an overall higher sentiment in these conversations, suggesting that the relative personal unimportance of a task lends to feelings of, “just being happy to be there”. Alternatively, experienced and seasoned players who favored topics of directives and encouragement maintained an overall heightened positivity over novices while experienced players favoring topics of frustration were significantly more negative than novices in sentiment score. These trends suggest a subtle paradigm at work, where internalized perception of skill and expertise lead to a higher contrast in sentimental valence between differing topics of conversation.

In conclusion, this study highlights the use of Natural Language Pro- cessing techniques, specifically Latent Dirichlet Allocation and Sentiment Analysis to elucidate both topic and hedonistic tone of conversation in a cooperative team-based competitive video gaming environment. These preli- minary findings warrant further investigation into the relationship between self-perceptions of skill and performance and its effect on the language of teamwork.

REFERENCES

Bisong E., (2019) “An overview of Google cloud platform services,” Building Machine Learning and Deep Learning Models on Google Cloud Platform, pp. 7–10. Blei D. M., Ng A. Y., and Jordan M. I., (2003) “Latent Dirichlet allocation,” Journal

of machine Learning research, vol. 3, no. Jan, pp. 993–1022. Greitemeyer T. and Cox C., (2013) “There’s no “I” in team: Effects of cooperative video games on cooperative behavior,” European Journal of Social Psychology, vol. 43, no. 3, pp. 224–228. Jelodar H. et al., (2019) “Latent Dirichlet allocation (LDA) and topic modeling:

models, applications, a survey,” Multimedia Tools and Applications, vol. 78, no. 11, pp. 15169–15211. Jones G. R. and George J. M., (1998) “The experience and evolution of trust: Implications for cooperation and teamwork,” Academy of management review, vol. 23, no. 3, pp. 531–546. Yadav A. and Vishwakarma D. K., (2020) “Sentiment analysis using deep lear-

ning architectures: a review,” Artificial Intelligence Review, vol. 53, no. 6, pp. 4335–4385.

Cover of the proceedings volume Neuroergonomics and Cognitive Engineering

Explore the complete digital volume

Neuroergonomics and Cognitive Engineering

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-18-6

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →