Kohei Nonose, Taro Kanno, Kazuo Furuta (2020). A Team Communication Evaluation Method for Assessing the Smoothness of Task Flow. In: Advances in Science, Technology, Higher Education and Society in the Conceptual Age: STHESCA. AHFE Open Access, vol. 10. https://doi.org/10.54941/ahfe100403
Effective communication is essential for positive teamwork across team tasks, and thus team communication analysis methods have received much attention; however, because of the great variety of team tasks, it is often impossible to apply analysis methods and findings about the relationships between team performance and communication in one team task to those of another task. It might therefore be necessary to develop analysis and assessment methods of team communication independent of team task characteristics. Our previous study developed an evaluation index of team communication named “Smoothness of Task Flow (STF),” which was expected to represent how well a lower task step smoothly shifts to the next higher one across different tasks; however, the index has certain problems and has been never applied to another task. The present study improves the evaluation index and validates the improved index by conducting team experiments in which the task requires more complicated communication than did the previous task. The results suggest that the improved index provides more accurate assessment of the smoothness of team communication than did the previous index. The characteristics of the improved index are also discussed using case studies.
Effective communication is considered essential for positive teamwork across different types of team tasks, because communication enables team members to share information to build team cognition (e.g., team situation awareness, shared situation awareness) (Cooke et al., 2003), and thus relationships between team communication and performance have received much attention from researchers in different areas to assess teamwork (MacMillan et al., 2004). For example, Muniz et al. (1998) reviewed behavioral indicators of high/low team situation awareness to propose behavioral indicators of such awareness. Their indicators include communication (e.g., communicated important information, confirmed information when possible) as an important factor.
One of the most common approaches in team communication studies is to segment communication into meaningful sequences and then classify each sequence into categories according to their contents or intentions in order to analyze teamwork by those categories (Bowers et al., 1998; Foltz et al., 2006). Parush et al. (2014), for example, coded communication in diverse healthcare contexts into two categories: contents (e.g., drug administration, patient Technology, Higher Education and Society (2020) status), and verbal behaviors (e.g., question, reply, read-back, clarification request). According to the analyses, they examined whether individuals collaborate with others differently in different contexts or the fundamental teamwork processes are similar regardless of diverse context characteristics.
Another approach is to calculate a statistical evaluation index of team communication for a certain category to assess teamwork on team communication criteria. For example, Serfaty et al. (1998) have proposed an index called the anticipation ratio, which is the ratio of the number of communications transferring information and action to the number of communications requesting them (Equation 1).
Anticipationratio=the numberof transferring
the numberof requesting .(1)
This ratio is recognized as a measure of communication efficiency demonstrated to be associated with effective team performance (Sperling, 2006). Values greater than one indicate that team members “pushed (sent)” information more frequently than they “pulled (requested)” information; they anticipated each other’s needs for information without a request.
A problem related to the relationships between team communication and team performance is that team tasks have diverse contexts, and thus it is often impossible to apply findings about the relationships in one team task to those of another task. To address this problem, it might be necessary to develop a communication analysis and assessment method that is independent of team task characteristics. In addition, popular approaches for analyzing team communications, such as the proportion and patterns of communication classification codes (Bowers et al., 1998; Parush et al., 2014), the duration of communication (Kiekel et al., 2002), social network analysis (Houghton et al., 2008), and sequences of speakers (Gorman et al., 2012), produce only a superficial description of team behaviors, rather than providing important insights through deep analyses of team cooperation. That is, they can indicate relationships between team performance and team communication but cannot identify the bottlenecks that can worsen team performance.
Our previous study developed an evaluation index of team communication as a new statistical evaluation index named “Smoothness of Task Flow” (STF) (Nonose et al., 2009). This index was obtained using the category of contents derived from a task analysis that divides task into a higher team task step and a lower one and three categories of intentions (“Query”—to query others to obtain necessary information or decisions; “Inform”—to inform others of task-related information or decisions, and to answer a partner’s “Query”; and “Acknowledge”— responses to “Inform”). This index was expected to represent how well a lower task step smoothly shifts to the next higher one. In this method, first, team communication data are classified according to the communication classification matrix. Then, for each task step, the method obtains the ratio of the number of “Query” to “Inform” intentions. Finally, the method obtains the STF by subtracting the ratio of a lower team task step from that of the next higher one (Equation 2).
STF= Inform(uppertask)
Query (lower task ) .(2)
Query (upper task) −Inform (lower task ) The previous study’s method also obtained the ratio of Inform (upper task) to Inform (lower task), the anticipation ratio, and the ratio of the number of utterances about the upper task to that of the lower task to examine the characteristics of the STF. The results demonstrated that the STF correlated best with the team performance score (mission scores).
To obtain the STF, team tasks are divided according to task step, and the ratio is obtained for each task step. If information exchanges between team members and/or cooperation activities encounter problems in a task step, the number of the utterances, especially “Inform,” related to that task step will increase; in addition, the number of “Query” utterances in the next higher task would also increase because team members would ask their partners whether the information about the defective task step was properly transferred. As a result, the ratio in the defective task step will be high and that in the next higher one will be low. This mechanism might partially explain why the STF had a better correlation than the other ratios. We concluded in the previous study that the STF can indicate bottlenecks of team cooperation in task steps.
The previous study and the index, however, have limitations. The task used in the previous study was so simple that Technology, Higher Education and Society (2020) there were no miscommunications between team members, and the three categories of intentions were insufficient to classify all intentions; in reality, people have a large variety of intentions behind utterances such as checking and correcting their partner’s thought (Kanno et al., 2013). Specifically, to assess team communication smoothness, or task flow, it is necessary to focus on miscommunication between team members that can cause or be caused by misunderstanding of the partner’s thought. In addition, one must also verify the versatility of the method for other tasks. The present study seeks to improve our previous STF index to more accurately represent the smoothness of task flow in teams and to validate the improved method by conducting a team experiment that requires more complicated communication to accomplish the team task.
The paper proceeds as follows. The Team Experiment section introduces the team experiment. The New Evaluation Index and New Smoothness of Task Flow sections introduce the new STF. Finally, the characteristics of the new STF are revealed by the results of the team experiment and case studies are discussed in the Results and Discussion section.
TEAM EXPERIMENT To validate the proposed method described in the next section, a team experiment was conducted.
Team Task A PC game (Bontago) in which participants were required to move and stack blocks to get three flags in the center against the opponent CPU players was used. In the experiment, each two-person team comprised one person who had only a mouse and could move blocks horizontally as well as change viewpoints and the other person who had only a keyboard and could move blocks vertically, zoom in and out, as well as rotate blocks. Neither team member could complete these tasks on his own, and thus they had to cooperate.
Team members were allowed to place a block at any moment within a time limit (5 seconds) and to hold it until the time was up. After the time limit, the block fell on its own and was replaced by the next block. In addition, because opponent players could cause an earthquake that destroys stacked blocks, participants had to check the status of their stacked blocks and sometimes re-stack blocks after an earthquake. They, therefore, had to make quick and good team decisions; otherwise, they could not accomplish the task.
Procedure Forty students (20 teams) participated in the experiment. They practiced the operation until the experimenter concluded that they had developed sufficient operational knowledge. When participants and opponents (CPU players) could not win the game within 10 minutes, the game ended in a draw and a new game started. When participants won or lost the game, a new game started. Each team attempted the task several times. The total trial duration was 30 minutes for each team. Participants sat face to face so that they could not use gestures (e.g., finger pointing, eye contact) for communication (Figure 1).
Mouse Keyboard
Horizontal moving of blocks
Vertical moving of blocks
Change of a viewpoint of a
Rotation of blocks
game screen
Zooming in/out
Display
Display
Free verbal communication
Figure 1: Setting of the team experiment
Figure available in the original PDF paper.
Technology, Higher Education and Society (2020)
Task Performance Trials were divided into three groups by task results: win trials in which teams accomplished the task, lose trials in which teams were defeated by the opponent player (CPU), and time-up trials in which the game ended in a draw.
NEW EVALUATION INDEX Contents of Communication
The category of contents is derived from a task analysis. The standard procedure of the task has two task steps. One is the lower task step in which participants understand the status of team and opponent areas (e.g., the stability of blocks, the progress of the opponent). The other one is the higher task step in which they make a team decision about the point where they put a block within 5 seconds and manipulate a block as smoothly and correctly as possible. Table 1 presents the contents of communication definitions.
Table 1: Contents of communication Contents Definition
Intentions of Communication Three types of intentions were used in our previous study: “Inform,” “Query,” and “Acknowledge.” The present study adds a new intention category, “Conflict,” defined as “To correct the partner’s misunderstanding about the speaker’s message and to disagree with the partner’s opinion” (Table 2). It is expected to represent communication between members that can cause loss of the team’s operational efficiency.
Table 2: Intentions of communication Intentions Definition
NEW SMOOTHNESS OF TASK FLOW First, team communication data is classified into the two categories (upper and lower) according to the communication classification matrix (Table 3). Utterances that were not related to the new STF were uncounted (e.g., “Thanks!”), because they were not necessarily required to accomplish the task. Then, for each task step, we
Technology, Higher Education and Society (2020) obtained the ratio of the sum of “Query” and “Conflict” plus one to “Inform” plus one. Finally, the new STF is obtained by subtracting the ratio of the lower team task step from that of the higher task (Equation 3). We added “plus one” to the denominators and the numerator to avoid a case where the denominator is zero. The reason for adding “Conflict” to the denominators is to represent loss of team communication caused by misunderstanding of the partner’s messages in the improved index. If team members frequently fail to inform team members of their messages about upper tasks, the new STF decreases by increasing the number of “Conflict (upper task)” and/or “Query (upper task).” This index is therefore expected to assess the efficiency of the smoothness of communication flow.
Table 3: The communication classification matrix Intentions Content s Mouse Keyboar d
Evaluation Index There were 12 win trials, 27 time-up trials, and 36 lose trials. Approximately 55% of the utterances were classified by the classification matrix (Table 3). Both the previous STF and the new STF were obtained for each trial. The anticipation ratio, which has been considered as values that correlate positively with team performance, was also obtained for each trial to verify the generality of the correlations. Figure 2 depicts the means of the previous STF, the new STF, and the anticipation ratio for each trial group (win, time-up, and lose). A T-test was used to identify the significant differences among the means.
The mean of the win trials was significantly higher than that of the lose trials in both the new STF and the previous STF (t (46) = 2.82, p < .01; t (46) = 2.78 p <.01). Although there was no significant difference between the mean of the previous STF of the win trials and that of the time-up trials (t (37) = 0.96, n.s.), the mean of the new STF of the win trials was significantly higher than that of the time-up trials (t (37) = 2.41, n.s.). There was no significant difference between the lose trials and the time-up trials in both the new STF and the previous STF indexes (t (61) = 0.30, n.s.; t (61) = 0.45, n.s.). There was also no significant difference among the anticipation ratios.
These results indicate that the new STF provides a better index for assessing team performance by smoothness of task flow than the previous STF. The fact that there was no significant relationship between team performance and the anticipation ratio suggests that there is no universal index that correlates positively across different tasks. The Technology, Higher Education and Society (2020)
Case Study section examines the characteristics of the new STF as applied in case studies.
6.0
5.0
4.0
3.0
Win
2.0
Time up
1.0
Lose
0.0
-1.0 -2.0 Previous STF New STF Anticipation ratio
Figure 2: The previous STF and the new STF
Figure available in the original PDF paper.
Case Study To examine the characteristics of the new STF in detail, we applied it to cases in which team members had to clear up a misunderstanding caused by poor communication between team members. In case 1 (Table 4), a team member (K), who had a keyboard, requested the partner (M), who had a mouse, to put a block on the front left of a certain block; however, M misunderstood K’s request and put the block on a point different from K’s request. K was aware of M’s misunderstanding and then corrected it. M checked his understanding of K’s request, and then K concretely stated his plan. Finally, M understood K’s request. In summary, K’s unclear request about the upper tasks caused M’s misunderstanding of K’s request, and then K had to express the same request twice, and M had to check K’s thought twice. The increasing numbers of “Conflict” and “Query” regarding the upper task in the process of building shared understanding of K’s request decreased the new STF. This case illustrates how the new STF represents the efficiency loss in team communication.
Table 4: Case 1
In a second case (Table 5), member K, who had a keyboard, requested partner M, who had a mouse, to move up a possessed block by saying “put it (a block) vertically.” However, M thought K wanted to rotate the block vertically, Technology, Higher Education and Society (2020) and he started to rotate it. M was aware of K’s misunderstanding and said, “No, no, horizontally!” to request K to rotate the block. However, K could not understand M’s request. Finally the block dropped to an unintentional point. This case illustrates, as did the former case, that K’s unclear request increased the numbers of “Conflict” and “Query” utterances about the upper task, and thus decreased the new STF. This case also demonstrates how the new STF represents the efficiency loss of team communication.
Table 5: Case 2
In both cases, the number of “Query” and “Conflict” utterances about the upper task step increased because of the member’s misunderstanding of the partner’s thought caused by unclear messages. This increase is reflected as decreasing of the new STF. This mechanism is probably one reason for the new STF being a better index than the previous STF.
If a team’s stacked blocks were collapsed by an earthquake caused by the opponents or if the opponent smoothly operated the task, the number of “Inform” utterances about the lower task step (e.g., “Our stacked blocks were collapsed,” “The opponents are getting there!”) probably increased so that the new STF decreased. In addition, because both team members can obtain information about the lower task step independently in the present task, the number of “Query” and “Conflict” utterances about the lower task increased only slightly. These factors enable the new STF to correlate positively with team performance in the present task.
One issue of the present study is that almost half of the utterances could not be classified according to the matrix. One reason is that each participant often says something to him/herself during the task. Because such monologues can help team members to share information and their feelings in teams, it might be additionally necessary to take the monologues into account for deeper examination. Another issue is that it is necessary to check the inter-rater reliability of the communication classification, because it might be difficult to distinguish between such monologues and the “Inform” category and between the “Conflict” and “Inform” categories.
It is difficult to find universal relationships between team performance and team communication because of the large variety of team types and team tasks. To better investigate the relationships, it is necessary to consider the taxonomy of team types and team tasks (Devine, 2002; Paris et al., 2000). One advantage of the method proposed in this study is that it might apply to other team tasks because it is based on a task analysis using task steps that previous studies have sometimes overlooked.
CONCLUSIONS
This study improved an evaluation index of team performance in terms of the smoothness of team communication proposed by our previous study. The comparisons between the previous index and the improved index indicate that the improved index has better sensitivity to team performance than the previous one. In addition, we introduced cases expected to represent the differences between the previous index and the improved index to specifically Technology, Higher Education and Society (2020) describe the characteristics of the improved method. These results suggest that the improved method can provide more accurate assessment of the smoothness of communication than did the previous index.
REFERENCES
Bowers, C. A., Jentsch, F., Salas, E., and Braun, C.C. (1998). “Analyzing communication sequences for team training needs assessment”, Human Factors, Volume 40, pp. 672-679. Cooke, N. J., Kiekel, P. A., Salas, E., Stout, R., Bowers, C., Cannon-Bowers, J. (2003). “Measuring team knowledge: A window
to the cognitive underpinnings of team performance”. Group Dynamics: Theory, Research, and Practice, Volume 7 No.3. 179. Devine, D. J. (2002). “A review and integration of classification systems relevant to teams in organizations.” Group Dynamics: Theory, Research, and Practice, Volume 6 No. 4, pp. 291-310. Foltz, P.W., Martin, M.J., Abdelali, A., Rosenstein, M., Oberbreckling, R. (2006). “Automated Team Discourse Modeling: Test
of Performance and Generalization” In Proceedings of the Cognitive Science Annual Meeting, pp. 1317-1321. Gorman, J. C., Cooke, N. J., Amazeen, P. G., Fouse, S. (2012). “Measuring patterns in team interaction sequences using a discrete recurrence approach”. Human Factors: The Journal of the Human Factors and Ergonomics Society, Volume 54 No. 4, pp.503-517. Houghton, R. J., Baber, C., Cowton, M., Walker, G. H., Stanton, N. A. (2008). “WESTT (workload, error, situational awareness,
time and teamwork): an analytical prototyping system for command and control”, Cognition, Technology & Work, Volume 10 No.3, pp. 199-207. Kanno, T., Furuta, K., Kitahara, Y. (2013). “A Model of Team Cognition based on Mutual Beliefs”, Theoretical Issues in Ergonomics Science, Volume 14 No. 1. Kiekel, P. A., Cooke, N.J., Foltz, P.W., Gorman, J. C., Martin, M.J. (2002). “Some promising results of communication-based
automatic measures of team cognition”. In Proceedings of the Human Factors and Ergonomics Society’s Annual Meeting, USA, 46, 298-302. MacMillan, J., Entin, E. E., & Serfaty, D. (2004). “Communication overhead: The hidden cost of team cognition. Team cognition: Process and performance at the interand intra-individual level”. American Psychological Association, Washington, DC. Muniz, E., Stout, R., Bowers, C., Salas, E. (1998). “A methodology for measuring team situational awareness: situational
awareness linked indicators adapted to novel tasks (SALIENT)”. NATO human factors and medicine panel on collaborative crew performance in complex systems, Edinburgh, North Atlantic Treaties Organisation, Neuilly-sur-Seine, 20-24. Nonose, K., Kanno, T., Furuta, K. (2009), “An Evaluation Method of Team Cooperative Communication Based on Task Analysis”, Proceeding of 17th Congress of the International Ergonomics Association (Beijing, China). Paris, C. R., Salas, E., & Cannon-Bowers, J. A. (2000). “Teamwork in multi-person systems: a review and
analysis”, Ergonomics, 43(8), 1052-1075. Parush, A., Kramer, C., Foster-Hunt, T., McMullan, A., Momtahan, K. (2014) “Exploring similarities and differences in teamwork across diverse healthcare contexts using communication analysis”, Cognition, Technology & Work, Volume 16 No. 1, pp. 47-57. Serfaty, D., Entin, E. E., Johnston, J. (1998), “Team coordination training”. In J. A. Cannon-Bowers & E. Salas (Eds.), Making
decisions under stress (pp. 221–245). Washington, DC: American Psychological Association. Sperling, B. K.(2006), “Information Distribution and Team Situational Awareness: An Experimental Study”. In Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting (pp. 477-481). Santa Monica, CA: Human Factors and Ergonomics Society.
Technology, Higher Education and Society (2020)
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Advances in Science, Technology, Higher Education and Society in the Conceptual Age: STHESCA
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-4951-2110-4
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional