Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

A Novel Methodology for Evaluating Military Teamwork and Team Marksmanship Performance

1 U.S. Army Combat Capabilities Development Command Soldier Center, Natick, MA 01760, USA ↗
2 U.S. Army Combat Capabilities Development Command Soldier Center, Natick, MA 01760, USA; Arizona State University, Tempe, AZ 85281, USA ↗
3 U.S. Army Aberdeen Test Center, Aberdeen, MD 21005, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1001899

Abstract

Marksmanship has been a key metric in evaluating total Soldier performance. While marksmanship assessment is typically done at the individual level, marksmanship performance is heavily embedded in team tasks and battle drills. Thus, an objectively measured and operationally-based assessment is needed to characterize teamwork in marksmanship tasks, as well as evaluate its impact on team marksmanship performance. This study was a proof-of-concept trial conducted during a 72-hour mission field study, using 39 active-duty male, infantry Soldiers. Thirteen 3-person teams completed a 6-minute scenario that simulated rapidly escalating firing engagement. The teams conducted a planning session to develop strategies for mission accomplishment prior to scenario start. At mission start, the team was situated in the center of a circle of 28 target light emitting diode (LED) displays, which they were required to cover and engage. The target LED displays were in one of three states (dark, non-threat, or threat), each represented by a pattern created by the research team. The scenario was split into six ~60-80 second segments, differing in number of targets presented, target identities (non-threat or threat), as well as density of targets displayed per sector. On-weapon and body sensor data was used to calculate team marksmanship performance (i.e., probability of target hits, probability of threat targets engaged). Additional teamwork data were gathered from observer ratings of teams’ communication and coordination during the scenario, and post-session questionnaires. To evaluate the effects of scenario segment, sector strategy, and communication strategy on team marksmanship performance, two 6x2x2 mixed analyses of variance (ANOVAs) were conducted with scenario segment as the within-subjects variable, and sector strategy (implicit or explicit) and communication strategy (plan or no plan) as the between-subjects variables. Team marksmanship performance outcomes included a team’s probability of hit, p(hit), and ratio of targets engaged, p(engage). Both marksmanship variables revealed significant main effects of segment, p<.001, where performance degraded as the scenario progressed. Additionally, p(hit) had a main effect of sector strategy, where those who used the external environment cues for sectoring (i.e., explicit) resulted in a higher probability of hit as compared to those who used teammate relative positions (i.e., implicit), p=.02. Team communication was trending towards significance, where having a communication plan resulted in lower p(hit), p=.07. For p(hit), there was also a three-way interaction between communication strategy, sector strategy, and segment, p<.01, where teams without a strategy performed more consistently in their shooting across the segments if they had an externally driven sector strategy (i.e., explicit), but performed just the same as those with a communication plan if they had an internally driven sectoring strategy (i.e., implicit). These results suggest that this methodology can not only characterize individual marksmanship skills but is beneficial to assessing team performance across operationally-based scenarios representing escalating short-term engagements and measuring the effects of team variables. However, our proof-of-concept analysis was limited by sample size and future development will strive to increase the number of teams participating. Additionally, future versions of this methodology will incorporate additional metrics of communication, physiology, and decision making.

INTRODUCTION

In recent years, the Army has prioritized Soldier lethality as one of their pri- mary modernization initiatives as laid out in their strategy for Multi-Domain Operations (Headquarters DOA, 2019a). Traditional military dismounted Soldier training regimes in the area of lethality focus on physical fitness, movement tactics, and basic marksmanship qualifications (Headquarters DOA, 2019b, Headquarters DOA, 2020). Soldiers are provided the equi- pment and minimum training necessary to be ready for close combat missions without sacrificing survivability. Individual skills, such as marksmanship and physical fitness, are quantitatively assessed based on a pre-determined sco- ring rubric and used regularly by both the training and test and evaluation communities (e.g., Bewley, Chung, & Girlie, 2003; Brown et al., 2017; Carbone et al., 2014; Headquarters DOA, 2016; Johnson & Kobrick, 1997;

© 2022. Published by AHFE Open Access. All rights reserved. 53

Johnson, McMeney, & Dauphinee, 1990; Son, Xia, & Tochinhara, 2010; Taylor & Orlansky, 1991). Although the military has recognized that per- formance of crews, teams, and units is essential to overall mission success (Goodwin, Blacksmith, & Coats, 2018; Turnage, Houser, & Hofmann, 1990), most of the team-based training is only qualitatively assessed by higher level trainers, or subject matter experts, and primarily in a binary fash- ion (i.e., pass or fail). While many of the aspects of individual proficiency may be generalized to the group level, the literature on military collective operational performance measurement and quantification is limited (Tur- nage, Houser, & Hofmann, 1990). To date, the authors are unaware of any published research and in practice methodologies that assess team marksman- ship skills and related mission performance in an objective manner in the field.

This current research developed a team level marksmanship assessment methodology, by leveraging previous research that developed measure- ments of individual marksmanship assessment methodology, by leveraging previous research that developed measurements of individual marksman- ship skills (Brown et al., 2019, Brown et al., under review). This new methodology took the established metrics to assess the entire marksman- ship process from approach to target discrimination, engagement, and transition, and expanded it to include collective measures of marksman- ship performance such as team communications, coverage strategies, and mission performance outcomes. Incorporating teaming into the quantifi- cation of lethality makes the assessment more representative of perfor- mance in close combat engagements, due to requirements of teamwork. In addition, this novel methodology is taking a significant step tow- ards establishing a quick technique to assess both individual shooting skills and collective close combat lethality performance of a military small unit.

METHODS

Participants The proof-of-concept assessment included forty-nine active duty infantry Sol- diers (all male). Five of the participants had sensor failures resulting in missing data and inaccurate quantification of team marksmanship performance. The- refore, this current analyses only included 13 of the 18 three-person fire teams, or 39 participants. Participants were between the ages of 18 to 29 years (M = 22.4, SD = 3.0), with 1.9 years of experience in the military on average (SD = 1.1). All were physically fit as judged by their self-reported physical fitness scores. All were qualified “marksman” through the Army Basic Marksmanship qualification process using the M4 carbine. Sixteen (41%) were Sharpshooters (score of 30-35 out of 40 on the standard mark- smanship test), twenty-two (56%) were Experts (score of 36+ and is the most skilled category), and one did not report a score. Three individuals (7.9%) were left-hand dominant, and 100% spoke English as their primary language. Distribution of experience level and marksmanship skills were approximately even across fire teams.

A Novel Methodology for Evaluating Military Teamwork 55

Figure 1: Diagram of TSS course layout, where Soldier fire teams are placed within the inner circle and engage targets around the 360-degree circle.
Original PDF Figure 1:

Test Procedures The TSS consisted of three-person fire teams completing a team marksman- ship course, wherein the team was tasked to detect, identify, and potentially engage targets over a period of approximately six minutes. In the scena- rio, each team was situated within a circle having a 2.5-meter radius which marked the boundaries of where they could maneuver. Surrounding the boun- dary circle was an outer ring of 28 light node targets set at 1.57-meters high (Figure 1). Each light node target was programmed by the research team to light up in a pattern that represented 3 states: dormant, threat, or non-threat. In order to maintain a consistent firing distance between the teams and tar- gets, light nodes were positioned 7.5 meters out from the center of the firing circle with each light node having 1.7 meters of space between it and adjacent nodes, while maintaining a 12.9-degree angle from the center of the boundary circle. The light node targets were scaled to approximately 75-meters.

Before the scenario began, each team had their equipment checked and calibrated, software zeroed their weapons and sensors per the manufacturer’s operating manual, were trained and tested on the target pattern discrimina- tion, and received a briefing from a member of the research team about the scenario task and conditions. A 5-minute planning period was allocated for each team to collaborate amongst themselves to establish and document their tactical strategy for the scenario.

Once this planning period ended, each team started the scenario. In an effort to simulate the volatile nature of firing engagements seen in operati- onal contexts, but often lost in the predictability of lab-tasks, the TSS was conducted in six successive segments that contained variations in the pre- sentation of targets by density (i.e., the number of targets being presented at a time), and/or identity (i.e., threat or non-threat target). Each segment

Table 1. Table of fires describing each segment of fire for the scenario.

Table available in the original PDF paper.

Segment

Description

Seg- ment

No. Targets

No. Threats

Display Time (s) Time (s)

1 52 52 3.5 60.5 1 threat target was quasi-randomly and consecutively presented per sector. 2 88 88 4.0 58.5 2 threat targets were concurrently presented per sector. 3 132 88 4.0 58.5 2 threats and 1 non-threat concur- rently presented per sector. 4 102 82 4.0 81.0 3 - 5 threats saturated one side with 1 non-threat presented on opposite side of circle concurrently. 5 132 88 4.0 58.5 2 threats and 1 non-threat concur- rently presented per sector. 6 90 60 4.0 39.5 2 threats and 1 non-threat concur- rently per sector, team member incapacitated.

lasted ~40–80 seconds and occurred in rapid succession. The table of fires can be found in Table 1, to include a description of each segment. Segment one was designed to have single target engagements across the team that were focused on the transition zones between potential sector areas to force com- munication to deconflict engagement between team members. The second segment increased workload by displaying two concurrent threats per pre- sentation with an increased exposure time was to allow for transition and engagement across multiple targets. The third segment increased workload again by introducing a go-no-go or target discrimination task. In the fourth segment, targets were presented as a roving saturation of threats, meaning three to five targets would simultaneously be presented in a single sector for 4.0s at a time and proceeded to move around the circle to the other sectors. While one sector was saturated with threats, the opposite side of the circle was presented non-threats only in order to try and induce team communi- cation of workload levels across the members. The fifth and sixth segment were same as segment three in target presentation, but the sixth segment was shorter and the emerging team leader was instructed by the research team that they were incapacitated and incapable of firing, which meant the remai- ning two team members had to coordinate coverage of the third sector of fire. Segment six required a change in strategy in order to successfully achieve the mission of full 360-degree coverage.

Test Apparatus and Measures In this research, participants engaged targets with a simulation M4 carbine manufactured by LaserShot, Inc., with an attached M68 close combat optic

A Novel Methodology for Evaluating Military Teamwork 57

Figure 2: Example light box target with LED screen displaying a shape that represents an enemy threat (left), placed on stands in a 360-degree circle (right).
Original PDF Figure 2:

(CCO) sighting system, and two side-mounted sensors (i.e., FN Expert opti- cal unit and an inertial measurement unit (IMU)). The CCO and FN Expert optical unit were mechanically zeroed to manufacturer standard. Shot data from the FN Expert weapon system were captured and processed using the FN Expert simulator and NOS pro software, with shot timing verification from the IMU data. A helmet mounted IMU also gathered data on scan coverage and engagement direction, however those data were not used in the current analysis. Light boxes with light emitting diode (LED) screens pro- grammed to display shapes were created and used as targets, and were affixed with diamond-grade reflectors in association with the FN Expert simulator system (Figure 2). A detailed account of how the system senses and generates shot data can be found in Brown and colleagues previous work on individual marksmanship assessment (2019).

Shot data from the FN Expert and IMU sensors were used to calculate established measures of individual marksmanship lethality from previous research (Brown et al., 2019; Brown et al., in press) including probabi- lity of hit (p(hit)) and shot accuracy. Mission performance outcomes were operationalized as the probability of total targets a team hit and proba- bility of total targets a team engaged, which were calculated from teams’ aggregated shot data in relation to the total number of targets presented per segment and the entire scenario. Insights into some of the qualitative aspects of team marksmanship were also captured in this study, including team com- munication and coverage strategies gathered from the planning documents and rater-observer notes. Team communication was classified into one of two categories: 1) having a communication plan, and 2) not having a com- munication plan. Sector strategy was also binned into two classifications: 1) implicit, or 2) explicit sectoring. Implicit sectoring included strategies that were implied but either not written down or laid out in a structured manner, such as teammate relative orientation and distancing. Explicit sectoring inclu- ded strategies that were based on concrete cues in the environment such as a

Table 2. Description of lethality measures, mission outcomes, and team strategies.

Table available in the original PDF paper.

Measure Description Probability of Target Hit [p(hit)]

Ratio of shots fired to target presentation.

Ratio of shots fired to number of target hits (within 500mm of target center). Shot Accuracy Distance of the shot hit coordinates from the target center (measured in mm). Probability of Target Engagement [p(engaged)] Communication Strategy Descriptions of communication themes identified from team planning documents and observations (e.g., cal- ling out target types, calling for help). Binary categories of having a plan vs. not having one were used in this analysis. Sector Strategy Descriptions of coverage and sectors of fire themes iden- tified from team planning documents. Themes were binned into two classifications (i.e., implicit sectoring based on teammate distance and explicit sectoring based on environmental cues).

Table 3. Comparison of individual accuracy and p(hit) lethality measure outputs between the ISS and TSS methodologies, showing no difference between results.

Table available in the original PDF paper.

Comparison

Measure ISS (Mean+SD)

TSS (Mean+SD)

Paired Student’s t-test Accuracy 353.9 + 69.6 358.5 + 147.8 t(37) = .21; p =.83 p(hit) .39 + .24 .34 + .11 t(37) = −1.29; p =.21

set number of targets relative to a designated target or external environment like the distant tree line, etc. Descriptions of these measures can be found in table 2.

Statistical Analyses Marksmanship skills were characterized by individual to ensure this novel team methodology captured similar outcomes as the individual shooting sce- nario (ISS) methodology described by Brown et al. (in press). The ISS and TSS lethality outcomes were compared using Student’s paired t-tests. Next, analysis of variances (ANOVAs) for each primary lethality marksmanship outcome were used to identify methodology sensitivity to differences in team marksmanship performance across fire teams. Finally, the effects of team stra- tegy performance on team marksmanship outcomes were assessed using two 6x2x2 mixed model ANOVAs, with scenario segment as the within-subjects variable, and sector strategy (implicit, explicit), and communication strategy (plan, no plan) as the between-subjects variables. The assumption of sphe- ricity was violated for p(engage), so the Greenhouse-Geisser correction was utilized. Tukey’s test was used for post hoc pairwise comparisons. Confidence intervals were set at 95% (alpha = .05).

A Novel Methodology for Evaluating Military Teamwork 59

Figure 3: Differences in mean shot accuracy across team (lower number indicates better performance). Each pair with ** indicates p < .01 and * indicates p < .05.
Original PDF Figure 3:

RESULTS

Individual Marksmanship Skill Characterization A comparison between ISS and TSS lethality primary outcomes was con- ducted in order to ensure that the new team level methodology can also characterize the individual team member skill levels. As seen in table 3, the two methodologies produce similar outcomes to characterize individual skills in marksmanship lethality.

Team Marksmanship Performance TSS methodology sensitivity to differences in team marksmanship perfor- mance across fire team was analyzed using two ANOVAs. Analysis of shot accuracy revealed a main effect of team, F(12, 65) = 7.52, p < .001. Post hoc analysis showed multiple pairwise differences as shown in figure 3.

Analysis of p(hit) revealed a main effect of team, F(12, 65) = 3.00, p < .01. Post hoc analysis showed that team 6 (M = .35, SD = .42) had a significantly higher ratio of hit as compared to team 4 (M = .14, SD = .32) and 2 (M = .13, SD = .28), (all pairs p <.05).

Team Strategy Performance Analysis of p(engage) revealed a significant main effect of segment, F(5, 45) = 47.8, p < .001. Post hoc analysis showed that segment 1 (M = .76, SD = .06) had a significantly higher engagement rate than all other segments (all pairs p < .05). Segments 2 (M = .66, SD = .04), 3 (M = .65, SD = .06), and 5 (M = .65, SD = .14) were significantly higher than segments 4 (M = .52, SD = .05) and 6 (M = .36, SD = .07) (all pairs p < .001). Segment 4 was significantly higher than segment 6 (p < .001). Sector and communication strategies did not significantly influence engagement rate.

Figure 4: Probability of hit per scenario segment as influenced by sector strategy type (top=explicit, bottom=implicit) and communication plan (teal=no plan, grey=yes plan).
Original PDF Figure 4:

Analysis of p(hit) also revealed a significant main effect of segment, F(5, 45) = 13.61, p < .001. Segment 1 (M = .35, SD = .12) had a significantly higher rate of hit than all other segments as seen in figure 4 (all pairs p < .001). Additionally, p(hit) had a main effect of sector strategy, F(1, 9) = 8.89, p = .02, where those who used the external environment cues for sectoring (i.e., explicit, M = .24, SD = .12) resulted in a higher probability of hit as compared to those who used teammate relative positions (i.e., implicit, M = .16, SD = .09). Team communication was trending towards significa- nce, F(1, 9) = 4.21, p = .07, where having a communication plan (M = .18, SD = .12), as compared to no plan (M = .23, SD = .12), resulted in a lower probability of hit. There was also a three-way interaction between communi- cation strategy, sector strategy, and segment, F(5, 45) = 3.92, p < .01, where teams without a communication plan performed better and more consistently in their shooting across the segments if they had an explicit sector strategy, but performed just the same as those with a communication plan if they had an implicit sector strategy, as seen in Figure 4.

DISCUSSION

This proof-of-concept achieved its initial goals of providing a quick, stream- lined assessment of both individual and team marksmanship performance,

A Novel Methodology for Evaluating Military Teamwork 61

quantifying team performance and strategies. The initial results presented here showed that the TSS method can characterize individual shooting skills equivalent to the baseline ISS method. Also, team marksmanship outcomes were compared across fire team group, finding that this methodology can successfully differentiate between small-group level shooting performance. The team strategy performance analyses results suggest that as the complexity of a firing engagement escalates over time, team marksmanship performance declines. Additionally, our results suggest that teams with a communication strategy in combination with an explicit (i.e., external cueing) sectoring stra- tegy have a lower hit ratio than those without a communication strategy. When the teams have an implicit (i.e., internal cueing) sectoring strategy, the type of communication strategy did not make a difference on mark- smanship performance. The additional cognitive burden of finding sector reference points to help communicate engagement intentions could poten- tially result in cognitive overloading during the segments with higher target density. Also, the greatest divergence in performance occurred during the final segment, where a member of the team is removed leaving only two individuals remaining. An implicit strategy may require less adjustment, simply utilizing non-verbal communication for body placement to relay sector coverage. Indi- vidual experience level could potentially influence the ability to dual-task and perform well on the segments with higher complexity, and was initially con- sidered as a covariate but was not a significant influencer and was removed from modeling. However, future analysis should explore quantifying team experience level and cohesion as potential modulators of performance.

Some limitations of this new methodology includes sample size, equipment, and metrics. Future studies will strive for a larger sample size to provide addi- tional data for verification of the full set of metrics and outcomes. Equipment limitations include the use of the weapon simulator, which utilizes a carbon dioxide chamber to provide the feeling of recoil and muzzle rise, but can only achieve ~30 percent of actual live-fire weapon dynamics. This reduced recoil may change some of our observations (i.e., less fatiguing than live fire resulting in better weapon handling/shot outcomes). However, the simulator allowed for a quick assessment and setup which would have been unachieva- ble in live fire due to range safety limitations. Another limitation is the lack of target sensor to discriminate between team members during engagements, reducing our ability to assess decision making at the individual level. Future designs will augment the TSS equipment to include target sensor detection capabilities.

This novel team marksmanship assessment is quick to execute (6 minutes total), yet still provides sufficient information for accurate characterization of individual shooting skills and assessment of mission-related team perfor- mance. While initially intended to support a specific human performance research program, the novel capability achieved can be extended to the acqui- sition/test and evaluation (e.g., evaluation of equipment such as body armor, helmets, augmented reality, etc.), and training communities (e.g., evaluation of training procedures for marksmanship or teamwork). This novel metho- dology is also portable for field use, and flexible for scenario adjustments to match skill level and training requirements.

REFERENCES

Bewley W.L., Chung, G.K., & Girlie, C. D. (2003). Research on USMC Marksmanship Training Assessment Tools, Instructional Simulations, and Qualitative Field-Based Research. UCLA CSE/CRESST N00014-02-1-0179. Los Angeles, CA Brown, S.A.T., Villa, J., Mitchell, K.B., Hussey, E.K., & Ramsay, J.W. (2019). Simu-

lated marksmanship performance methodology: Assessing lethality, mobility and stability during a military field training exercise. In: Cassenti D. (eds) Advs in Hum Factors Sim, 958, Springer, Cham. Brown, S. T., McNamara, J. A., Mitchell, K. B., & Eddy, M. D. (2017). Development of a building clearing methodology for the assessment of soldier. Journal of Science and Medicine in Sport, 20, S67. Brown, S. A., Christopher, J. J., Villa, J. D., Goodenough, R. R., Hancock, C. L.,

O’Donovan, M. O., & Mitchell, K. B. (in press). Methodology to assess indi- vidual shooting skills that is predictive of squad level performance in a close combat training engagement. In Proceedings of the Human Factors Society Annual Meeting. Carbone, P.D., Carlton, S.D., Stierli, M., & Orr, R.M. (2014). The impact of load carriage on the marksmanship of the tactical police officer: a pilot study. Journal of Australian Strength and Conditioning. 22(2), 50–57. Goodwin, G.F., Blacksmith, N., & Coats, M.R., (2018). The science of teams in the

military: Contributions from over 60 years of research. Am Psyc, 73(4), 322–333. Headquarters DOA. (2016). Rifle and Carbine. TC 3-22.9. Washington, DC. Headquarters DOA. (2019a). Army modernization strategy: investing in the future.

Washington, DC. Headquarters DOA. (2019b). Training and qualification – individual weapons. TC 3-20.4. Washington, DC. Headquarters DOA. (2020). Holistic Health and Fitness Testing. ATP 7-22.01.

Washington, DC. Johnson, R.F., & Kobrick, J.L.(1997). Effects of wearing chemical protective clothing on rifle marksmanship and on sensory and psychomotor tasks. Mil Psychol, 9(4), 301–314. Johnson, R.F., McMenemy, D.J., & Dauphinee, D.T. (1990). Rifle marksmanship

with three types of combat clothing. In Proc Hum Factors Ergon Soc 34th Annual Mtg, 1529–1532. Son, S., Xia, Y., & Tochihara, Y. (2010). Evaluation of the effects of various clothing conditions on firefighter mobility and the validity of those measurements made. Jnl of the Human-Environment Sys, 13(1), 15-24. Taylor, H.L., & Orlansky, J. (1991). The effects of wearing protective chemical war-

fare combat clothing on human performance. No. IDA Paper P-2433. Alexandria, VA: Institute for Defense Analysis. Turnage, J., Houser T., & Hofmann D. (1990). Assessment of Performance Measurement Methodologies for Collective Military Training. ARI Research Note 90-126. AD-A227 971.

Cover of the proceedings volume Digital Human Modeling and Applied Optimization

Explore the complete digital volume

Digital Human Modeling and Applied Optimization

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-22-3

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →