Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Soldier Perspectives in AI and Autonomous Systems

U.S. Army Combat Capabilities Development Command Armaments Center, Picatinny Arsenal, NJ 07806, USA ↗
Publication year
Digital Object Identifier10.54941/ahfe1006236

Abstract

Artificially intelligent autonomous systems are a growing component of the Army’s battlefield assets. However, there is a realization that design of these systems, especially armaments, must incorporate the user’s perspectives. These perspectives include desired features, concept of operations, limitations on the intelligence and autonomy of the system. DEVCOM Armaments Center’s Tactical Behavior Research Laboratory gathered information from over 130 Soldiers about their insights into artificial intelligence and autonomous weapons in operations. The survey was distributed in person to an international group of Soldiers (predominantly US) following force-on-force exercises and also distributed online. This presentation will give an overview of the guidance that these Soldiers provided engineers and human systems integrators who are involved with developing intelligent autonomous weapons.

INTRODUCTION

The U.S. DoD is directing a great deal of resources toward development of Artificial Intelligence (AI) aids for the military (Department of Defense, 2018; Smith, 2019). The capabilities of artificial intelligence are envisioned to give individual Soldiers unprecedented power. Given the novelty of artificial intelligence aids and the potential for unintended consequences, there have been efforts to provide guidelines for the critical testing, evaluation, verification and validation of these capabilities (Haugh et al., 2018), as well as the responsible development and use of artificial intelligences (DoD Department of Defense, 2022; DoD Responsible AI Working Council, 2022; Dunnmon et al., 2021).

Of key importance in determining the effectiveness of the artificial intelligence are the users’ subjective responses to the artificial intelligence aid. Smith (Smith, 2019) has put forth the Human-Machine Teaming Framework for Designing Ethical AI Experiences in an effort to support development of systems that are “accountable, de-risked, respectful, secure, honest, and useable.” Within this framework, a trustworthy AI ensures that humans are always in control, able to monitor and control risk, and “preserve the human

© 2025. Published by AHFE Open Access. All rights reserved. 184

Soldier Perspectives in AI and Autonomous Systems 185

responsibility for final decisions that affect a person’s life, quality of life, health or reputation.” The degree to which users deem an AI trustworthy is therefore determined by their perception of these system characteristics.

The importance of responsible creation, testing and evaluation, and usage of artificial intelligence aids takes an even more critical role with the inevitable emergence of autonomous weapon systems (Office of the Secretary of Defense Policy Force Development and Emerging Capabilities, 2023; Office of the Under Secretary of Defense for Policy, 2023). The rules of engagements and risk protocols for autonomous weapon systems are yet to be crafted. However, it is clear again, that much of the guidance depends on the subjective responses of U.S. Commanders toward deployment of autonomous weapon systems. For example, an article by Trotti and colleagues (Trotti et al., 2022) posited several questions: What level of risk are U.S. Commanders willing to accept in deploying autonomous systems? What is the acceptable collateral damage risk for autonomous weapon systems? Is it more or less acceptable than the risk for a human operator? Who decides when, where, and how often autonomous weapon systems are deployed? Who is responsible for the actions of autonomous weapon systems: the operator, commander, or programmer who writes the targeting algorithm? How are decisions evaluated “right of boom” (i.e., in the wake of an attack)? What degree of AI explainability should be required? (Trotti et al., 2022).

THE PRESENT STUDY As an initial investigation, the present study queried U.S. Army Soldiers and others to systematically gather answers to these questions and others, related to autonomous weapon systems. Data was gathered from Soldiers participating in several Army live fire, force-on-force, range exercises, and online from December 2023 to September 2024 in the U.S. Because of the nature of the data collection, limited demographic information could be collected. All procedures were approved by the DEVCOM Armaments Center Human Research Protections Program. From this population, four data set sets from four survey instruments were derived:

1. Lethal Autonomous Weapon Systems Questionnaire (N = 88). 15-item free-text questionnaire asking for opinions and insights and recommendations to give guidance to engineers developing fully autonomous lethal armament systems. 2. Robotic Assets Questionnaire (N = 140). This 10-item Likert scale questionnaire asked Soldiers their level of agreement with statements on the use of robots on the battlefield (not necessarily as weapons). 3. AI Confidence Questionnaire (N = 22). This 18-item questionnaire asked Soldiers to indicate the percentage level of confidence that an artificial intelligence would need to report for the Soldier to take action. This question was repeated for three different targets (Human, Robotic Quadruped, Vehicle) at different phases across the kill chain. 4. AI Reported Confidence Levels Questionnaire (Weapons Tight (WT)/ Weapons Free (WF)) (N = 10). This 16-item pilot questionnaire asked

Soldiers to indicate the percentage level of confidence that an artificial intelligence would need to report for the Soldier to take action in several scenarios, under Weapons Tight and Weapons Free rules of engagement.

The purpose of this project is to provide engineers and developers initial Soldier feedback into development of lethal autonomous weapon systems, into the characteristics that Soldiers want to see in these armaments, or limitations they feel these systems will exhibit. The results are only a start— one primary goal for the work is to merely find out the right questions to ask. This chapter presents a high-level summary from the preliminary analyses from these four surveys.

HIGH-LEVEL SUMMARY AND CONCLUSION Results from these four questionnaires (Lethal Autonomous Weapon Systems Questionnaire, Robotic Assets Questionnaire, AI Confidence Questionnaire, AI Reported Confidence Levels Questionnaire (Weapons Tight (WT)/Weapons Free (WF)) provide preliminary and valuable insights into Soldier perspectives regarding lethal autonomous armaments. Inasmuch as this population can be thought as representative of Soldiers at large, a few observations can be made and possibly generalized. Again, the free text responses to questions are found in the Appendices; readers can review this material and draw their own conclusions.

One unsurprising observation is that there are variabilities in responses about the willingness to employ lethal autonomous weapon systems. There was a range of responses, from the absolute responses that they would never employ such as system, to acceptance of only human oversight or permissions or trigger pulls to full acceptance of AI decisions for identification, targeting and engagement. There was a general greater comfort with using AI for non- lethal tasks, such as target detection, identification, and surveillance. Further data gathering and discussions are needed to come to a DoD consensus on the acceptability, guidance, and doctrine on possible use cases for lethal autonomous weapon systems.

In general, development of artificially intelligence-enabled autonomous armaments is encouraged by respondents. Soldiers expect these systems will be force multipliers, increasing lethality and survivability and decreasing workload; Soldiers, at this time, do not consider them to be possible battle buddies, teammates, or substitutes for flesh and blood fighters. Respondents seem less confident in the mobility of AI autonomous systems and their capabilities for providing security. There may be more interest in non-lethal sensing capabilities for increasing accuracy and distance in finding possible targets.

Perceived weaknesses revolved around technical reliability, malfunctions, weather, battery life, and network connectivity. Of note is the lack of flexibility and adaptability in a dynamic environment with unexpected scenarios. Developers need to be mindful that AI and autonomous components are not under evaluation separately from the entire system— Soldiers evaluate the entire function of a weaponry system, not just the AI

Soldier Perspectives in AI and Autonomous Systems 187

or the autonomy. Specifically, ease of use, reliability, mobility, power, weight, ruggedness, resistance to malfunction are in the forefront of Soldier concerns, just as with any other conventional armament. Obstacles to full acceptance for some, consist of doubts about safety to friendly troops, avoidance of collateral damage, and actual capabilities for lethality. This reticence is also reflected in responses which indicated that AI systems would require both physical proximity and increased vigilance to ensure the system was functioning correctly.

In addition to conventional necessary characteristics, the survey revealed requirements unique to AI-enabled autonomous armaments versus conventional weapons. Utility of the fully autonomous lethal weapon system hinges on 1) its ability to accurately sense and distinguish non-targets and possible targets; 2) the ability of the system to convey its own system status, the output of its algorithmic processes, and how it arrived at that output, that is, explainability (Haugh et al., 2018); and 3) the ability of Commanders and Operators to control the armament throughout the kill chain, especially with respect to emergency stops to prevent collateral damage or friendly fire.

These Soldier-voiced requirements resonate with requirements stemming from DoD 3000.09 for systems to have means for Commanders and Operators to exercise appropriate levels of human judgement (Office of the Secretary of Defense Policy Force Development and Emerging Capabilities, 2023; Office of the Under Secretary of Defense for Policy, 2023). They also align with the concepts adopted in the wider global ethical community for meaningful human control over autonomous weapon systems (Amoroso & Tamburrini, 2021; Canellas & Haga, 2015; Horowitz & Scharre, 2015; Riebe et al., 2020). Readers may recognize that these Soldier responses as related to ethical concerns over whether AI can show the proper characteristics of distinction and proportionality, required by international laws in the conduct of war (Amoroso & Tamburrini, 2021).

Based on aforementioned ethical concerns, a high bar is set for these armaments. In free text responses, participants reported requiring high accuracy in friend/foe identification—at least 90%. 100% accuracy in identification for friendly forces and civilians, with 0% chance of friendly fire, has been offered for thresholds. When asked specifically about targets and kill phases, at least 60% reported AI confidence was needed for Soldier actions, with the mean of 70–80% confidence level reported as required AI confidence level. However, again, the small number of respondents means that these thresholds should be viewed with caution; further investigation is warranted.

A WAY FORWARD More information is needed. Another theme in responses concerned the Soldier demand for information about the systems. Minimally, this information should include how the system identified and distinguished friend from foe, how accurately the system decides, past success and failure rates in similar missions, information about capabilities, and safe and effective/maximum ranges. Clearly, decision-makers in particular, requested

direct experience with the armament system in many different and adverse operational scenarios. Following that, information from Soldiers with direct experience with the systems in many different and adverse operational scenarios was most valuable. Results from testing and evaluation in different and adverse operational scenarios conducted by disinterested third parties need to be made available to Soldiers making command decisions as well. A critical part of the way forward is to endeavor to create these databases and make them readily available to Commanders and Operators.

Generally, Soldiers seemed to welcome the chance to provide input toward development of these and other systems. Involvement of Soldiers in as many aspects of the acquisition as possible is indicated. In terms of research methods, errors made in this data gathering point to needed improvements for future Soldier inquiries about these types of systems. Future data gathering should differentiate more between semi-autonomous and fully autonomous systems, as well as ask more pointed questions relevant to the two different configurations. For further data collections, clearer/more easily read questions need to be crafted—questions need to be simpler and easier to read and write answers for. Practical questions that are tied to familiarity/use of a specific autonomous lethal weapon system are needed, so that responses are based on actual use of the system. Questions about specific functions of specific systems after hands-on operations, or at least demonstrations, are needed. These approaches should decrease reliance on a Soldier’s abstract theoretical speculations and give more accurate insights into their views on actual systems.

Armed with improved methods, researchers should further probe for Soldier guidance on development of lethal autonomous weapon systems, including Warfighters from the other U.S. DoD Services. Questions include optimal information inclusion/presentation to Commanders and Operators, optimal e-stop and other controls for overwatch, more specific factors that underlie decisions about uses for AI-enabled systems. These qualitative approaches should be complemented by a parallel laboratory research program where experimental controls could yield further insights about factors affecting Soldier interactions with lethal autonomous armament systems. This information in turn, will assist engineers and developers configuring Soldier-centered designs for increased lethality. Therefore, further research is warranted.

ACKNOWLEDGMENT

The authors would like to acknowledge all Soldiers for their service to the country. We would like to thank the Calibrated Trust Measurement and Evaluation, 10X, Army Expeditionary Warrior Experimentation researchers and staff for their support. Also, much thanks are due to our lab mates who worked on the several projects included in this work: Kenneth Short, Joanne Arias, Nasir Jaffery, Keith Koehler, Mark Germar, Larry D’Aries, and Gladstone Reid.

Soldier Perspectives in AI and Autonomous Systems 189

REFERENCES

Amoroso, D., & Tamburrini, G. (2021). Toward a Normative Model of Meaningful Human Control over Weapons Systems. Ethics & International Affairs, 35(2), 245–272. Canellas, M. C., & Haga, R. A. (2015). Toward Meaningful Human Control

of Autonomous Weapons Systems through Function Allocation 2015 IEEE International Symposium on Technology in Society (ISTAS). Department of Defense. (2018). Summary of the 2018 Department of Defense Artificial Intelligence Stratey: Harnessing AI to advance our security and properity. DoD Department of Defense. (2022). Responsible artificial intelligence strategy and

implementation pathway. DoD Responsible AI Working Council. (2022). U.S. Department of Defense Responsible Artificial Intelligence Strategy and Implementation Pathway. Dunnmon, J., Goodman, B., Kirechu, P., Smith, C., & van Deusen, A. (2021).

Responsible AI Guidelines in Practice. Haugh, B. A., Sparrow, D. A., & Tate, D. M. (2018). The Status of Test, Evaluation, Verification, and Validation (TEV&V) of Autonomous Systems (IDA Paper P-9292). Horowitz, M. C., & Scharre, P. (2015). Meaningful Human Control in Weapon

Systems: A Primer (Project on Ethical Autonomy CNAS Working Papers, Issue. Office of the Secretary of Defense Policy Force Development and Emerging Capabilities. (2023). Autonomous Weapon Systems Review and Approval Process Implementation Guidance v.1. Office of the Under Secretary of Defense for Policy. (2023). DoD Directive 3000.09

Autonomy in Weapon Systems. Riebe, T., Schmid, S., & Reuter, C. (2020). Meaningful Human Control of Lethal Autonomous Weapon Systems. IEEE Technology and Society Magazine, December 2020, 36–51. Smith, C. J. (2019). Designing Trustworthy AI: A Human-Machine Teaming

Framework to Guide Development. Trotti, C., Wetzel, T., Nurkin, T., & Hu, E. (2022). What does the future of autonomous warfare look like? Four critical questions, answered.

Cover of the proceedings volume Human-Computer Interaction & Emerging Technologies

Explore the complete digital volume

Human-Computer Interaction & Emerging Technologies

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-964867-71-7

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →