Autonomy and AI are the future of warfighting, but what about the human-in-the-loop? The Integrated Enablers project, as part of a larger ongoing autonomy initiative, was focused on answering that very question in an environment of UxS rapid iteration and system evolutions. The Test Resource Management Center (TRMC) called upon a group of contractors, including my employer Cognitive Performance Group, to conduct data collection and analysis to help evaluate the systems under test against the objectives given to them by the Defense Innovation Unit (DIU). Our problem statement was as follows:
How might we ensure UxS Command and Control (C2) systems enable the needs of their Operators, while allowing for the Collaborative Autonomy (CA) to achieve maximum mission effectiveness and efficiency.
As a 3rd party to this effort outside of the vendors and performers, our Team was uniquely positioned to conduct non-partial, data-driven observations and collection to determine current state of C2 user interface (UI) systems as well as evaluating how well they support the needs of the Operator.
Our portion of the Integrated Enablers effort grew out of the need for a White Cell to collect this impartial data in order to facilitate, document, and help determine a way forward for the C2UIs and CAs being tested, while ensuring they were building towards an operationalized MVP. Our team's efforts were focused primarily on the C2UI piece, originally looking at three seperate UIs under test, which were later down selected to two primary UIs. The intent was to understand their current state, and use observations and testing data to give the Operators a voice on how the systems could be improved.
Our team used Observer Reporters (ORs) to collect live data from Operators using each C2UI, then used those data to create scorecards for consideration, and conduct larger thematic analyses to uncover potential avenues for improvement with these systems. Our efforts culminated with a dissemination of our raw observations, the roll-up scorecards, and a final report that outlined the biggest opportunities for improvement based on Operator feedback. Additionally, during the Capstone event, our team provided UX heuristic-style suggestions based on a live-collection framework.
As a rapidly evolving effort, the approach our team used for the Integrated Enablers project shifted focus several times from the formative stages in December 2024, to the Capstone event in August 2025. The focus on warfighter use of the systems, combined with evluation of the autonomy, was consistent. The project initially started as a current-state documentation effort through gathered data for 3 C2UIs and 2 CAs. I was primarily responsible for the tool creation and real-time observation data collection efforts of the C2UIs
The data collection framework employed was an iterative process, beginning with an initial attempt to gather qualitative data to create a pass/fail scorecard of 10 objectives, with 32 total sub-objectives present in the first testing event Test Plan. The collection sheet used a combination of raw data entered line-by-line by each Observer Reporter (OR), then compiled under their respective objective to answer the pass/fail question. This initial framework is illustrated below.
Initial OR Data Collection Tool
This framework hit the immediate goal of gaining an understanding of system function, but lacked the level of robust observations and instantly accessible feedback for White Cell and the performers. To address this need, I created and iterated on a new OR collection tool that employed Excel automation and more robust formulas to quickly capture data in real-time, automatically grabbing the time stamp of each observation to aid in post-trial analysis. This allowed the ORs to simultaneously narrow their focus to the instances seen that triggered the required data, while allowing their focus to be on the mission operation as a whole.
Automated Timestamp OR Collection Tool
Once the revised framework's potential to aid real-time situational awareness was proven at the second testing event, I iterated on the design to deepen the automation features. This allowed an expansion of the observation collections, using a newly developed series of trigger phrases to auto-populate important testing events, while simultaneously generating a real-time scorecard to capture these observations. This was done with a Task Success framework developed by our team, that allowed us to map the DIU objectives seen based on the actions and tasks the Operators undertook. It also allowed us to notate when there was an error or issues during these events, to automatically generate a numerator and denominator on our scorecard to create a final task success completion percentage. While this framework would have benefited from earlier testing to help calibrate and refine the relevant tasks, it was still an effective method to capture the broadstrokes of Operator task successes within the C2UI systems. The most recently employed version of this tool, broken down by it's component parts, is below.
Data Collection & Analysis Tool (DCAT)
Data Collection & Analysis Tool (DCAT) Task Success Rate
Data Collection & Analysis Tool (DCAT) Scorecard
The final area our team collected on during the Capstone event was an adaptation of a traditional UX Heuristic Evaluation. I developed a tool in the same vein as the automated DCAT shown above, that allowed ORs to capture real-time heuristic violation data while observing event Trials. This allowed us to rapidly identify the largest categorical issues present in the two remaining C2UI systems, in an effort to help provide actionable suggestions for future development. These live heuristic evaluations were done by multiple observers and then the largest, most pervasive issues were identified to focus the analysis effort post-event. The collection tool and sample dashboard are below:
Heuristic Evaluation & Analysis Tool (HEAT)
Heuristic Evaluation & Analysis Tool (HEAT) Dashboard
The primary results of this effort were aiding the project decision-makers on downselection of C2 systems, providing actionable feedback to developers to improve their systems, and capturing a performance baseline to aid in future operationalization efforts of the C2/CA systems as a whole.
Warfighting is increasingly moving towards autonomous systems, but the need for tailoring to the human-in-the-loop is paramount to success. Our live data collection and analysis for autonomous vehicle C2UIs helped enable a downselection process, as well as an iterative improvement cylce for developers.
Air Force Development Team (DT) boards are responsible for selecting the upper echelon of Air Force officers. Through a research effort combined with system design based off user design criteria, my Team and I improved both the efficiency and efficacy of the DT board process.
Innovation is the conduit for so many meaningful ideas, but it is also important to identify programs that fall short of their proposed intent. Enter MOTAR, a research-heavy initiative culminating in a business case analysis lite to inform decision-makers of the Air Education and Training Command.
Maryland State Library knew their user experience, particularly surrounding navigation, was flawed and wanted data to back it up. Our Team siezed the opportunity to create an updated navigation structure to go along with a modernized UI.
Copyright by John Roberts