T&E as a Continuum Background
Test and Evaluation as a Continuum
Building the future Joint Force that we need to advance the goals of the National Defense Strategy requires broad and deep change in how we produce and manage military capability.
Shifting the Traditional Testing Paradigm to a “Testing Continuum”
The T&E as a Continuum builds off the DoD 2018 Digital Engineering Strategy making T&E an integral part of the Systems Engineering and Mission Engineering processes.
T&E as a Continuum Attributes
Capability-Outcome Focused Testing
- Often called “mission-focused testing” – focus as early as possible on the performance of the military capability when fielded.
- “Test like we fight” – ensure early CT & DT engineering and tech verification testing provides information that directly supports evaluation of performance and risk in a mission context culminating in OT.
- Supports future evolution of discrete KPPs & KSAs to “system behaviors” enabling rapid evaluation against those desired system behaviors within the context of the model-based environment.
- Focus Cyber contribution on Operational Resilience – moves the discussion towards adverse impacts significantly expanding from the “controls” approach to containing impacts of cyber threats.
Agile, Scalable Evaluation Framework
- Responsive to Decision Space across the capability lifecycle from ME through O&S – develop the foundational and holistic framework before program initiation.
- Continually update, identify information & test infrastructure needed and how it will be obtained using ME, SE, and T&E activities and tools.
- Framework should be adaptable and scalable across the six AAF pathways – tailorable to pre-Program of Record S&T, P&E efforts, and integrated across DT & OT.
- Should include information to perform VV&A of M&S and other elements of MBSE – development within a model-based environment helps manage integration complexities across sub-system, system, and, especially, SoS elements.
- Holistic framework underpins a consistent “informed risk management” methodology, providing improved understanding of “as is” system performance against the threat environment.
Enhanced Test Design (ETD)
- Expands recent efforts to improve Integrated Testing and better supports early evaluation of performance in a mission context.
- ETD incorporates VV&A across the whole of capability evolution to provide initial data informing whether desired capability outcomes can be achieved.
- Develops a robust evaluation of human-system integration focusing on demonstrating the trustworthiness necessary for warfighter acceptance and utilization of AI/M-enabled systems.
- ETD uses a comprehensive “design-test-validate” approach assessing evolving “system behaviors” enabling capability developers to adapt designs driven by changing operational environments.
- Incorporates appropriate Scientific Test and Analysis (STAT) techniques to gain efficiencies and help enable rigorous evaluation of test results.
T&E as a Continuum Enablers
Robust Live, Virtual, Constructive (LVC) Environment
- LVC approaches address significant challenges with conducting live open-air testing of complex and sensitive DoD systems.
- Increased use of M&S and other constructive approaches, spanning the spectrum of threats and operational environments, is essential to obtaining a comprehensive understanding of systems’ performance.
- LVC approaches enable Systems-of-Systems (SoS) testing – an essential requirement supporting evaluation of emerging concepts and improved capability validation.
- Requires continued evolution of test ranges and facilities to incorporate combined LVC capabilities – this evolution will entail a “shared” infrastructure within the DE environment.
- Increased emphasis on LVC does not diminish the need for live, open-air testing to evaluate key aspects of systems’ performance.
Model-Based Environment
- The most critical enabler of a continuum of testing:
- Require a digital backbone and model-based approaches to manage the continuum of T&E activities.
- Integrates T&E activities with ME and SE activities conducted in parallel.
- Manages, curates, and analyzes data generated for all ME, SE, and T&E activities.
- Incorporates the agile, scalable evaluation framework and helps assure consistency with the full range of ME, SE, and T&E activities.
- Supports transition to a “model-test-validate-design-test” process providing early and continual information on expected mission capability.
- Uses “digital threads” displaying progress in modelling system performance beginning early in ME through the potential development of high-fidelity “digital twins”.
- Includes realistic representations of the joint battle space within which systems will operate.
"Digital" workforce
- Underpinning adoption of the model-based environment requires a “digital” workforce that is savvy with the processes and tools associated with MBSE, Model-Based T&E (MBTE), and other model-based processes.
- Workforce transformation has been initiated at an initial level with the revamping of the DAWIA certification process and associated Functional Domain credentialing.
- The DoD T&E Functional Integration Team (FIT) is leading efforts for the T&E community regarding changes to the overall DAWIA certification process.
- Uses “digital threads” displaying progress in modelling system performance beginning early in ME through the potential development of high-fidelity “digital twins”.
- Additional efforts to obtain and develop within the DoD the technical and management skills needed to lead and manage a continuum of ME, SE, and T&E activities, conducted in parallel using model-based approaches, are necessary.