Our Client is an Israeli military systems and technology provider. Their Unmanned Aerial Vehicle (UAV) systems operate multiple missions that include tactical and military operations with a very high standard of control and precision.
Our Client releases regular software updates and patches to ensure optimal UAV performance for their clients. Because of the mission-critical nature of these updates, the client needs to accelerate their development and testing cycles.
The Client develops multiple projects with “System-of-Systems” engineering, which is tested as part of the development cycle. A complete acceptance test is conducted at the system level as each cycle completes.
To better serve the needs of their customers, the Client wanted to:
Existing testing systems were too restrictive and reliant on manual processes that were slow, inefficient and completely unworkable under a continuous delivery model.
Initially, Qualitest implemented a test automation tool alongside their testing services. This allowed the client to streamline their test cycles and increase the rate of approvals for each subsequent version deployment.
At the same time, Qualitest used their experience to take the lead on several key processes. Working alongside the Client’s development team we were able to design the strategy, process and framework that would be used to automate testing. We were also able to assist with code reviews on deliverables and to provide technical accountability, applying industry best practices to raise overall standards of testing and deliverables.
To help the Client understand the improvements being made, Qualitest distributed weekly reports and bi-weekly progress reviews, enabling continuous visibility of automation scripts and delivery status. These were accompanied by weekly progress reviews with the development team that highlighted main achievements, issues, needs or blockers, as well as outlining the key tasks planned for the next stage of the project.
“We developed automatic tools and mechanisms to improve quality and speed, enable scalability and provide continuous visibility as well as achieve DO-178C certification.”
The Client operates two distinct Command, Control, Communications, Computers and Intelligence (C4I) product lines. Although very different, Qualitest analyzed both to identify technical similarities that could be used as process building blocks. These would then be used to integrate with the specific process or scenario on each product.
For the Client, this resulted in a reusable code infrastructure comprised of one main suite of ePF with tens of reusable scenarios. Each scenario could be applied on both major product lines (subject to minor changes) with a coverage of 80% of the Sanity processes and an increased percentage of regression and ATP coverage.
Further improvements were made by abstracting data input into individual scripts into flat .csv or .tsv files. This abstraction enables the Client’s automated solution to be data-driven so that different test runs can explore different combinations of test data without needing to alter the actual test scripts.
Automatically testing designated software features was just part of the Client’s challenge. Acceptance testing also needed to address the full aerial system and environment of the UAV.
Qualitest incorporated multi thread monitoring functions that runs in the background of each automatic test case and logs all abnormal events as defined by the Client, such as loss of control, crossing pitch and roll boundaries or there is a system failure in one of the flight critical sensors.
By introducing these monitors, Qualitest was able to reduce the manual oversight previously required for testing. The tool not only keeps track and record of the main tested component but also zooms out to validate the entire system and environmental state. The monitoring solution was incorporated across the Client’s software – GCS reports and display, mechanical line test station SW etc. – all crucial parts of the tested system.
Once processes and tools were aligned, the new automation solution underwent a Tool Qualification Process to confirm it meets DO-178C definitions. The Client can now prove to aerospace certification authorities (FAA, EASA and CAAI) that their commercial software-based aerospace systems adhere to industry standards.
The new scenarios built by Qualitest now run in software development labs and system labs, allowing for repeatable tests to be executed extremely quickly as part of the standard testing cycle. With the new tools and processes in place, the Client’s test effort has now shifted, from repetition of sanity and regression testing to testing of major changes and testing of complex and hard to retrace defect paths.
“The Client was able to increase testing scope by 500% and to shorten regression cycle time by 150 working hours.”
Throughout the project and beyond, the Client has realized some significant benefits, including:
