Non-regression testing

Non-regression testing (NRT) is an approach of software testing. The purpose of non-regression testing is to verify that, after introducing or updating a given software application, previous software functions have not been compromised. This is aimed at preventing software regression, where adding new features results in software bugs.

Software development

The development process of a software application can be divided in several steps, where each step culminates in a new software version, containing a certain number of new features. This process continues until the final release, when all the contents that satisfy the customer’s requirements included in the software. As the complexity of software architecture grows up, the probability of bugs increases. Bugs occur when the software code is modified for two main reasons:

• new procedure in conflict with an old one;
• modification to upgrade a pre-existing procedure.^{[citation needed]}

Usually, the occurrence of software bugs can result in unexpected delays to the project. Due to time-to-market restrictions, the validation phase of software functionalities must be well organized and efficient so that this phase is as short as possible. In this context, non-regression testing provides a systematic procedure for fast and efficient validation and discovery of bugs within the software architecture.^{[citation needed]}

How to perform a non-regression test

A non-regression test can be performed according the following steps:

1. Define a benchmark software release;
2. Define a set of routines able to stimulate as many functionalities as possible of the software;
3. Launch these routines on both software (the benchmark and the new release under test) and acquire data which represents their behaviour;
4. Analyse this data with a post-processing tool, able to provide statistic results;
5. Report the outcome.

Exploratory testing is performed following similar steps, but it differs from NRT concerning their analysis phase and focus, searching, hence, for different results and conclusions. The NRT aims to check if any undesired behaviour comes out after the last modifications applied to the software. There, the new behaviour of the application is previously known, making possible the identification of an eventual regression (bug). Exploratory testing, on the other hand, seeks to find out how the software actually works, conciliating simultaneous testing and learning, and stimulating testers to create new test cases.^[1]

Regression and non-regression testing

While the intent of regression testing is to assure that a software bug has been successfully corrected by retesting the modified software, the Non-Regression Testing aims to verify that no software bugs have been introduced after the software is updated.^[2] In general, this difference of definitions can be assumed as based on the outcome of each test.

When a new software version is released without any new features relative to the previous version (i.e. the differences between the two versions are limited to bug fixes or software optimization), both the releases are expected to present the same functionalities. In this case, the tests applied to both versions are not expected to result in different behaviours, but only to assure that existing bugs have been fixed and no new bugs have been introduced. This testing methodology characterizes regression testing.

On the other hand, when the new release presents new functionalities or improvements that lead the software to behave in a different way, then the tests performed on the previous and new version can result in:

• desired differences, related to an expected new behaviour; and
• undesired differences, which indicate a software regression generally caused by a side-effect bug.

In this case non-regression testing is appropriate.

Who performs the non-regression testing

Once the customer has set all the requirements, the supplier will introduce all the contents, release by release, until the final one. In this context, NRT can be performed by both the customer and the supplier.

It can be made by the supplier as a beta testing service to guarantee a higher quality product with a very low percentage probability of bugs. Basically the client is equipped with simulation environment that enables an easy way to perform routines and acquire data. In case of regression the supplier, owing the know-how, can quickly solve the problem and avoid releasing a malfunction software version to the customer.

On the other hand, NRT can be performed by the customer as an acceptance testing in order to prevent his final product from damages and eventually charge the supplier for the mismatch with requirements. Moreover, the customer, having a reduced know-how about the software structure, can perform the NRT as a black-box testing and, after meeting a regression, refuse the new software release.

How to define a good non-regression testing strategy

Automate regression testing is not always possible, or not always economically viable in terms of maintenance costs for automated testing.

In the case of manual testing the challenge is to identify the relevant tests to minimize the testing effort while maximizing the coverage of regression risks. However, to avoid to missing regression the test strategy should be based on facts.

To have these facts, the analysis of the application (code, ....) and the comparison of each version can help to identify all the changes and risks. The difficulty is to get an view of these risks that is usable for functional testing: beyond the modified file it is more important to assess the impact on existing functionalities.

To improve this analysis, a solution is to take the "footprint" of each test on the application (what is executed in the application by each test). This footping is the link between code modules and functional test scenarios. Once this link is established, it is possible to know exactly what is covered by a particular test.

Thus, when a new version has to be tested, it is possible to identify which tests will cover all the risks regression based on changes to the application. Defining an efficient strategy for regression testing becomes possible.

With this method, test automation is not the only solution because the number of tests to play is reduced to the right cases.

NRT automotive applications

Throughout the years engine control unit (ECU) software requirements are getting more complex and harder to reach due to the more and more stringent emission norms and ambitious performance in terms of fuel consumption and power request, which also increase the demand and complexity of in-vehicle driving tests and diagnostic functionalities. As a consequence, along engine control systems development, each new software release results from a sequence of many others, each one introducing new functions seeking to satisfy, time after time, the crescent demands. In this context, Non-Regression testing is useful to verify that the performance and robustness of each software release does not decrease in relation with the previous one, or, in other terms, does not introduce regression.

NRT is applied, along each software release testing phase, as the final stage during integration testing, right before the execution of system testing and after the module testing (or unit testing) phase.^[3] In the module testing phase, single software modules are evaluated individually, which allows the identification of elementary errors like overflow, underflow, round-off, and also discrepancies between algorithm model simulation results and the signals coming from the engine management system (EMS). The integration testing phase, performed afterwards, aims to verify if the tested module is correctly integrated in the overall software system. Finally, functional testing (also called validation testing) is applied to validate the algorithms concerning functional requirements. This stage is usually performed after the calibration phase and characterizes an overall system testing, concluding the new software testing phase, and allowing, therefore, its release.

In automotive applications, non-regression testing is performed as follows:

1. Selection of test manoeuvres and definition of engine parameters to be monitored;
2. Execution of the selected manoeuvres on benchmark software and the software under test;
3. Post-processing and analysis of data acquired during these tests.

The selected test manoeuvres must be able to stimulate as many algorithms implemented in the software as possible. cold start, overshoot of rounds per minute, and ECE cycle (a standard manoeuvre used to calibrate on-board diagnosis) are relevant examples. In addition, the engine parameters selected to be monitored must represent the engine global operating state along the manoeuvres executed, such as Accelerator pedal deflection, Engine speed, Vehicle speed, engine temperature and throttle body opening percentage. It’s also necessary to monitor the mean variables of the control chain of Air and Torque estimation. All these diagnostic variables must be kept under control during the execution of the manoeuvres.

The tests are performed in simulation environments such as hardware-in-the-loop (HIL) simulators or Micro HIL (feed-forward systems that work as HIL downsized simulators), which support the drawing and execution of complex maneuvers usually very difficult to perform on a real engine or car (mainly because of time, cost and equipment restrictions).^[3][4]

Afterwards, a post-processing tool is required to process the acquired data, offering graphic analysis and statistical data, generally addressed to skilled personal able to identify possible regression on the software. This kind of tool can also be endowed with an automatic report generator, which gathers in a single document all the analysis results and conclusions from the comparison between the two software releases during the NRT.

See also

• Integration testing
• Regression testing
• Software testing

References

1. ^ Kaner, Cem; Bach, James; Pettichord, Bret (2001). Lessons Learned in Software Testing. John Wiley & Sons. ISBN 0471081124. 
2. ^ T. Jéron, J.M. Jézéquel, Y. Le Traon and P. Morel, "Efficient Strategies for Integration and Regression Testing of OO Systems", 10th IEEE International Symposium on Software Reliability Engineering (ISSRE'99), Florida, p. 260-269, November 1999. By VerTeCs (accessed on 01/10/10).
3. ^ ^{a b} A. Palladino, G. Fiengo, F. Giovagnini, and D. Lanzo, "A Micro Hardware-In-the-Loop Test System", IEEE European Control Conference, 2009. Abstract (accessed on 01/10/10).
4. ^ S.Raman, N. Sivashankar, W. Milam, W. Stuart, and S. Nabi, "Design and Implementation of HIL Simulators for Powertrain Control System Software Development", Proceedings of the American Control Conference, 1999.

External links

• Advanced Testing Methods for Automotive Software