Industrial case studies for evaluating search based structural testing

Evolutionary structural testing has been researched and promising results have been presented. However, it has hardly been applied to real-world complex systems and as such, little is known about the scalability, applicability and acceptability of it in an industrial setting. The European project EvoTest (IST-33472) team has been working from 2006 till 2009 to improve this situation and this paper informs about the results. We start with an overview of tools and techniques which we have developed for automated evolutionary structural testing. Subsequently, we describe the empirical setup used to study the applicability of evolutionary structural testing in industry through two case studies. The test objects used for the studies are selected functions (handwritten and generated) from production systems at Daimler and Berner & Mattner Systemtechnik (BMS) like, for example, Rear Window Defroster, Global Powertrain Engine Controller, Window Lift Control System, etc. The results of the case studies are described and research questions are assessed based on the obtained results. In summary, the results indicate that evolutionary structural testing in an industrial setting is worthwhile and profitable. Hardly any detailed knowledge of evolutionary computation is required to search for interesting test data. The case studies also research the benefits of using techniques like automated parameter tuning and search space smoothing.