Using genetic programming to evolve action selection rules in traversal-based automated software testing: results obtained with the TESTAR tool

Anna I. Esparcia, Francisco Almenar, Tanja E. J. Vos, Urko Rueda

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Traversal-based automated software testing involves testing an application via its graphical user interface (GUI) and thereby taking the user’s point of view and executing actions in a human-like manner. These actions are decided on the fly, as the software under test (SUT) is being run, as opposed to being set up in the form of a sequence prior to the testing, a sequence that is then used to exercise the SUT. In practice, random choice is commonly used to decide which action to execute at each state (a procedure commonly referred to as monkey testing), but a number of alternative mechanisms have also been proposed in the literature. Here we propose using genetic programming (GP) to evolve such an action selection strategy, defined as a list of IF-THEN rules. Genetic programming has proved to be suited for evolving all sorts of programs, and rules in particular, provided adequate primitives (functions and terminals) are defined. These primitives must aim to extract the most relevant information from the SUT and the dynamics of the testing process. We introduce a number of such primitives suited to the problem at hand and evaluate their usefulness based on various metrics. We carry out experiments and compare the results with those obtained by random selection and also by Q-learning, a reinforcement learning technique. Three applications are used as Software Under Test (SUT) in the experiments. The analysis shows the potential of GP to evolve action selection strategies.
Original languageEnglish
Pages (from-to)257-265
Number of pages9
JournalMemetic Computing
Volume10
Issue number3
DOIs
Publication statusPublished - Sep 2018

Fingerprint

Software Testing
Software testing
Genetic programming
Selection Rules
Genetic Programming
Testing
Software
Reinforcement learning
Graphical user interfaces
Q-learning
Graphical User Interface
Reinforcement Learning
Experiments
Sort
Exercise
Experiment
Metric
Evaluate
Alternatives

Cite this

@article{1ec3355377794910a982536fc900c20f,
title = "Using genetic programming to evolve action selection rules in traversal-based automated software testing: results obtained with the TESTAR tool",
abstract = "Traversal-based automated software testing involves testing an application via its graphical user interface (GUI) and thereby taking the user’s point of view and executing actions in a human-like manner. These actions are decided on the fly, as the software under test (SUT) is being run, as opposed to being set up in the form of a sequence prior to the testing, a sequence that is then used to exercise the SUT. In practice, random choice is commonly used to decide which action to execute at each state (a procedure commonly referred to as monkey testing), but a number of alternative mechanisms have also been proposed in the literature. Here we propose using genetic programming (GP) to evolve such an action selection strategy, defined as a list of IF-THEN rules. Genetic programming has proved to be suited for evolving all sorts of programs, and rules in particular, provided adequate primitives (functions and terminals) are defined. These primitives must aim to extract the most relevant information from the SUT and the dynamics of the testing process. We introduce a number of such primitives suited to the problem at hand and evaluate their usefulness based on various metrics. We carry out experiments and compare the results with those obtained by random selection and also by Q-learning, a reinforcement learning technique. Three applications are used as Software Under Test (SUT) in the experiments. The analysis shows the potential of GP to evolve action selection strategies.",
author = "Esparcia, {Anna I.} and Francisco Almenar and Vos, {Tanja E. J.} and Urko Rueda",
year = "2018",
month = "9",
doi = "10.1007/s12293-018-0263-8",
language = "English",
volume = "10",
pages = "257--265",
journal = "Memetic Computing",
issn = "1865-9284",
publisher = "Springer Verlag",
number = "3",

}

Using genetic programming to evolve action selection rules in traversal-based automated software testing : results obtained with the TESTAR tool. / Esparcia, Anna I.; Almenar, Francisco; Vos, Tanja E. J.; Rueda, Urko.

In: Memetic Computing, Vol. 10, No. 3, 09.2018, p. 257-265.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - Using genetic programming to evolve action selection rules in traversal-based automated software testing

T2 - results obtained with the TESTAR tool

AU - Esparcia, Anna I.

AU - Almenar, Francisco

AU - Vos, Tanja E. J.

AU - Rueda, Urko

PY - 2018/9

Y1 - 2018/9

N2 - Traversal-based automated software testing involves testing an application via its graphical user interface (GUI) and thereby taking the user’s point of view and executing actions in a human-like manner. These actions are decided on the fly, as the software under test (SUT) is being run, as opposed to being set up in the form of a sequence prior to the testing, a sequence that is then used to exercise the SUT. In practice, random choice is commonly used to decide which action to execute at each state (a procedure commonly referred to as monkey testing), but a number of alternative mechanisms have also been proposed in the literature. Here we propose using genetic programming (GP) to evolve such an action selection strategy, defined as a list of IF-THEN rules. Genetic programming has proved to be suited for evolving all sorts of programs, and rules in particular, provided adequate primitives (functions and terminals) are defined. These primitives must aim to extract the most relevant information from the SUT and the dynamics of the testing process. We introduce a number of such primitives suited to the problem at hand and evaluate their usefulness based on various metrics. We carry out experiments and compare the results with those obtained by random selection and also by Q-learning, a reinforcement learning technique. Three applications are used as Software Under Test (SUT) in the experiments. The analysis shows the potential of GP to evolve action selection strategies.

AB - Traversal-based automated software testing involves testing an application via its graphical user interface (GUI) and thereby taking the user’s point of view and executing actions in a human-like manner. These actions are decided on the fly, as the software under test (SUT) is being run, as opposed to being set up in the form of a sequence prior to the testing, a sequence that is then used to exercise the SUT. In practice, random choice is commonly used to decide which action to execute at each state (a procedure commonly referred to as monkey testing), but a number of alternative mechanisms have also been proposed in the literature. Here we propose using genetic programming (GP) to evolve such an action selection strategy, defined as a list of IF-THEN rules. Genetic programming has proved to be suited for evolving all sorts of programs, and rules in particular, provided adequate primitives (functions and terminals) are defined. These primitives must aim to extract the most relevant information from the SUT and the dynamics of the testing process. We introduce a number of such primitives suited to the problem at hand and evaluate their usefulness based on various metrics. We carry out experiments and compare the results with those obtained by random selection and also by Q-learning, a reinforcement learning technique. Three applications are used as Software Under Test (SUT) in the experiments. The analysis shows the potential of GP to evolve action selection strategies.

U2 - 10.1007/s12293-018-0263-8

DO - 10.1007/s12293-018-0263-8

M3 - Article

VL - 10

SP - 257

EP - 265

JO - Memetic Computing

JF - Memetic Computing

SN - 1865-9284

IS - 3

ER -