- Çağrı Murat Karapıçak (1)
- Feyza Nur Kılıçaslan (1)
- Serafettin Senturk (1)
- Erdem Yildirim (1)
- Uğur Yılmaz (1)
Transitioning from Manual to Automated Software Regression Testing: Experience from the Banking Domain
Akin, Abdurrahman ; Senturk, Serafettin ; Garousi, Vahid - \ 2019
In: Proceedings - 25th Asia-Pacific Software Engineering Conference, APSEC 2018. - IEEE computer society - ISBN 9781728119700 - p. 591 - 597.
experience report - mobile software - regression testing - Software testing - test automation
Regression testing is needed when a software or the environment hosting that software changes. Motivated by a real-world industrial need in the context of a large financial (banking) corporation in Turkey, the authors and their colleagues developed and introduced an automated regression testing infrastructure for automated testing of one of the main mobile applications of the company. Before this project, regression testing was conducted manually which incurred a lot of costs and was by nature subjective. We report in this paper our experience in 'transitioning' from manual to automated regression testing, and in developing and introducing a set of large automated test suites (more than 16 KLOC in total), using best practices in state-of-the art and -practice, and to report its observed benefits by conducting cost-benefit analysis. The project was conducted based on the principles of case-study and 'action research' in which the real industrial needs drove the research. Among the best practices that we used are the followings: (1) modularity in test code, (2) creating test-specific libraries, and (3) separating test data from test logic. By serving as a success story and experience report in development and introduction of automated test suites in an industrial setting, this paper adds to the body of evidence in this area and it aims at sharing both technical (e.g., using automated test patterns) and process aspects (e.g., test process improvement) from our project with other practitioners and researchers.
A survey on software testability
Garousi, Vahid ; Felderer, Michael ; Kılıçaslan, Feyza Nur - \ 2019
Information and Software Technology 108 (2019). - ISSN 0950-5849 - p. 35 - 64.
Software testability - Software testing - Survey - Systematic literature mapping - Systematic literature review - Systematic mapping
Context: Software testability is the degree to which a software system or a unit under test supports its own testing. To predict and improve software testability, a large number of techniques and metrics have been proposed by both practitioners and researchers in the last several decades. Reviewing and getting an overview of the entire state-of-the-art and state-of-the-practice in this area is often challenging for a practitioner or a new researcher. Objective: Our objective is to summarize the body of knowledge in this area and to benefit the readers (both practitioners and researchers) in preparing, measuring and improving software testability. Method: To address the above need, the authors conducted a survey in the form of a systematic literature mapping (classification) to find out what we as a community know about this topic. After compiling an initial pool of 303 papers, and applying a set of inclusion/exclusion criteria, our final pool included 208 papers (published between 1982 and 2017). Results: The area of software testability has been comprehensively studied by researchers and practitioners. Approaches for measurement of testability and improvement of testability are the most-frequently addressed in the papers. The two most often mentioned factors affecting testability are observability and controllability. Common ways to improve testability are testability transformation, improving observability, adding assertions, and improving controllability. Conclusion: This paper serves for both researchers and practitioners as an “index” to the vast body of knowledge in the area of testability. The results could help practitioners measure and improve software testability in their projects. To assess potential benefits of this review paper, we shared its draft version with two of our industrial collaborators. They stated that they found the review useful and beneficial in their testing activities. Our results can also benefit researchers in observing the trends in this area and identify the topics that require further investigation.
What We Know About Smells in Software Test Code
Garousi, Vahid ; Kucuk, Baris ; Felderer, Michael - \ 2019
IEEE Software 36 (2019)3. - ISSN 0740-7459 - p. 61 - 73.
multivocal literature review - Software testing - systematic literature review - test anti-patterns - test automation - test scripts - test smells
As a type of anti-pattern, test smells are defined as poorly-designed tests and their presence negatively affects the quality of test suites and production code. There is a large body of knowledge on the subject of test smells. Thus, it is not practical for practitioners and researchers to synthesize and benefit from such a large literature. Motivated by the above need, we report a ‘multivocal’ literature mapping (classification) on 46 sources from the scientific literature and 120 grey literature sources. We present the largest catalogue of test smells, along with the summary of guidelines, techniques and tools to deal with test smells. This article will benefit practitioners and researchers by serving as an “index” to the vast body of knowledge in this area, and by helping them develop high-quality test scripts, and minimize occurrences of test smells and their negative consequences in test automation projects.
Testing embedded software : A survey of the literature
Garousi, Vahid ; Felderer, Michael ; Karapıçak, Çağrı Murat ; Yılmaz, Uğur - \ 2018
Information and Software Technology 104 (2018). - ISSN 0950-5849 - p. 14 - 45.
Embedded software - Embedded systems - Software testing - Systematic literature mapping - Systematic literature review - Systematic mapping
Context: Embedded systems have overwhelming penetration around the world. Innovations are increasingly triggered by software embedded in automotive, transportation, medical-equipment, communication, energy, and many other types of systems. To test embedded software in an effective and efficient manner, a large number of test techniques, approaches, tools and frameworks have been proposed by both practitioners and researchers in the last several decades. Objective: However, reviewing and getting an overview of the entire state-of-the-art and the –practice in this area is challenging for a practitioner or a (new) researcher. Also unfortunately, as a result, we often see that many companies reinvent the wheel (by designing a test approach new to them, but existing in the domain) due to not having an adequate overview of what already exists in this area. Method: To address the above need, we conducted and report in this paper a systematic literature review (SLR) in the form of a systematic literature mapping (SLM) in this area. After compiling an initial pool of 588 papers, a systematic voting about inclusion/exclusion of the papers was conducted among the authors, and our final pool included 312 technical papers. Results: Among the various aspects that we aim at covering, our review covers the types of testing topics studied, types of testing activity, types of test artifacts generated (e.g., test inputs or test code), and the types of industries in which studies have focused on, e.g., automotive and home appliances. Furthermore, we assess the benefits of this review by asking several active test engineers in the Turkish embedded software industry to review its findings and provide feedbacks as to how this review has benefitted them. Conclusion: The results of this review paper have already benefitted several of our industry partners in choosing the right test techniques / approaches for their embedded software testing challenges. We believe that it will also be useful for the large world-wide community of software engineers and testers in the embedded software industry, by serving as an “index” to the vast body of knowledge in this important area. Our results will also benefit researchers in observing the latest trends in this area and for identifying the topics which need further investigations.
Introducing automated GUI testing and observing its benefits : An industrial case study in the context of law-practice management software
Garousi, Vahid ; Yildirim, Erdem - \ 2018
In: Proceedings - 2018 IEEE 11th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2018. - Institute of Electrical and Electronics Engineers Inc. - ISBN 9781538663523 - p. 138 - 145.
Action research - Experience report - Industrial case study - Software testing - Test automation - Test code engineering - Test patterns
Motivated by a real-world industrial need in the context of a large IT solutions company based in Turkey, the authors and their colleagues developed and introduced automated test suites for GUI testing of two large-scale law-practice management software (comprising of 414 and 105 KLOC). We report in this paper our experience in developing and introducing a set of large automated test suites (more than 50 KLOC in total), using best practices in state-of-the art and -practice, and to report its observed benefits by conducting cost-benefit analysis in the specific industrial context. The project was conducted based on the principles of case-study and 'action research' in which the real industrial needs drove the research. Among the best practices that we used are the followings: (1) the page-object test pattern, (2) modularity in test code, (3) creating test-specific libraries, and (4) using systematic guidelines to decide when and what (test cases) to automate. To assess the cost-benefit and Return On Investment (ROI) of test automation, we followed a hybrid measurement approach to assess both the quantitative and qualitative (intangible) benefits of test automation. The empirical findings showed that the automated GUI testing approach has indeed benefitted the test and QA team in the company under study and automation has been highly welcome by the test engineers. By serving as a success story and experience report in development and introduction of automated test suites in an industrial setting, this paper adds to the body of evidence in this area and it aims at sharing both technical (e.g., using automated test patterns) and process aspects (e.g., test process improvement) of our project with other practitioners and researchers with the hope of encouraging more industry-academia collaborations in test automation.
Smells in software test code : A survey of knowledge in industry and academia
Garousi, Vahid ; Küçük, Barış - \ 2018
Journal of Systems and Software 138 (2018). - ISSN 0164-1212 - p. 52 - 81.
Automated testing - Multivocal literature mapping - Software testing - Survey - Systematic mapping - Test anti-patterns - Test automation - Test scripts - Test smells
As a type of anti-pattern, test smells are defined as poorly designed tests and their presence may negatively affect the quality of test suites and production code. Test smells are the subject of active discussions among practitioners and researchers, and various guidelines to handle smells are constantly offered for smell prevention, smell detection, and smell correction. Since there is a vast grey literature as well as a large body of research studies in this domain, it is not practical for practitioners and researchers to locate and synthesize such a large literature. Motivated by the above need and to find out what we, as the community, know about smells in test code, we conducted a ‘multivocal’ literature mapping (classification) on both the scientific literature and also practitioners’ grey literature. By surveying all the sources on test smells in both industry (120 sources) and academia (46 sources), 166 sources in total, our review presents the largest catalogue of test smells, along with the summary of guidelines/techniques and the tools to deal with those smells. This article aims to benefit the readers (both practitioners and researchers) by serving as an “index” to the vast body of knowledge in this important area, and by helping them develop high-quality test scripts, and minimize occurrences of test smells and their negative consequences in large test automation projects.