Improving test case diversity for functional testing in computer vision-based systems

Abstract

Artificial Intelligence (AI) can serve as a powerful tool to enhance software testing within the Software Development Life Cycle (SDLC). By leveraging AI-driven testing, Quality Assurance Engineers (QAEs) can automate both simple and complex tasks, improving overall task accuracy and significantly accelerating the testing process. Traditionally, software testing is performed either manually or through automation. Manual testing, however, can be time-consuming and tedious, as QAEs must thoroughly review user stories with complex requirements, then translate them into comprehensive test cases. The challenge lies in ensuring that no critical steps are missed and that a sufficient number of test cases are created to fully meet all the requirements of the user story. Automation testing takes a different approach to that of manual testing. It involves creating scripts that can be applied to various types of software testing. However, building these scripts requires an experienced QAE who can translate test cases into programming classes, each containing multiple functions that cover the steps of the test case. Coding plays a crucial role in developing automation scripts that require updating, as well. This can lead to additional time and costs, making it essential to have the right resources to ensure a smooth deployment and maintain customer satisfaction. While both manual and automation testing are necessary tools for testing new software, they often demand more resources than smaller or even larger QAE teams can easily allocate. With advancements in AI, we can integrate computer vision (CV), a subfield of AI, to enhance automation testing by enabling navigation through websites in both mobile and desktop views. CV can be used to extract key information from applications, which is then utilized to automatically generate test cases. To further refine and complete the test case descriptions, a Large Language Model (LLM) is employed, providing more detailed and accurate documentation. In this dissertation, we introduce a novel concept designed to assist stakeholders across the SDLC during the testing phase. Additionally, we aim to evaluate the effectiveness of our approach by addressing key research questions that will guide us in determining whether using CV and LLMs to generate test cases offers broader test coverage and requires less maintenance compared to traditional manual and automated testing methods. The system is built on a supervised learning approach, utilizing 2000 labeled images from websites and mobile applications which combine represent 26 classes of UI components. These images were trained using two different CV algorithms: YOLOv8 and Detectron2, with a recommendation to explore AWS Rekognition in future research. To enhance the system's adaptability, robustness, and efficiency, we applied the Predictive Selection with Secondary engines pattern to further optimize its design. The detection results are leveraged to generate test cases, with ChatGPT, an LLM, assisting in the creation of detailed descriptions for each test case. The performance of YOLOv8 is evaluated using metrics such as mAP, precision, F1-score, and others across various YOLOv8 models trained for 100 epochs. Similarly, results for Detectron2 are evaluated over 20 epochs using two different models: R101 (RetinaNet) and X101-FPN (Faster R-CNN). ChatGPT successfully generated comprehensive test case descriptions, and various evaluation techniques, such as A/B testing, were implemented to analyze the quality of the generated text. Once the test cases were created, they were compared to both manually and automatically generated test cases to determine which areas of functional testing were covered. The primary goal of this research is to provide QAEs with an additional tool or approach (that is, a process) to enhance their software testing efforts, ultimately improving testing efficiency and coverage.