What is Continuous Integration?

Continuous integration (CI) is a development practice that involves frequently merging the changes in the code. A development team member may merge the code multiple times frequently in the day. Every integration change that is pushed in the code repository is verified by an automated test which runs as soon as the code change is pushed, this provides with the faster feedback on the code committed and gives a fair insight whether the merged code works as expected.

Continuous Integration is becoming an essential part of Agile and DevOps environments as it speeds up development cycles, reduces manual effort and improves code quality.

Automated testing is a fundamental part of the CI process, not only it helps in providing the faster feedback but also enables teams to detect bugs and errors early and ensure only high-quality code is deployed to the pipeline which eventually leads to production.

What is an Automated test?

An automated test script is a set of instructions written in a programming/scripting language. It is designed to test the functionality of a software product as per the business validation. Automated test scripts are usually based on a set of test requirements and carry out tests on the software which would otherwise need to be completed manually. Automated tests help in quick verification of the functionality in less time and provides us with the timely feedback on the code changes pushed to the repository.

Why write Automated tests?

If we look at the traditional SDLC (Software Development Life Cycle), whenever a member from the development team used to merge the code in the repository, the QA team was required to check if the code was working as expected by running the manual regression tests. This actually had multiple drawbacks like lots of manual efforts involved, late feedback on the code changes, no continuous testing of the build. This all eventually led to an increase in the development cost and time to production.

By writing automated tests we can get the feedback on builds quickly, reduce the manual efforts on testing and it also serves as a documentation for the functionality as tests once written will be modified only if there is change in the feature otherwise it is expected to run and pass. Automated tests also make it easier to test across multiple platforms, browsers and devices without having to manually re-run tests

Automated tests also help in running the regression test cycles quickly, so we don’t have to put in lots of manual efforts. Hence, automated tests serve as a safety net by verifying that new changes to the code don’t break existing functionality.

After writing the automated tests we need to integrate it into the automated pipeline so it performs the tasks to run the tests. Let’s now discuss the automated pipeline as it is a building block for CI/CD.

What is an Automated Pipeline?

An automated pipeline is a process of software development which helps in building, testing and deploying the code. It helps in minimizing the manual efforts and thereby removing the human errors and provides a consistent process to release the software to different development environments and finally to production.

Hope now you have a basic understanding of Continuous Integration, automated tests, and automated pipeline. Let’s now move towards discussing how we can integrate the automated tests in the continuous integration workflow and get best output on our builds.

How to integrate automated tests with Continuous Integration workflow?

The following guide provides an overview of how automated tests can be integrated within a Continuous Integration workflow

Defining Testing Strategy

Before we can start creating automated tests, we need to define the testing strategy. The objective of defining a test strategy is to:

• Explain what the tests are intended to achieve.
• Create Different Test environments.
• Running different tests like Unit, Integration, Systems and End to End Tests.
• Deployment of the builds in different environments.
• Automated deployment of Build to UAT/Production.
• Planning for Separate pipeline to run performance tests.
• Planning for running security checks to test for the vulnerabilities in the code.

Test Strategy document should outline how the software development will progress. It should also provide an overview of the QA Process.

Choosing a Test Automation Framework

After finalizing the testing strategy, we need to pick an appropriate automation testing framework. Now, for choosing an automation framework following points can be taken as a recommendation:

• Tech Stack of the Software under development: It is good to choose the framework which has the same programming language as the tech stack of the software under development, as this gives the benefit of getting help from the developers as well while writing automated tests.
• QA Team’s knowledge on programming language: Now, this is a point which can be taken up after discussion with the team. It emphasizes choosing a framework which is built in the programming language which majority of the QA Team members are familiar with. This way we might get quality code written for the tests as well as easy implementation of the framework. The other perspective to look around this point is, if the suggestion is to choose a framework which is robust and popular and has a large community however the team is not much familiar with the programming language it uses, then it should be seen what the learning curve is, if the team is ready to learn the new language and adapt it and implement the framework and how much time it will take for implementation. Decision should be taken after discussion with the team to choose a framework must be taken after consideration of the mentioned points.
• Community Support and ease of use: These are the basic points that should be checked when considering a framework for writing tests. How easy it is to configure and write tests. Does the framework have good community support in case there is an issue faced by the team implementing it. How can it be configured and run in the automation pipeline, are the points which can be thought upon and accordingly a choice should be made.
• Cost: There are a variety of tools and frameworks available, so you need to consider the environment and application requirements when selecting an appropriate solution. It’s important to evaluate the cost associated with each tool. Check the pricing and the license cost and also about the long term offerings the tool has to make.
• Open-Source automation framework: If an Open-Source automation framework is to be chosen, it must be checked if there are any issues open which are related to security vulnerability, as it might indirectly affect the application under development. Also, point to be noted here is if the project or framework is actively maintained by its owners, precaution should be taken to not choose a framework which is archived or not actively maintained as it may create a blocker for implementation if any issue is found, and since it is not actively maintained, it would be easy to fix the issue.

Once we choose the automation framework, we should follow the best practices so it caters to the news of the organization.

Creating Test Scenarios/ Test Cases

We need to write the test scenarios and respective test cases for scenarios. Test Scenarios/cases should be created after analyzing the requirements. A list of pre conditions that needs to be followed should also be generated to test the application. Most importantly, scenarios and cases should be segregated correctly based on which tests can be automated and which ones can be tested using manual exploratory tests.

Writing Automated Tests

In this step we will be writing the automated tests. This involves creating scripts that can be executed when new code is pushed to the repository. It should be noted that appropriate assertions should be performed in each test which should provide us with right feedback about the feature under development. Automating your tests will help to reduce the time and effort for running the regression tests and also ensures that new code is properly tested before it is deployed.

Execution and Reporting

Test Execution

We need to decide how we are going to execute the tests. Hence, we need to keep the following points in consideration:

1. Test Environment
2. Automated Pipelines
3. Ensure proper logging of errors/ warning/ debug to monitor the test execution.

Test Reporting

A detailed test execution report helps in providing the correct situation of the respective build, if it is good or has issues. Test Reports should include description of the tests that were executed, test data used for executing the tests, information on any bugs that were detected, as well as performance metrics.

Summary

CI/CD has become an integral part of the Software Development Life Cycle. With Continuous Testing we can always be production ready and reduce the manual efforts and improve the code quality as well.

To integrate our automation tests in CI/CD pipeline we should define a test strategy, create test cases and scenarios which can be used for writing automation tests, choose an appropriate test automation framework by discussing with your team.

We should also make sure that we have appropriate knowledge about the framework we are choosing, be it a commercial one or an open source, and how that framework will be integrated with the CI/CD pipeline. Finally, we should check out how the tests will be executed and reports will be generated which can come handy for showing the overall picture of the automation test run to the Stakeholders.

About VTEST

VTEST is an independent software testing company catering to product and service teams across the globe. We are a team of software testers who are passionate about quality and love to test. We develop an in-depth understanding of the applications under test and include software testing services strategies that deliver quantifiable results.

In short, we help in building incredible software.