Integrating Testing into a CI/CD pipeline
Presently we are in the age of DevOps, a framework that integrates every step of the life-cycle of software development, bringing all the operations together, collaboratively, without any organizational silos, under one umbrella. A Continuous Integration and Continuous Delivery (CI/CD) pipeline empower a DevOps-driven SDLC.
Although Continuous Integration is the process of integrating each developer's developed code into the main source code, Continuous Delivery allows fully automated, low-risk, and low-cost releases to be deployed with one click. Automation thrives on a CI/CD pipeline, which is the justification for the quick deliveries and quicker feedback loops. But, test automation and continuous testing (CT) are the keys to achieving truly effective CI/CD. Although Continuous Integration and Continuous Delivery are essential for product releases to be accelerated, Continuous Testing is what brings in quality to these accelerated releases.
Continuous Testing within a CI/CD pipeline
A CI pipeline will have automated unit test cases that can be run quickly. This gives the first level of feedback to developers if their code has broken any of the unit test cases.
The CI pipeline can also contain automated integrations tests that look at API based integrations between modules of the system and/or API based integration of the system with surrounding applications. These API based integration tests may use test stubs for surrounding applications. This gives the second level of feedback to developers if their code has broken any of the integration tests.
Projects also have GUI based automated regression tests. These tests take longer to run and are typically not run as part of the CI pipeline, as the objective of CI is quick feedback to developers on the impact of their code changes. However, automated smoke tests for each build deployed to environments bring a lot of RoI by eliminating manual smoke test efforts and providing faster feedback. These smoke tests can be included in the pipeline to ensure that the system under test works as a whole. To increase the speed of these tests, they can be run in a headless browser (unless your application does not support a headless browser, or the focus of the test is on real world conditions). This gives the third level of feedback to developers if their code has broken any key flows.
The Pipeline should also integrate end-to-end (E2E) tests to be run on an integrated environment so that all the E2E tests covering both internal and external integrations are executed as part of an automated pipeline.
Each of the above levels of feedback can have intelligence built in to proceed to the next stage only if all tests have passed. This gives the developers the opportunity to identify and fix issues quickly.
Organizations use different approaches to run the entire GUI based automated regression tests. Some run them periodically as a batch (outside the CI/CD pipeline) at certain times of the day or is manually triggered. Some run them as part of the CI/CD pipeline by splitting these tests into suites which are run in parallel to decrease the time taken to run all tests and to provide feedback faster – this requires multiple hardware for running these automated tests in parallel. The need for multiple hardware can be met by dynamically creating and destroying them in the cloud from the pipeline (this approach can also be used for smoke tests mentioned above if the number of tests is high). Irrespective of the approach, the key is to run the automated tests frequently to identify & fix issues early. Again, as in the case of smoke tests, one can explore running these tests in a headless browser to increase the speed of testing.
Performance tests can also be integrated into the CI/CD pipeline. Performance tests as part of the pipeline are appropriate for applications that are not fit for purpose if they do not meet certain minimum performance thresholds. Early white box performance engineering activities like Unit Performance Profiling (Code and system) for End to End Stack can be made part of the pipeline along with unit and integrations tests. Performance Tests including Load, Stress, Endurance and Soak tests can also be integrated with Automated CI/CD Pipeline. Since performance tests will involve creating user load, the DevOps pipeline will need to set up the performance test environment, run the tests, copy the test results and then destroy the environment for optimal use of the test infrastructure. The scenarios that are covered in the performance should be functionally stable and can be run as part of the pipeline in parallel to regression test automation or post the regression test automation, based on infrastructure availability.
Security test automation is similar to automated functional tests, but targeted at verifying that security features from authentication to logout work as expected. They can be automated and integrated with the CI/CD pipeline. Specific security testing includes testing known weaknesses and mis-configurations such as lack of the http Only flag on session cookies or the use of known weak SSL suites and ciphers, etc. These are particularly well suited for automation because the weaknesses are known upfront from the threat modeling done by the security team. To successfully automate application scanning, one should ensure that all the content to be scanned is navigated and populated in the security scanning tool, before starting to spider and scan the application through the CI/CD pipeline. Post the scanning, one will have to manually analyze the detailed security test reports/logs from the tool to verify false positives (vulnerabilities).
The tests mentioned above will need test data. Typically, unit tests do not run against a real database. But smoke, regression, and performance tests will need a real test database. So, the CI/CD pipeline will need to set up the test data before the execution of these automated tests. At the end of the test, the test data should be reset to avoid dependencies between different test stages.
The Benefits and Importance of CI/CT/CD
Underpinned by automation, CI/CT/CD has become core to DevOps for delivering software in short cycles of build-configure-
deploy-test-release. The following are some of the many benefits that businesses reap from this.
- Early testing to identify and fix defects early
- Speedy innovation in the industry for a competitive advantage
- Reduced manual effort and increased efficiency of delivery
- Quicker revenue generation
How Sonata can help?
With a wealthy involvement in testing and Implementing of DevOps we assist our clients to promote a Continuous Integration (CI) / Continuous Testing (CT) framework to integrate automated software testing with deployment and operations, to improve code quality and time to market. By doing so, we accelerate the release of applications into production, improve the efficiency and utilization of resources, and streamline the CI/CD pipeline.