Continuous Integration is becoming a key element in modern day Agile development. When integrated with performance testing, CI can be vital in detecting and mitigating performance issues before they have a chance to derail your system in the real world. A solid return on your investment is when you are able to prevent a failure from escaping to production and helping your company avoid lost revenue and damage to your brand reputation.
We took a look at the current CI landscape to learn who is using CI, and why and how companies are using it. View the complete infographic here.
While automated testing is mainstream, automated performance testing is cutting edge.
Automated testing is becoming top of mind for companies with 88% of companies across industries building applications using a continuous integration server. However, even though a majority of companies are using CI servers, only 13% of them are running both automated functional and performance testing, while 69% are still running functional testing only. While still a young discipline, automated performance testing is a practice which is steadily evolving and being adopted by an increasing numbers of companies who want to get ahead of the curve.
Using a CI server shortens iterations… to a point.
While 64% of companies using CI servers are able to get down to 1-2 months for their development iterations, this isn’t as short as some Agile methodologies (like Scrum) suggest they could be. People often have to cut certain steps out of their Agile processes (like performance testing), and save them for a “hardening” sprint. With performance testing, this is usually because the tool doesn’t integrate with the CI server to enable automation or because the design of test cases takes too long.
The number one goal of Cl performance testing is to catch performance issues earlier in the development process
57% of companies state that the number one goal for continuous performance testing is to catch performance issues earlier in the development process, while 25% of companies seek to automate more tests to increase the productivity and efficiency of development time. Catching performance issues early on in the development process is both a time and money saver since developers can fix the problem then and there before it becomes exponentially more expensive.
Catching performance issues are critical to a majority of companies
53% of companies state that it is critically important that their applications meet performance SLAs every release since performance lags run the risk of causing users to drop off their application if it takes too long to load. 41% assert that while it is not critical since a small amount of lag time doesn’t typically affect their usership, they recognize that meeting performance SLAs are importance to the overall success of their application and satisfaction of their customers. In short, not many companies can get away with ignoring an application’s performance as only 6% of companies claim that meeting SLAs every release is not important to their organization.
The frequency of automated performance testing depends on who is running the tests.
38% of companies who have QA engineers and testers with dual responsibilities in both functional and performance testing use a CI server that runs automated performance testing. On the other hand, only 7% of companies with dedicated performance engineers and testers use a CI server that runs automated performance testing. These numbers reflect a fairly typical scenario where testers with multiple responsibilities (in both functional and performance testing) are integrated into their organization’s agile development team, as opposed to dedicated performance testers where the organization commonly separates QA and development.
Interested in accelerating your performance testing?
As organizations increasingly adopt continuous integration, many struggle with questions about performance testing in an Agile or Continuous Testing environment. This whitepaper on Load Testing at the Speed of Agile not only outlines some of the challenges of load and performance testing in an Agile environment, but also provides best practices like prioritizing performance goals and automating testing with Continuous Integration servers like Jenkins. With the right tools, Continuous Performance Testing can become a reality in your organization.