How meetups can get you to explore the larger world is what came true to me to participate in this year Virtual PAC
For those starting to know how to fit performance testing into an Agile model, hope this blog provides you with basic information to start with continue to work and refer. We have heard of Shift Left, Shift Right in Devops world or otherwise. Although to me adapting to Agile methodology it was a Life shift in a performance engineer’s life.
It changed the very thought process and it wasn’t easy. There were quite few challenges and lesson learnt along the way. Those experiences are part of my presentation
Performance testing is a science and an ART is something similar to baking a cake and icing it. So we better make that as artistic as possible.
What do the Executives want?
User experience of their customers using the website to stand out and provide the best experience in terms of navigating, making any transactions etc.
ROI – Return on the Investment. As they invest on the newer technologies and provide more and easy capabilities they want their user base to increase and bring more business
Handle the market volatility: Markets in bear and bull modes impact economy across the world and these are times we have seen websites crashing and the systems coming to their knees. Not to mention the financial and credibility loss of the customers and the company.
And now the question to be asked is what makes you sleep so well like a carefree child rather than what keeps you up at night?
So in order to achieve these, we need to understand how important performance testing is for an organization and as performance QA what a great responsibility lays in our hands.
Important phases in Performance Testing
We all know the different phases in the performance testing life cycle. I would like to highlight the most Key phases which need a deeper understanding of the system and its architecture. Considering that, I found the Gather NFR, Design, Development, Analysis and Capacity data as the key phases in the cycle.
- NFR is the first step to understand, exploring, scoping the project. Asking the right questions, getting hold of the right data, SLA’s, KPI’s. Collaborating across the teams (business, technology, DevOps, System Ops, Capacity team)
- Design: It involves strategizing, planning the path ahead creating a blue print and laying the foundation. This is the phase where all the key artifacts are designed and key decisions being made. WorkLoad modelling, PERF Env setup, Script design, types of tests to run, Scenario design
Those 2 involve collaborative work with the stake holders, DevOps, Ops. I could see the need of performance architect role to be responsible to conduct these phases
- Development: Get the testing code right so that the test script itself is NOT a bottleneck. It is very important the scripts are robust, agnostic, light weight, flexible and modular.
- Analysis: the investigative, deciding phase in the cycle, GO/NO GO. This involves wearing different hats as Dev, System Engineer, Investigator, Splunk expert, Architect. This phase also demands some level of collaboration, identifying and mitigating the risks
- Capacity data: This piece of data is a critical feedback into the performance requirements in terms of TPS, workload model, transaction mix, Response times and trends on those from the production.
The Playbook for Performance testing Process
Different approaches applies to different methodologies, let’s see how
- Standard Approach (Waterfall SDLC)
All phases of performance testing (mentioned in previous bullet) are followed except for the regression cases. Regression testing is performed based on the type of changes the application has
- Agile Approach
Sprint based testing is followed where each component or small changes are tested within the sprint which could be in DIT or DIT-PERF
By adopting Agile methodologies a close collaboration with Dev and team helps in early testing and finding bugs early in the cycle. Work along with Dev to help troubleshoot and fix the defects
- Scenario based/toned down approach
Based on the type of scenario, the perf process customizes the process for fast failure and focusing on the particular change or functionality or capability which may also involve running particular types of tests
The most important part of performance processes is educating the team, stake holders and management on the performance concepts, terminologies like Avg TPS, 5 min Peak TPS etc and the process itself. This gives more clarity to others of our world and be on the same plane
What’s and How’s of Method Level Testing in Agile World
Here’s a typical performance testing cycle in a sprint. Ideally the first round of performance testing should be done in one sprint; however in reality it may take multiple sprints
The process we followed to do method level testing is to first build a DIT-PERF Env which is a VM pointing to the performance backends. Perform a baseline on the DIT-PERF which is used to refer to the subsequent tests done for a particular method. As the methods or functionality are built and performance testing is done before the code is fully baked. Introducing Dynatrace into the mix gave great insight into the code behavior which helped both the parties. Definition of Done (DoD) greatly helps in defining exit criteria for the phase and avoids falling in the loop of fix, test, fix, test.
Scripting the flow is one which has immensely helped by moving to method level testing, although the robustness and flexibility of the script to be taken care of. Once the code is ready for Release build which is when the full-fledged performance testing begins in the performance environment.
Importance of Results Analysis and Monitoring
Performance results analysis is an integral part of testing where we try to make sense of the tests executed. The results may be interpreted in various ways. Few things to consider
- What are the critical data points/parameters that are considered to decide test results outcome
- How many different parameters correlated to understand the system behavior under different loads during different types of tests
- What type of Reports? Another critical piece to know which types of reports to be produced and what that report consists of. This report tells the story of the testing that has been done. The report has to cater to the different stake holders involved.
- PROD monitoring: The performance testing doesn’t end when the release is signed off, it extends post Release install to monitor the app and working with Ops team on setting any new alerts and action to take on those if needed. Also, there are chances that some issues creep in ‘cos of the user case or because of the system resources issue. Active monitoring in PROD helps to catch the issues early on before it is widespread. As a result a performance team wears multiple hats as an investigator, System Engineer and many more as needed.
Performance Testing Troubleshooting Techniques
By adopting Agile methodologies, a close collaboration with Dev and the team helped in early testing and finding and fixing the bugs early in the cycle. Few points to consider on troubleshooting the issues
- Dependency of the type of error/issue at hand
- Where to start?
- Look for errors in the tool, in the app error logs
- Which layers to look
- Tool->Script->Data->App->Webserver->Middle layer->DB
- Different Hats to wear! Investigator hat, System engineers hat
- Finding thread NOT safe code
- Thread dump Analysis
- GC behavior
Performance Reports Representation
Having been looked at the analysis, monitoring and troubleshooting the next one is how these are represented in a report. Below listed are some ways of creating the reports
- Types of Reports (HTML, Word, PDF, xlsx)
- Parameters included in the report and what they say
- Key metrics indicator
- Comparison reports
- Trending reports
- Presenting and Sharing with the team and stake holders
Collaboration within the Agile Team and the Stakeholders
Going Agile made the team accountable which resulted in more team collaboration on day to day basis. Scrum team communication improved with persistent chats/emails/phone calls
- Internal team communication
- Direct Communication with Stake holders
- One-on-ones with team members
- Educating the team on the performance processes
- And knowing what help is needed from performance testing and supporting the team
The two teams depicted in the diagram with red lines are to be connected on regular basis to make the sure end product does meet the stakeholders and customer’s needs.
Retro’s after every Sprint closure is one of the powerful tools in Agile. With still fresh in memory, team review and reflect on the job well done and which needs improvement. Also recognize peers for their contributions, help and support during the sprint which forms a motivation and reenergizes the team member
The Retro’s also bought up a non-biased feedback, good, not so good. Action items which were included as part of the sprint to track down the status.
- The Bi-weekly dedicated time to reflect and review on the work done by team is a powerful time
- Helps to tune the process early in the cycle
- Different perspectives on an issue/problem
- These Retro’s helped pave the way for incremental improvement in more of the areas
This religiously practiced process can bring and has bought a great deal of change in terms of processes, timelines and in one’s life as well. It demands disciple involvement in whatever one does.
To end with, the days are not far when AI takes care of most of the performance testing which feeds the results and gives the results of the test. It will be a matter of hours!!