This past May, I was honored to be invited along with Frederik Byl (co-founder of Probit) as #NeotysPAC speakers for June 6th’s virtual session.
It was great to be alongside Frederik with whom I’ve worked for a long time in maintenance and support. We were responsible for the deliverable’s quality, the status of the test environment, test automation, and support of the second/third line incident solution.
At this time, we had our share of struggles with quality measurement, test coverage/quality definition, motivating developers for test writing, and load testing design. It is a dirty job, but somebody’s got to do it.
Frederik and I have always risen when faced with the impossible. The more difficult the problem, the more interesting it is for us. Naturally, test automation caught our attention. It can be difficult to organize what with the complexity of test automation, and QA, in general, on many levels:
- Unit testing: Testing each part of the code for the functionality it is supposed to do
- Integration testing: Testing multiple components together and see that everything works together
- End to end testing: Testing it all together, and several ways are trying to accomplish that, monkey testing, canary testing, load testing
- Motivating developers for writing meaningful test cases
- Defining the right regression tests
- Including test estimation in the development estimations
- Calculating coverage and reporting about it in a meaningful way
- Deriving the benefit of spending more money on testing in early stages
The landscape of problems and challenges with test automation is extraordinarily vast with lots of possible improvements.
Another passion of ours is artificial intelligence. I recall sitting in a bar one night when all of a sudden it hit me – asking myself why we can’t just write intelligent test robots for test automation who get their intelligence from real user behavior on the production environment. It baffled us as to why nobody had yet to do this? How hard could it be? It sounded simple enough. So, we finished our beers and quickly got to work.
Looking back now, it was a lot more difficult and challenging than what was thought sitting on that bar stool.
User Behavior Input
To start, what we needed to ask ourselves was, “where would we get our production action data?” We had to identify a way to recognize what the user was doing.
At this point in my career, I was a Dynatrace specialist for a bank in Belgium where we used Dynatrace for incident resolution and performance measurement. Then, it dawned on me! This was exactly what we needed. For the test automation system, we could use user experience monitoring data, similar to Dynatrace’s, Google Analytics’, or anything custom logging for that matter.
We started by analyzing the data that were coming from Dynatrace to see if it was fit for input into our system. In the Dynatrace data, we had clicks, scrolls, user input, page loads, touch, and other useful actions – all the things our system would need. So, we sat down and started designing our core model of the system we then called Probit – Pattern Research of Behavior and Intelligent Testing. In design, we generalized as much as possible so the system would take Dynatrace, or any other user monitoring information, to create the test bot intelligence system build.
Usage Behavior Intelligence Model
Next step was to build the model and put the artificial intelligence to work. We were extremely skeptical of AI. We believed it was powerful, just as Professor Andrew NG (founder of Coursera and Associate Professor, Stanford University) felt when he said that, “… AI could be the next technology revolution in the world.” But, we also knew that there was a significant gap between “artificial intelligence” as management perceived it vs. “artificial intelligence” as everyone else saw it.
Again, we hunkered down and got back to work. We had a particular case of clustering involved because we were streaming real-time production action data from Dynatrace, thus needed an online clustering algorithm. It turns out, this was common, and many options were available, but none of them fit what we needed. We chose to write one ourselves based on Markov online spherical algorithms (k-means) combined with a handful of other text clustering techniques. Patent pending for this one and details can be found there.
Suffice it to say we succeeded and saw our artificial intelligence effectively identifying and defining behavior patterns for users and grouping them, in real time!
Check it out!
Test Bot Population
At last, we had the input, the model. We just needed to create the control test bot population which was going to attack the test environment’s intelligence. Perhaps we had seen too many Star Wars movies, but because we were now in control of this test bot clone army, we felt like their emperors. From then on, we referred to the system as the “EMPEROR CONSOLE.” We couldn’t help ourselves.
We started testing, measuring, and analyzing how our system was performing, whether we were achieving the results we were hoping for. Initially, stuff went wrong regularly. Too many errors prevented actions from taking place as our test subjects didn’t have the proper access permissions. We were not about to give up, especially since, it seemed, that everything going wrong presented the challenge that we so regularly thirst for. “It’s not mission difficult, Mr. Hunt, it’s mission impossible.” Using monitoring the success rates of actions and error checking per Probit client, we successfully found defects on the applications that were using Probit. Frederik and I achieved sweet victory and were proud to have persevered!
Probit works. It is ready for people to start using it.
At present, we are in the phase trials set up of Probit with three different customers who are integrating it into their organization.
Through trials, we will improve Probit further and will bring to market in 2019 a handy, useful, time and cost-saving test automation platform which will do automatic tests based on production behavior of your customers.
If you’re a Dynatrace customer and already interested in a demo or joining us for a trial, contact us. We would be delighted to unleash our intelligent bot army on your test environment.
Want to learn more about this event, see Thomas Rotte’s presentation here.