Democratizing API Load Tests with Code

Enabling people who are affected by decisions to be part of the decision-making process is a critical element of democracy. People tend to be more accepting of events when they have some say in the matter. And, as described in the book, “The Wisdom of Crowds,” given that a group is well-informed, the decisions they make collectively are usually better than ones made by an individual’s mandate. Just as members of a city council must be well informed about the details of local current events and the concerns of their constituents, modern software engineering teams need as complete a picture as possible about the impact of their changes at every point along the delivery process. And, to be well-informed, the teams need continuous, easy access to information that’s current and accurate. Otherwise, they’re flying blind. It’s that simple.

In the realm of IT, it’s not easy, particularly when it comes to API load testing. Having easy access to test information in a mission-critical situation can be limited. Typically the problem is not caused by ill intent; instead, something due to issues with planning and process.

Many times inadequate planning of fragmentary approaches to API load testing can lead to an incomplete view of the outcome, particularly in complex distributed systems. The good news – to provide the continuous stream of information necessary to democratize the load testing experience, the pieces needing to be in place are achievable. It’s a matter of following a set of three guidelines.

  • Enable feedback loops sooner vs. later
  • Make code the lingua franca
  • Prioritize early and often API feedback

Let’s take a look at the details.

Enable Feedback Loops Sooner vs. Later

In many engineering cultures “just shifting things left” is widespread if not sometimes cavalier approach to API load testing. A more mature way to approach Shift Left is to avoid the impulse to perform busy work of testing and analysis. Instead, it’s often better to first consider what’s most valuable relative to understanding the risks, putting in place the proper processes and tools to facilitate a useful feedback loop. This can take the form of well-placed telemetry in production-like rollouts or implementing a few well-placed API load tests.

Use Shorter, Faster Feedback Loops

To be exposed, some performance risks require exerting a significant burden on systems under test, over extended periods. Therefore, the timeliness of the feedback loop is paramount. The faster and shorter the feedback loop, the easier it is to address issues. Earlier testing using small volumes of traffic exposes obvious performance issues that will be a problem to solve even at massive scale.

Test APIs Before Above all Else

Testing of the API before anything else is an excellent example of opening up feedback loops sooner rather than later. An API represents a core component that serves the end user. Likewise, it’s probably the most prominent culprit that prevents the timely release of applications. If an API doesn’t perform properly, there’s no way end-to-end testing will meet the expectation.

Test in Parallel to Create Faster Feedback Loops

Processing through millions of rows for API regression testing quickly by using your highly parallel load testing tool will result in the quick turn around time of data regression coverage. Hence, speedier feedback loop.

Quick Feedback Loops Preserve Technical Context

Preserving technical context saves time ensuring fix accuracy. Technical context is all the information needed to do a job right quickly. It evaporates soon unless an open/fast feedback loop exists. Open feedback loops ensure the integrity and viability of the technical contexts your team needs to do the best possible work.

Express Everything as Code

As Agile and DevOps continue to grow in prominence across cultural paradigms in large-scale engineering organizations, the code that drives their processes/artifacts becomes the common language distributed teams use to collaborate.

Whereas in the past the traditional perception was that that language, such as Java, Python, and C# are used only to build apps and services. Today, more computing environments are provisioned with automation and infrastructure as code (IaC). The idea of “code” in its many dialects has become the lingua franca of modern IT.

Code is used to describe operational environments and constraints (i.e., Terraform). It’s used to facilitate deployment processes (e.g., Puppet, Chef, Ansible). API description formats such as OpenAPI, BDD statements like Gherkin for functional testing, compliance-as-code via InSpec and even load-as-code test descriptions in NeoLoad are all driven by some programming language. Code makes possible the reusable, versionable, automation-ready artifacts that best fit into the modern continuous delivery mindset. Today’s IT departments are subject to extraordinary, unrelenting demands that require using automation everywhere, all the time. Not doing so, to rely on manual or undocumented processes is a sign of waste.

For API load testing, the code that you share has to be easy for everyone to understand, based on open standards. This means ensuring all code is accurately documented, and naming conventions conform to enterprise guidelines. Also, the way the code is organized at the component level should conform to well-known design patterns.

Hard to understand code that’s unorganized and defies standards will result in things getting fragmented and feedback loops becoming inconsistent, particularly across organizational portfolios. As a result, software delivery becomes unpredictable. This leads to unplanned work, lack of confidence in deployment activity and a waste of financial resources.

Prioritize Feedback on APIs Early and Often

APIs aren’t the only places where performance feedback matters, but are ubiquitously dispersed along with the user experience and up-and-down distributed system call chains. You need to verify that your APIs don’t let performance problems creep in over time. It’s critical to keeping your overall performance in check. As a practitioner who also speaks to hundreds of organizations per year, the most consistent and predictable performance issues show up in API calls. Large payloads, hard to cache request patterns, and latency due to variances in payloads are issues that can be overcome early by writing a few low-complexity API load tests within development cycles.

Also, trends over time are crucial. They provide context at the moment a particular performance issue presents itself. Without longitudinal analysis, you’re a racehorse with blinders. Even with phenomenal Application Performance Monitoring (APM) and tracing tools in place, you don’t know if the issue is an outlier or a symptom of a bigger problem. Armed with historical data, you can focus on themes and distributed patterns rather than log entries. Keeping focus reduces the toil of getting paged again when the same problem pops up in a different place in your API ecosystem.

Load testing, even at low volumes (as part of a continuous delivery process) also provides engineering teams a short-cycle feedback loop that reduces context switching delays.

Pull requests, and pre-merge approval sessions provide opportunities to learn about recent changes that have caused a noticeable increase in response latencies. Such events encourage the team to address technical debt in real-time. People can ask questions about performance, such as “what is our SLO on this service.” Without these types of opportunities for investigation, introspection and remedy, potentially critical issues would have otherwise been ignored only to cause production firefighting right in the middle of the next sprint.

Putting it All Together

Process and planning are critical for creating the feedback loops required to democratize API load testing activity. All members of a company’s engineering culture need to have open access to information that is timely, accurate and easy to understand. This means that procedures need be in place to open up feedbacks loops sooner rather than later. Also, companies need to have an understanding that code is fast becoming the central means of expression in the automated enterprise. It’s the lingua franca of modern business. Today, more companies are requiring that all its employees, not just technical staff, have essential fluency in the fundamentals of programming and data formats. So, the code needs to be accessible, predictable in structure and understandable at a cursory level. Finally, companies need to give ongoing attention to their existing API feedback loops to reset priorities to meet immediate and emerging release requirements. Useful feedback loops that are accurate as well as flexible make democratization possible.

Democratizing API load testing by opening up feedback loops is essential to the modern enterprise because it allows employees to react to information accurately on time. Forward-thinking companies understand that having an informed, empowered workforce is the best way to achieve profitable results in a continually changing business environment.

Learn More about API Load Tests

Discover more load testing and performance testing content on the Neotys Resources pages, or download the latest version of NeoLoad and start testing today.


Paul Bruce
Sr. Performance Engineer

Paul’s expertise includes cloud management, API design and experience, continuous testing (at scale), and organizational learning frameworks. He writes, listens, and teaches about software delivery patterns in enterprises and critical industries around the world.

Leave a Reply

Your email address will not be published. Required fields are marked *