Getting Started with Performance Testing As Code

Introduction

Welcome

NeoLoad is an enterprise-ready, lean and mean load testing platform that supports both graphical and code-based approaches to performance engineering. This tutorial is focused on our As-Code approach and will walk you through examples to demonstrate what it can do.

Notable Features and Assumptions

NeoLoad uses a YAML-based description format which is human readable, implementation agnostic, and domain specific to load testing. It’s great for developers, performance engineers, and business experts to collaborate on SLOs (Service Level Objectives), environment details such as service endpoint details, and load testing details such as ramp-up semantics.

Once running, we also will visualize test results using NeoLoad Web in a browser, since command-line print-outs of test details are only somewhat useful in a developer/workstation context and don’t really scale past that context and into CI testing environments.

NeoLoad As-Code is built into our existing product, and instead of a traditional desktop installation path, we’ll be using modern tools such as Docker and Git CLI (command line interface) to simplify the process of API testing that fits with both development and continuous integration workflows.

Feedback and Contributions

These examples are always provisional and greatly benefit from your feedback!
Please email any comments or suggestions to as-code-feedback@neotys.com

Installation

From a Shell Prompt (Windows & Mac)

First, let’s open up a shell prompt and initialize our basic example toolset. If you want to try this out on cloud-based infrastructure instead of your own workstation, see our ‘Trying As-Code Out on Cloud VMs’ section. Our CLI-based examples require Docker and Git to be installed, so make sure you’ve set that up first and it’s running in the background.

These examples also assume that you can access Github (public), so if you can’t, you can either ask someone else to clone the repo into your organization’s local repo provider, or simply skip to the ‘Get Your NeoLoad Web SaaS Token’ and fill out the download form to get in contact with us to help you through this process.

For Windows shell: open a Command Prompt and type…
For Mac shell: open a Terminal window and type…

Shell:


git clone https://github.com/paulsbruce/neoload-as-code.git
cd neoload-as-code

If you are on a Mac/Linux machine, you’ll also need to make our samples script executable:

Shell (Mac/Linux):

./neoload-cli.sh --verify

On Windows, simply run the following:

Shell (Windows):

neoload-cli.bat --verify

You should see that Docker pulls down the latest images and prints out the following message:

Get Your Neotys SaaS Token

Secondly, you’ll need a token to send statistics to NeoLoad Web SaaS. No proprietary information is sent, just the numerical aggregates of these example results once we get to running a test. To do this:

  1. Go to https://www.neotys.com/download and complete the form. This will create an account and prompt to download a license key (not necessary for these examples).
  2.  

  3. You should also receive an ‘Activate Your Neotys Account’ email from ‘donotreply@neotys.com’.
    Click the activate link.
  4.  

  5. Set your initial password during the account activation process.
    Then go to https://neoload.saas.neotys.com/
    If you’ve already done so, simply log in using your credentials.
  6.  

  7. On the left navigation menu, select ‘Profile’
  1. Click ‘Generate the Access Token’ button, then click the ‘Copy to Clipboard’ button:

     

With your freshly minted access token on the clipboard, go back over to your shell prompt and type in (don’t forget to substitute your actual token below):

Shell (Mac/Linux):
./neoload-cli.sh --init=[your token here]

Shell (Windows):
neoload-cli.bat --init=[your token here]

This should run a verification to ensure that everything’s working pristinely and we can get on to writing our first NeoLoad As-Code test!
You won’t have to specify your token again, as it persists this detail to a new ‘.conf’ directory under the project:

Your First NeoLoad As-Code Test

Because you cloned the example repo and switched to that directory, you already have everything you need to write and run your first test. To do so:

  1. In the ‘neoload-as-code’ repo folder you cloned, create a directory called ‘my_tests’
  2. Create a text document in that directory called ‘my_first_test.yaml’
  3. Open that new text document in your favorite text editor, something that automatically syntax-highlights YAML files. We suggest:
    • Atom editor
    • Notepad++
    • Notepad (Windows) or textEdit (Mac) with a monospace font enabled
    • Microsoft Code

When load testing, you typically want to separate out the following test assets so they can be reused later:

  • “User Path” or “Virtual User script”
    Definition: steps performed for each concurrent workload thread
  • “Population”:
    Definition: a collection of one or more User Paths and client simulation options
  • ‘Scenario”:
    Definition: a plan including how and from where populations should execute their load
  • ‘SLA / SLO”:
    Definition: a collection of expectations over metrics produced by elements during a test

     

Let’s start with the simplest building block of a load test: the User Path.

Define a Simple API User Path

In your text editor, enter the following:

Editor<my_first_test.yaml>:


name: My First Test

user_paths:
- name: geo_search
  actions:
    steps:
    - request:
        url: http://host.docker.internal:3000/search?format=json&q=Boston
    - delay: 1s


The sample containers you spun up with the docker-compose step above contain a simple microservice which translates readable city names to latitude and longitude data via the Open Street Maps public web service. The YAML above tells NeoLoad to make one request to this service and then wait for one second (otherwise, we might machine-gun our server to death).

Define a Placeholder Population

Before we define how long the test should run in a scenario, we need to also add a population as an intermediate step. Add the following to the bottom of the file:

Editor<my_first_test.yaml>:


populations:
- name: pop1
  user_paths:
  - name: geo_search

This simply means that the path we just defined should be 100% of the work this population will perform, and later we’ll extend but for now, that’s all we need to move on to defining a scenario.

Define a Scenario and Arrival Rate

A scenario brings together the notion of which paths/workloads you wish to run AND how much you wish to run them. Again, add the following to the bottom of the file:

Editor<my_first_test.yaml>:


scenarios:
- name: sanityScenario
  populations:
  - name: pop1
    rampup_load:
      min_users: 1
      max_users: 10
      increment_users: 2
      increment_every: 5s
      duration: 1m

This tells NeoLoad to ramp up to 10 parallel threads, adding 2 threads of work every 5 seconds, and maintain 10 parallel threads worth of load until 1 minute has elapsed.

 

Note: “Users” is a throwback to web-related simulation which, but in our case, this is equivalent to the notion of parallel threads that handle workloads. In API terms, you can think of “users” as “consumers”, regardless of whether the request is coming from an actual human user or another external process.

Don’t forget to save your file. 😉

Execute Your First NeoLoad As-Code Test

To check that we did everything right, let’s run your test. Finally! The blissful moment you’ve been waiting for! Let’s go back to your shell, and type in:

Shell (Mac / Linux):


./neoload-cli.sh --scenario=sanityScenario --file=my_test/my_first_test.yaml


Shell (Windows):


neoload-cli.bat --scenario=sanityScenario --file=my_test/my_first_test.yaml

This script creates:

  1. A few containers representing the system under test (SuT), a geolookup microservice
  2. A neoload-cli helper container that simplifies the syntax required to run your test
  3. A ‘neoload-cli-ctrl’ NeoLoad Controller container as an engine to actually run our test

 

It also remembers your API token from earlier and immediately opens a browser to visualize your test. You should see some test engine semantics in the shell output, particularly a URL to NeoLoad Web, as seen below:

A few next steps:

  • To speed demonstration of the many additional capabilities of NeoLoad, we’ve already created example scripts. Feel free to poke around in the YAML files.
  • If you want to skip right to implementing this in your own CI environment (such as Jenkins), see the ‘Load Testing in Continuous Integration’ section

Define Your Expectations Using SLAs

Because this is an API, we measure RPS (Requests Per Second) in NeoLoad Web; but this is only an outcome of putting pressure on our microservice and doesn’t say anything about whether the RPS meets our expected goals. To do this, let’s go back and add SLAs to the top of the file, just above the ‘user_paths’ directive:

Editor<my_first_test.yaml>:


sla_profiles:
- name: geo_3rdparty_sla
  thresholds:
  - avg-resp-time warn >= 100ms fail >= 500ms per interval
  - error-rate warn >= 2% fail >= 5% per test

user_paths:
...

We also need to apply this SLA profile to specific components of our test, particularly in this case our 3rd-party HTTP request. Go back up to your user path definition and add the following ‘sla_profile’ line:

Editor<my_first_test.yaml>:


user_paths:
- name: geo_search
  actions:
    steps:
    - request:
        url: http://localhost:3000/search?format=json&q=Boston
        sla_profile: geo_3rdparty_sla
    - delay: 1s

Re-run to See SLAs in NeoLoad Web

Once again, save your file, then in your shell, type in:

Shell (Mac / Linux):


./neoload-cli.sh --scenario=sanityScenario --file=my_test/my_first_test.yaml

;

Shell (Windows):


neoload-cli.bat --scenario=sanityScenario --file=my_test/my_first_test.yaml

;

As the test is running, if you switch over to NeoLoad Web and click into the test results, in real-time you should see the Average Response Time SLA begin to show pass/fail:

Congratulations! You’ve run your first NeoLoad As-Code test!

Using NeoLoad As-Code

Before we dive into more detail, let’s run a pre-built test that illustrates the core concepts we’ll cover (be sure to copy both lines as one if wrapping occurs):

 

Shell (Mac/Linux):


./neoload-cli.sh --scenario=sanityScenario --file=projects/example_1_1_request/project.yaml

Shell (Windows):


neoload-cli.bat --scenario=sanityScenario --file=projects/example_1_1_request/project.yaml

Once the test kicks off and a NeoLoad Web URL is available, you should see test results automatically open in your browser.

Anatomy of an HTTP request

For REST API testing, NeoLoad sends HTTP/S requests that include standard elements such as HTTP method, headers, and body content. Below is a complete sample anatomy of a request in NeoLoad As-Code:

Editor < projects example_1_1_request/complete_request.yaml >


...
user_paths:
- name: ex_1_0_complete_request
  actions:
    steps:
    - transaction:
        name: External Geo-lookup (may cache)
        description: Call Open Street Maps to translate city names...
        steps:
        - request:
          url: /search?format=json&q=${cities.City}
          server: geolookup_host
          sla_profile: geo_3rdparty_sla
...

There are *many* other features beyond the above simple example, some of which we’ll be going through in this tutorial. Full YAML schema documentation can also be found here.

Out-of-the-Box Metrics

By default, NeoLoad captures the following load metrics:

  • Response time (Average, Min, Max)
  • Elements/sec (Transaction, Page, Request)
  • Throughput in Mbs
  • Time to first byte (Average, Min, Max)
  • Errors (Total, Rate, Per Sec)

 

This is great for basic API testing as a baseline to understand:

  • Total perceived wait time due to particular API calls
  • Total number of API transactions (i.e. requests) processed within a particular timeframe
  • TTFB (time-to-first-byte) vs. response time (time-to-last-byte) for network latency
  • How much network I/O is being used by a particular request and how it affects TTFB
  • How total volume affects error rate (are they linear?)

 

You get all these data points just from the simple test we just ran, and there are additional things we can do to increase granularity and observability in your tests.

For now these core metrics provide a reliable baseline that we can trend on moving forward. If you run this test over and over again, you’ll start to see trends like this in NeoLoad Web:

NeoLoad does offer much more in terms of server monitoring beyond our As-Code examples in this tutorial. Since this is more of an advanced topic, reach out to us for more information on server monitoring integrations like Dynatrace, AppDynamics, NewRelics, AWS CloudWatch, and other APM platforms.

Assertions for Validating Responses

It’s important to know that your API is, in fact, processing your requests correctly during a load test, otherwise, you may be reporting performance data based on errors instead of correctly functioning code and configuration. One way to do this with NeoLoad As-Code is to define a value extractor on a success/positive condition in the response content. For example:

Editor<projects/example_1_1_request/complete_request.yaml>:

...
- request:
    url: /search?format=json&q=${cities.City}
    server: geolookup_host
    sla_profile: geo_3rdparty_sla
    extractors:
    - name: latitude
      jsonpath: $.latitude
      regexp: (.*)
      match_number: 1
      extract_once: true
...

Transactions

In many cases, a set of User Path steps (like HTTP requests and javascript blocks) should be grouped so that the average time taken and throughput of the lot of them are also calculated together. 

Dynamic Data and Tokens

In many cases, requests to the server require values or tokens that change each time, such as OAuth workflows and gated API resources. To get our test to flow this data through to the right places, we need to A) extract these tokens and B) inject them into the right places.

 

Extracting Values from Requests

This is essentially what we did with validators in the prior section, but you can have as many as you need to pass to subsequent requests. Let’s open the following file in our editor and look at how both extraction and injection of dynamic values works in NeoLoad as-code:

 

Editor<projects/example_1_2_variables/oauth_workflow.yaml>:

...
- request:
    url: /js/bundle.js
    server: app_server
    method: GET
    sla_profile: sla_static_resources
    extractors:
    - name: oauth_client_secret
      regexp: OAUTH_CLIENT_SECRET:"(.*?)"
...

In the above example, we’re using a simple regular expression to extract a piece of information from our bundled client app that is important to the OAuth workflow, the client secret key. We capture this value into a variable named ‘oauth_client_secret’ and subsequently inject it into the following request to obtain an OAuth token:

 

Editor<projects/example_1_2_variables/oauth_workflow.yaml>:

...

- request:
    url: /platform/oauth/token
    server: app_server
    method: POST
    sla_profile: sla_internal_api_calls
    headers:
    - accept: application/json
    - Content-Type: application/json
    body: '{"grant_type":"client_credentials","client_id":"ushahidiui","client_secret":"${oauth_client_secret}","scope":"posts media forms api tags savedsearches sets users stats layers config messages notifications contacts roles permissions csv"}'
...

Additionally, from this step, we want to further extract the OAuth token specific to our current session with the server, so we include the following extractor declaration:

 

Editor<projects/example_1_2_variables/oauth_workflow.yaml>:

...

- request:
    url: /platform/oauth/token
    ...
    body: ...
    extractors:
    - name: oauth_bearer
      jsonpath: $.access_token
...

The name of the extractor will be the name of the variable in use later, and the ‘jsonpath’ property is the selector we use to locate the specific piece of data we need from the response content. Pretty simple, huh? Now let’s use the OAuth token in a request header:

 

Editor<projects/example_1_2_variables/oauth_workflow.yaml>:

...

- transaction:
    name: Get internal/gated resource
    sla_profile: sla_internal_api_calls_big_response
    steps:
    - request:
        url: /platform/api/v3/posts?order=desc&orderby=post_date
        server: app_server
        method: GET
        headers:
        - Authorization: Bearer ${oauth_bearer}
...

Data From a CSV File

There are lots of reasons why tests require external date: login credentials, order or submission details, search parameters, etc. With API testing, these data sets can get pretty huge, sometimes on the order of tens of thousands of rows.

 

NeoLoad can automatically parse and represent text file contents as variables. The first step is to declare your data as a ‘file variable’:

 

Editor<projects/example_1_2_variables/project.yaml>:


name: NeoLoad-as-code-examples-1_2
includes:
- oauth_workflow.yaml

variables:
- file:
    name: cities
    path: data/cities.csv
    start_from_line: 1
    delimiter: ","
    is_first_line_column_names: false
    column_names: ["City"]
...

The ‘name’ property will be the name of your data set, and columns will be row properties, represented in the following usage format:

 

Editor<projects/example_1_2_variables/oauth_workflow.yaml>:


- request:
    url: /search?format=json&q=${cities.City}
    server: app_server
    method: GET
...

It really is that simple, but consider that data is often different per test environment. In this case, instead of using only one data file and variables list, you could easily split variables definitions into multiple files, one for each environment, then combine these files later in interesting and useful ways per environment. You might also have multiple subdirectories for different data files, then change the main project data ‘path’ specification to include an environment variable provided at runtime:

 

Editor<projects/example_1_2_variables/project.yaml>:

...

variables:
- file:
    name: cities
    path: ${ENVIRONMENT}/cities.csv
...

A useful analogy for overlaying YAML files together dynamically is like drawings on flat plastic transparency sheets. Layering multiple sheets together produces a composite project that can be as dynamic as you need for development, testing, and continuous delivery activities alike.

 

Reusable Server Details

In our very first example, we wrote a request specification that included the full URL including protocol, host, path, and query. However, if any of these details change, such as when new build candidates need testing or when the API base path is different per environment, we would have to copy and rewrite our scripts, which is just not good at all.

 

A built-in alternative in NeoLoad is to declare server details once, then refer to their common name at the request level:

 

Editor<projects/example_1_2_variables/project.yaml>:

...

servers:
- name: app_server
  host: host.docker.internal
  port: 3000
...

You may have already noticed this in the prior examples that each request can define which host it reuses:

 

Editor<projects/example_1_2_variables/oauth_workflow.yaml>:

...

- request:
    url: /search?format=json&q=${cities.City}
    server: app_server
    method: GET
...

This approach simplifies the ‘url’ to just a path, making it easy to control server details in one place (such as a ‘servers.yaml’ per environment, just like the pattern discussed in the data variables section above.

 

You can also transpose variables in for host, port, and paths, making it possible to construct flexible, resilient API tests that adapt to highly dynamic test environments:

 

Editor<projects/example_1_2_variables/project.yaml>:

...

servers:
- name: app_server
  host: ${APP_HOST_NAME}
  port: ${APP_PORT_NUM}

variables:
- constant:
    name: API_BASE_PATH
    value: /platform/api/v3

...

- request:
    url: ${API_BASE_PATH}/posts?order=desc&orderby=post_date
    server: app_server
    method: GET
...

User Path Lifecycle

By default, the NeoLoad engine spins up ‘virtual users’ (a.k.a. “VU”, a throwback to when everything was just about users on a web page) as completely separate threads. The concept of a VU still applies in API terminology to refer to a “consumer”, and goals can be set for how much volume you want to throw at a service using “VU count”.

 

Then there’s ‘iterations’. A VU can (and often does) run many iterations of itself during a load test. For a 50VU test 5 minutes long, NeoLoad spins up 50 threads and keeps each thread running over the actions of the user path as defined continuously until the test duration elapses. This is one of the most common types of load test and is the simplest to understand, but there are many other ways to configure NeoLoad to iterate and maintain concurrent load threads.

 

There may be times where you want to perform an action once, such authentication or search, run a particular workflow many times, then maybe log out or something. NeoLoad user paths provide three base action groups for handling these types of semantics: Init, Actions, and End.

 

Editor<projects/example_1_2_variables/oauth_workflow.yaml>:

...

user_paths:
- name: oauth_workflow
  init:
    steps:

... do some workflow setup activities, login, etc for this thread ...

  actions:
    steps:

... these steps use iteration-based load duration and variation ...

  end:
    steps:

... like ‘init’, these steps execute only once per parallel thread ...


...

Note: if you define actions in the init container, NeoLoad will automatically infer that your VU session should not reset after each iteration; this is opposite when there are no actions defined in either ‘Init’ or ‘End’ groups. Basically, if you use the init/end steps, session details such as cookies and VU-level variable cycling will not be reset when it loops to the beginning of the actions steps again.

 

Though not in use [much] with most modern APIs, it’s important to note that cookies are handled automatically by NeoLoad, meaning that if a server responds with ‘Set-Cookie’ headers, NeoLoad will automatically append these to subsequent request that apply, just like a browser would do.

What’s Next?

There’s a whole lot more to NeoLoad than this, but hopefully you’ve got a taste of how As-Code works and what it can do for you.

 

We welcome feedback and questions with open arms: as-code-feedback@neotys.com

 

We also selectively provide engineering coaching sessions, so contact sales@neotys.com if you get stuck and we’ll figure out how to help you out.

 

Appendix

Verifying Your Docker Installation

Usually, Docker is very easy to install. Here are a few useful links for getting set up properly.

Installing Docker in WINDOWS

Installing Docker on a Mac

Installing Git CLI Tools

Generally, this is very simple. In fact, you may already have these installed. To check, open a command prompt or terminal, then type: git –version

If you don’t have it installed, follow the instructions below:

Installing Git for Windows

Follow the official instructions here.

Installing Git for Mac

Follow the official instructions here.

 

Trying As-Code Out on a Cloud VM

Sometimes installing prerequisites or enabling virtualization on your corporate laptop isn’t possible. If you have access to Amazon EC2, Azure, or Google Cloud, you can easily run these samples in a sandbox environment of your choosing.

 

For Amazon EC2, we suggest an m2.large.

No matter what cloud, we suggest either a Ubuntu image with LXDE (Desktop experience) or Windows Server 2019 with Containers.

If you chose to run Windows, you will likely have to switch from using Windows containers to Linux containers as the world of all things relevant revolves around linux-variants and Microsoft has chosen to go their own route when it comes to containerization. Here are some useful steps to do so on Windows Server 2019: https://www.altaro.com/msp-dojo/linux-containers-windows-server-2019/

Keep Me Informed