The Renault Nissan Mitsubishi Alliance sets up a performance testing platform for the connected vehicle using NeoLoad.
The Renault–Nissan–Mitsubishi Alliance is one of the world’s largest car manufacturer. Formerly an industrial company, the Alliance is facing new challenges associated with adding software components to its products and services. As cars are embedding an increased amount of technology, the connected vehicles represent a significant stake for the organization. Whether referred to as the connected car, IoT or automotive intelligence, digitalization and connectivity are changing how people are moving and using their vehicles. Car manufacturers need to develop software, deploy central and embedded IT systems, secure and manage connected cars across the world.
In this context, performance testing is a critical step because the use case includes millions of cars. Load testing is key. And since the service may affect people lives, systems must be operating with the highest reliability standards, thus the need for in-depth performance load testing.
In recent years, connected vehicles have been a significant play in the automotive industry, bringing customers an array of new services through cloud and web connections. Traditional functions like navigation are now commodities, and new services are emerging, such as the ability to make emergency calls or to remotely open a car. This may appear as a technical detail, but can genuinely revolutionize the car rental industry. This simple capability could alter the way fleet management is operated, for example.
The connected car is also responsible for the first essential element of a broader concept – the autonomous vehicle. While a compelling use case, the delivery of these new services would bring considerable technological challenges, including load and performance testing.
A Technology Challenge
Renault itself manufactures thousands of cars each day. This comprises more than one million connected vehicles that will be shipped by 2020, and this will only grow over time as the adoption of new usages increases. The volume of cars deployed is a challenge for performance testing because the load testing infrastructure needs to generate a vast amount of virtual users.
Each vehicle generates more than three thousand daily transactions, including the remote control of the car, the vehicles’ “health” and navigation (4 requests every 2 minutes). The GPS update via Firmware Over-The-Air (FOTA) done wirelessly), happens about ten times a year, generating additional transactions and using a significant amount of data.
In addition to car health, the organization needs to ensure its software systems enable fleet management, especially as volume and demand increase. As a result, Renault’s servers are updated every five seconds, sending alerts when required.
Beyond car/transaction volume, the diversity of technologies involved presents another challenge when trying to simulate the behavior of the end-to-end application.
- Modern web technologies
- API technologies
- Double authentication
- Mobile and cloud technologies merged into a multi-components
Renault answer – a robust set of expertise, and appropriate tooling has been deployed to execute performance load tests before cars are available to users on the roads.
Performance Load Testing Strategy
The criticality and size of the application led to an intensive performance test campaign with NeoLoad as the software platform was being delivered. Every release would be tested under realistic load conditions to identify potential performance bottlenecks updating the code or infrastructure.
Typical NeoLoad tests would simulate 25K vehicles connecting simultaneously to the central application from several different geographic locations in Europe, USA, and Japan (including other remote areas around the world). In this context, it is crucial to rely on a performance testing cloud platform that enables user simulation from many different geo locations to fairly represent realistic diversity.
The technical environment of the application to test is heterogeneous:
– Azure cloud platform for the central server part with Pivotal Cloud Foundry to manage the service
– API technologies (based on XML or JSON)
– Dialogues between the vehicles and central servers using the secured MQTT protocol (certificate management in both directions). Note this type of technology is not common in the automotive industry.
– Double authentication with SSL V3 certificates. Not a new technology, but complex to use and to test
– OMA/DM (Open Mobility Alliance protocol)
NeoLoad met the challenge: combine, into at least one test, all required protocols (at the same time) to simulate a realistic behavior of the car to central server communication. To achieve this level of acceptance, the NeoLoad’s ability to reproduce realistic protocol transactions is paramount.
The NeoLoad load testing tool used during the campaign served several objectives:
– Simulate vehicles to inject load into the central system (its primary use)
– Emulate vehicles that receive requests (like firmware updates) with the tools acting as a receiver
– Monitor the central system on its own and in interface with the Dynatrace APM tool used to monitor Azure and Pivotal Cloud Foundry
– Create vehicle data to insert in the database. Renault itself could generate no more than ten files to fill the database. With NeoLoad, we have been able to create tens of thousands of records
– Act as a scheduler: to make sure that the update happens at the right time
An Organizational Challenge
The connected car project is based on several applications having many components across many connected vehicle types. NeoLoad enabled disparate teams from Europe, US, and Japan, etc. to work together sharing test assets and test execution.
What did they achieve?
The team has been able to regularly identify and explore performance bottlenecks earlier in the software development lifecycle – an identification which has led to successful application code and infrastructure optimization. The team has been able to scale vehicle test capacity from 10K to 1 million.
… NeoLoad’s plug-and-play UI was so easy-to-use ourselves that we were able to train colleagues within a few days quickly
“NeoLoad met the challenge to combine, into one test, all required protocols to simulate a realistic behavior of the connected car to central server communication. NeoLoad’s ability to reproduce realistic protocol transactions is paramount.”