Performance testing is not a simple subject and many leading experts still do not fully agree on the topic. How it's used and how people mis-use it has been a topic of debate for as long as computer systems had to support multiple users. Not is measuring the performance of an application had to measure, it's even harder to predict and it can even have an impact on penetration testing as both (D)DoS attacks and performance testing attempt to emulate great volumes of users. Knowing this it becomes easier to see why managers should start paying attention to this obscure branch of software quality charactaristics.
All of this originates from somewhere, the big bang as you will and we will need to go back to the beginning to find out more about this monster. When software developers make their artworks on the keyboard, they do so on the instruction of an analyst who designs a feature according to their needs. These features will include requirements like "As a user i need to be able to log in if i enter a correct username and password".
I intentionally made a mistake in that previous sentence that's not so easy to find, go back and see if you can spot it. The mistake i refering to is that i said a correct username and password but never mentioned they had to be a valid combination. This shows how easy is to make a small mistake that might be misinterpreted and it's an issue companies have been plagued with for ages. These are requirements and they are functional requirements. We also have non-functional requirements and they describe things like performance testing and security testing. For our purposes, we need to talk about performance testing and how it's done because if we desribe our variables wrong, we might overload our production servers which might even lead to 1 person easily killing a whole system and when combined with an IP rotator, performance testing tools might become powerful weapons in the hands of a bad actor.
When creating a performance testing script, we have several variables that we need to take into account:
The first one might seem to speak for itself but it's not as easy as it seems. Getting the statistics for existing calls is hard enough as it is without proper logging but with new features, these statistics often have to get estimated. This statistics can be for example "100 calls per minute to /login" or "500 calls per hour on the invoices endpoint.
The second point is because every user should be treated differenly when it comes to caching and these caches can make or break a system as requesting a resource from a caching system costs far less resources than rebuilding the resource from scratch on every request. This is why it's very important to treat every user as someone who does not have a cache themselves in their browser.
The sequence of calls matters a lot because calls might influence each other and we can never predict this fully in advanced, that's why tracing is so important. This will usually also tell us a little bit about the load distributed over the different endpoints as well.
At the end we should requirements to start writing our scripts and to be able to manipulate the expected load after creating the proper load profiles.
We will start with the load test, this is a test where will go a small % over the expected amount of traffic, it's usually between 25 and 50% higher than what we expect in production environments and it's going to consist of calling the server but we have to be careful to set the proper caching protocol here and that we give our scripts a warmup grace period.
Afterwards the duration tests come into play and this is where it might get more interesting for potential black hat hackers as we try to see if memory leaks exist within the application that could cause a DoS attack if a specific call is make often in a row for a long duration of time.
The last type of test I will cover in this article is the stress test, in this test we want to put our system under a very heavy load to determine how the system will react and at what point it will break.