Sometimes even 1% degradation can be a huge problem.

An example: Let’s say we have a web server that processes requests. We host this server in the cloud, and we pay a cloud provider for the time resources at a fixed rate. In our spherical example in a vacuum, each request always takes 100 ms. 1% degradation means that we will get 101 ms per request after a deployment. If we have billions of such requests, the total processing time will increase noticeably.² The most important thing is that our bills will also increase by 1% .

Sometimes even 500% degradation can be not a problem.

An example: We have a server that displays statistics about user activities. Let’s say that we don’t need real-time statistics; it’s enough to refresh it daily. So, we have a console utility that regenerates a statistic report and deploys it. With the help of cron,³ we run it every day at 02:00 AM. The utility takes 1 minute, so the report is ready at 02:01 AM. A developer from your team decided to implement additional “heavy” calculations: now the report contains new useful information, but the total generation time is 6 minutes; the report is ready at 02:06 AM. Is this a problem? Probably not, because analytics will review the report only in the morning. If the utility takes 10 hours, it can be a problem, but nobody cares about five extra minutes in this case.

Sometimes it’s impossible to talk about degradations in terms of percentages.

An example: Because of a complicated multilevel hierarchical cache, 20% of requests take 100 ms, 35% of requests take 200 ms, and 45% of requests take 300 ms. After some changes, 20% of requests take 225 ms, 35% of requests take 180 ms, and 45% of requests take 260 ms. Is this a good change or a bad change? Do we have a performance regression in this case? (Try to calculate the average processing time for both cases.) Well, this is another trade-off problem: we can’t answer this question without business requirements.

Evaluation of the effectiveness of performance tests

Detection of weaknesses and pieces of code that should be covered by additional performance tests

If you detected many problems in time, it will encourage the team to write new performance tests

If you didn’t have any significant issues (both detected and nondetected), you probably don’t need performance tests for these projects, and it doesn’t make sense to invest time into it in the future.

Automatic reports

You can generate a full report about the problem automatically. Such a report could include links to the commits (if you have a web service that allows browsing your code base), a list of authors of these changes, performance history of this test, links to other tests from the same test suite with new performance problems (they can be related), and so on. The main idea here is that the analyst shouldn’t look for additional data; all necessary information should be collected automatically. You can even automatically create an issue in your issue tracker and easily track all performance problems.

Automatic bisecting

It’s not always possible to run all performance tests for each commit. Imagine that one of your daily performance tests is red and there are N=127 commits in this day by ten different people. How do you find the commit that introduces the problem? It’s a good idea to start to bisect these commits. Let’s check the commit 64 (for simplification, assuming that we have a linear history without branches). If the test is red, it means that the problem was introduced before this commit, and we are going to check commit 32. If the test is green, it means that the problem was introduced after this commit, and we are going to check commit 96. If we continue this process, we can find the commit with problem after log₂(N) iterations (in the perfect world without branches). Manual bisecting is a waste of developers’ time. This process can also be automated: the report should include the specific commit and the author of this commit (this person should start to investigate the issue).

Automatic snapshots

One of the first steps in such investigations is profiling. Once we get a slow test, we can automatically take a performance snapshot before and after the change. In this case, the analyst can just download both snapshots and compare them. It can allow finding the problem even without the need to download the sources and build it locally: many stupid mistakes can be found only with the snapshots.

Automatic step-by-step analysis

If you have a 1-minute degradation in a huge integration test, you probably have a problem in a single subsystem instead of a project-wide problem. In this case, you can measure separate steps for both cases and compare them automatically. After that, a notification (or an issue) can contain additional information like “it seems that we have a problem with these two steps; the rest of the steps doesn’t have noticeable degradation.”

Automatic continuous profiling

If you have a pool of servers with services that sometimes suffer from accidental performance drops, you can try to profile them automatically. If the overhead of such profiling is too big, you can randomly profile only a part of the pool. For example, pick 10% of the servers and profile them for 30 seconds, then pick another 10%, and so on. You can play with the exact numbers and get a profile snapshot at the moment the problem reproduced (maybe it will not be on the first try). The randomized approach helps to reduce the profiling overhead on your production system.

Method cold start

When you run a method for the first time, a lot of time-consuming things may happen on different levels: from JIT compilation and assembly loading on the runtime level to some first-time calculations for static properties on the application logic level.

Feature cold start

Difference between cold and warm time for a method can be negligibly small. However, it can be noticeable when we are talking about thousands of methods and many assemblies. Because of that, a user can experience delays when he or she launches a feature for the first time (especially if this feature involves tons of methods that were not invoked before).

Application cold start

Startup time is important for many kinds of applications. And it’s definitely crucial for desktop and mobile applications. The perfect situation is a situation when the user instantly gets a ready application after a double-click on a shortcut (or launching it any other way). Any delay can make him or her nervous. Imagine a situation when you should quickly make a few edits in a file. You open it in your favorite text editor and… . And you have to wait a few seconds until the text editor is initialized. If you edit files often and close the editor each time, these few seconds can be irritating. For some people, startup time is critical; they might prefer a pure-featured text editor that starts instantly over a full-featured text editor that starts in a few seconds.

OS cold start

If your benchmark interacts with different OS resources, a physical restart can be required for a cold start test.

Fresh OS image

Sometimes it’s not enough to reboot the operating system; we may need a fresh image of the system. The old test runs can make any changes on the disk that can be important for subsequent launches. For example, Rider uses a pool of TeamCity agents for running hundreds of build configurations with tests every day. TeamCity refreshes the agent images once per several days: then the fun begins. Sometimes, we have a significant performance difference between the last (warmed) test run on the old image and the first (cold) test run on the new image (without any changes in the source code base). We don’t use a fresh OS installation each time, because such approach has a huge infrastructure overhead and the described problems are not frequent.

We want to know the duration of list.Add .

Actually, we want to gain knowledge of the list.Add duration and use it for solving a real problem (e.g., writing a fast algorithm). The solution of the problem is our “true” goal, but not the knowledge itself. This is important because the correct way to benchmark list.Add depends on how you are going to use it.

We want to add many elements in a list and want to know how much time does it take.

In this case, we probably have to benchmark the addition of N elements instead of a single one. Remember that not all of the Add calls are equal: some of them can produce resizing of the internal array. You can play with the initial state, the initial capacity, the number of elements, and so on. If you want to know the duration of the adding of N elements, you should benchmark this. The performance cost of a single Add is useless for you because you can’t multiply it by N (in the general case) to get the result.

We are going to make a few edits in the Add implementations and check for performance improvements/degradations.

Any performance changes in the Add method will also affect the performance of the Add/RemoveAt pair. It will be hard to say something about how much the edits affect the Add method (quantitative changes), but we can say is it better or worse (qualitative changes). Also, we still have to check cases with the resizing of the internal array carefully.

We are going to use a list as a stack (with Push/Pop operations) with the known maximum capacity and want to know the duration of the “average” operation.

In this case, the Add/RemoveAt benchmark is a great solution because there is no difference between Add and RemoveAt here: we have to measure these methods together.

Portability

Results almost always don’t depend on hardware: we should get the same result on slow and fast computers.

Benchmarks take less time

The inspection performance impact can be noticeable only in huge files. There are hundreds of inspection; we have to wait too long until we benchmark each inspection on each huge file from the test data. The asymptotic approach allows getting reliable results in less time. We can apply an inspection to a few small files, measure the analysis durations, and calculate the asymptotic complexity. Thus, we can check that the the inspection works fast enough without using huge files.

Many iterations

We can’t build a regression model with one or two iterations. We have to run many iterations if we want to build a reliable model that produces correct results.

Complicated implementation

It’s not easy to build a good regression model. If you are lucky enough, your performance function is polynomial. If you are not lucky, the performance function can’t be approximated by an analytic function. Even if the function type is known (and you have only to find the coefficient), it’s not always easy to build such model with a small error.

(A) “How much time (T) do we need to process N requests?”

The metric here is the latency of processing of N requests (the time interval between the start and end of processing).

(B) “How many requests (N) can we process in the fixed time interval T?”

The metric here is the processing throughput. Such case is also called capacity planning or scalability analysis .

(A) In the first case, N is fixed. Thus, we have to do N iterations and measure the time between start and finish:

(B) In the second case, T is fixed. We don’t know how many requests can we process, so we will process requests until the time is over. In real life, it’s typically complicated multithreaded code, but we can write a very simple single-threaded benchmark:

(A) Let’s check how much time (T) it takes to process N = 1024 requests. When C = 2, T = 20sec. When C = 4, T = 40sec. The performance drop is 40sec/20sec or 2x.

(B) Let’s check how many requests (N) we can process in T = 20 seconds. When C = 2, N = 1024. When C = 4, N = 32. Performance drop is 1024/32 or 32x.

Explicit performance tests

These tests were designed to evaluate performance. Explicit tests may require special hardware and tricky execution logic (with warm-up, many iterations, metrics calculation, and so on). The result of such test is a conclusion about performance (like “the test works two times slower than before” or “the variance is too huge”).

Implicit performance tests

These tests are “usual” tests that are designed to check logic. Each run of such tests has a duration, its performance number, which we get as a side effect. The result of such a test is a conclusion about correctness (green status for correct logic and red status for incorrect logic). “Implicit performance tests” means that these tests are not designed as performance tests, but we still can use them as such.

“Mixed” performance tests

It may sound obvious, and we will not discuss such tests in detail, but I still have to highlight this idea: you can check logic and performance at the same time. For example, we can write a huge integration stress test that covers the most performance-critical pieces of our code. Such a test can check that everything works correctly even under load (some race conditions can appear in such situations) and that we don’t have a performance regression in such a case.

Persistent CI agent

When we measure performance, it’s a good idea to run performance tests on the same hardware each time. It’s very hard (or sometimes impossible) to evaluate the performance impact of your changes when you compare the “before” performance data from one agent with the “after” data from another agent. It’s always better to have persistent CI agent (or set of agents) for explicit performance tests. This is not mandatory, but it’s highly recommended. In case of implicit performance tests, there is no such requirement⁸; they should work correctly on any agent.

Virtualization

Virtualization is a great invention that helps us to organize a flexible cloud infrastructure. However, a virtual environment is a poison for the accuracy of explicit performance tests. You never know who else is running benchmarks on the same hardware at the same time. Explicit performance tests usually require a dedicated real (not virtual) agent. Implicit performance tests should work correctly in any environment.⁹

Number of iterations

Most explicit performance tests require several iterations. Remember that performance of a method is not a single number; it’s a distribution. We can’t evaluate errors and build a confidence interval if we have only one iteration. And we can’t compare two revisions if we don’t know errors and variance. Of course, sometimes a test can be too expensive (it consumes too much time), so you can’t afford to run it several times. Implicit performance tests typically need only one iteration.¹⁰

Writing easiness

It’s easy to write implicit performance tests.¹¹ I mean that every method which somehow calls your code can be a test. Different teams have different standards of coding, but most of them agree that the source code should be covered by tests. Some good development practices require writing tests (e.g., before writing a bug fix, you should write a red test for this bug and make it green with your fix). Typically, you get tests as an “artifact” of the development process. You write tests because it will simplify your life in the future and make you more confident in the quality of your code. Most of the unit tests are deterministic: a test is red, or a test is green. Moreover, it’s usually obvious when a test is green. If you are writing a method Mul(x,y) that should multiply two numbers, you know the expected output. Mul(2,3) should be 6. Not 5, not 7; there is only one correct answer: 6. When we are writing explicit performance tests and making performance asserts, it’s always complicated. For example, yesterday Mul took 18 nanoseconds; today it takes 19 nanoseconds. Is it a regression or not? How should we check it? How many iterations do we need? How should we evaluate errors? And the most important question: is the test red or green? If you have clear answers to all questions about performance asserts, ask your teammates about it. Are you sure that you have the same point of view? It’s so hard to write performance tests because there are no strict rules here. You should come up with your own performance asserts that satisfy your performance goal. It’s hard because there is no “absolute green status,” and there is no single “correct” way to write “performance asserts.” There are only trade-offs.

Time of execution

Speaking of trade-offs, the most interesting one is between accuracy and the execution time. Performance tests wouldn’t be so fun if we had unlimited amount of time. I wish I could perform billions of iterations for each of my benchmark or performance tests. Unfortunately, the world is cruel, and we don’t have such opportunities. There is the natural upper limit for the total execution time of a test suite. It can be 10 seconds, 10 minutes, 2 hours, or 5 days: it depends on your workflow. But you have this limit anyway; you can’t spend months and years for a single suite run. It would be great if you could run all of your performance tests during a few hours. If the total time is limited and you have too many tests, you can afford the only small number of iterations. It can be 100 iterations, or 10 iterations, or even a single iteration. And sometimes you have to deal with this single iteration. Implicit performance tests should be as fast as possible, there is no reason (typically) to repeat the same thing over and over. In the case of the explicit performance tests, each additional iteration can increase the accuracy. Of course, there is a “desired” level of accuracy and a “recommended” number of iterations. Usually, it doesn’t make sense to “pay” for additional iterations by execution time after that point.

Variance and errors

Since the explicit performance tests are designed to get reliable performance results, we do everything to stabilize them: use real dedicated hardware, make many iterations, and calculate statistics. In case of the implicit performance tests, we (typically) don’t care about variance and errors: we can run it inside a virtual machine, we can choose a new CI agent each time, we can always do only one iteration, and so on. Variance and errors are typically huge.

Monitoring

Monitoring is a typical solution for web servers: we can watch for life indicators of the server with the help of special tools like Zabbix¹² or Nagios.¹³

Telemetry

Telemetry is a widely used technology in software development¹⁴ that allows collecting information on the usage of user applications. Such data is typically anonymous and doesn’t include any sensitive information. However, it can include important information about the performance of different operations. While usual monitoring is a great approach for web services, telemetry is our main “monitoring” tool for desktop applications (however, it can also be useful for the client side of web services). There is an existed API for telemetry by Microsoft,¹⁵ but we can implement our own set of tools.

For example, Mozilla Firefox collects data¹⁶ about memory usage and operation latencies.

Of course, telemetry can include only general usage data without any performance statistics. For example, .NET Core CLI Tools use¹⁷ telemetry for collecting information about .NET Core SDK usage.¹⁸ The collected telemetry datasets are open and available for everyone, but they don’t include any information about performance.

Common trends

It’s hard to perform a precise analysis, but you can track common trends. For example, you can compare statistics (like average, p₉₀, p₉₉, and so on) of a web page load duration on the previous week (with the previous version of your web service) and the current week (with an updated web service version). If you see a statistically significant difference, it’s a reason for a performance investigation.

Thresholds

If you have a low latency requirement for some operations, you can introduce thresholds and send telemetry data in cases of failure. Imagine that you develop a desktop application and you want to keep the startup time low. Let’s say that 1 second on modern hardware (you can collect information about the hardware as well) is your upper limit. Of course, a user can have some heavy processes running at the same time, so let’s say that the threshold is 2 seconds. If the startup time is more than 2 seconds, a telemetry alarm should be sent. Probably, you will get a few such alarms every day because you can’t control the user environment. However, if you start getting dozens or hundreds of such alarms after the publishing of a new version, you have an issue for investigation.

Manual watching

It’s hard to predict all the things that can go wrong. It’s even harder to automate the analysis of performance plot and write a system that automatically notifies us about all suspicious things. We will talk about performance anomalies later in this chapter. Thus, it’s a common practice when a special person (or a group of people) are looking for performance charts. Popular services require 24/7 monitoring: in case of any problems (not only performance problems but also availability and business logic issues), the reaction must be immediate. Unfortunately, it’s almost impossible to automate this process. But you can use dashboards and alarm systems to make life easier.

External services

In Rider, we have some tests that cover NuGet features like install, uninstall, or restore. The logic of the test is simple: we just check that we can correctly perform these operations in small and huge solutions. Most of the tests are using our local NuGet repository, but some of them are using the nuget.​org and myget.​org servers. The primary goal of these tests is checking that the logic is correct, but we can also use it as performance tests. In Figure 5-2, you can see a typical performance plot for one of our NuGet tests. On March 22, 2018, nuget.​org was down (see [Kofman 2018]). On April 16, 2018, api.​nuget.​org was blacklisted in Russia.¹⁹ On May 6, 2018, there were some serious problems with search API in the NuGet Gallery (see [Akinshin 2018]). We learn about these incidents immediately because we are watching the performance plots all the time. On the one hand, it’s hard to use such tests for honest performance regression testing: we get false positive results (a performance test is red, but there are no changes in the code base). On the other hand, all these problems are relevant to the behavior that users have in the product. It’s good to be notified about it as soon as possible.

External devices

Many years ago, I was involved in an interesting project. My colleagues and I worked on a program that communicates with OWEN TRM 138.²⁰ This is an industrial measurement device with eight channels that can measure different characteristics, such as temperature, amperage, and voltage. If you connect it to eight different points of a machine detail and measure the temperature at these points, the program can extrapolate the data and build a 2D map of the temperature surface. Everything should work in real time: if the user changes some connection points, the map should be recalculated instantly. The real-time visualization was an important feature, so we checked that the time intervals between changes in the experimental setup and a new visualization. Unfortunately, sometimes we experienced unpredictable delays: OWEN TRM 138 provided data a few seconds late. Thus, it was almost impossible to make reliable performance measurements (because the delays were unpredictable). Eventually, we stopped to measure the whole cycle and started to measure different stages: fetching data, extrapolating, building an image, and so on. It solved the problem because measurements of the device-independent stages were pretty stable.

Stress/load tests

You should always know the limitations of your software product. Usually, it’s a good idea to cover these limitations by performance tests. When we are talking about performance stress tests, we usually mean integration tests. Such testing is especially useful for web services that handle a huge number of users at the same time. A typical mistake for server application benchmarking is focusing only on a situation without load (we send a single request to the server and measure the response time). In real life, you have many users who send requests at the same time. The most interesting thing is that the way the server process these requests depends on the volume of these requests. Fortunately, there are existing solutions that can help to automate this process (e.g., Apache JMeter, Yandex.Tank, Pandora, LoadRunner, Gatling).

User interface tests

It’s not always easy to implement a correct infrastructure for user interface tests, because you usually can’t run it a “headless” mode; you need a “graphical environment” for such tests. For example, in the IntelliJ IDEA code base, there are some user interface tests that check whether the IDE interface is responsive. In the CI pipeline, these tests are running on dedicated agents that are connected to physical 4K monitors.

There are also many libraries and frameworks that can help you to automate testing of the interface in your product (e.g., Selenium).

Fuzz tests

Cliff

A cliff degradation is a situation when you have a statistically significant performance drop after a commit. You can see an example of the cliff degradation in Figure 5-3.

Incline

An incline degradation is a situation when you have a series of small performance degradations. Each degradation can’t be easily detected, but you can observe a performance drop when you look at the history for a period. For example, your current performance can be 2 times worse than a month ago, but you can’t point to a commit that ruined everything because there are too many commits with a small performance impact. You can see an example of the incline degradation in Figure 5-4.

Expected accelerations

Unexpected accelerations

An unexpected acceleration is always suspicious. You can meet a lot of developers who can say something like the following: “I didn’t change anything, but now the software works faster. Hooray!” Unfortunately, an unexpected speedup can often mean a bug. I had observed many situations when a developer accidentally turned off a feature and got a performance improvement. Such situations can pass all the tests, but you can’t hide them from the performance plots! Investigations of unexpected accelerations don’t help you with performance, but they can help you to find some bugs.

Suite degradation

Most of the projects have a test hierarchy. You can have several projects in a solution, several test classes in a project, several test methods in a class, and several input parameter sets for a method. When you are looking for performance degradation or another performance anomaly, you should try to highlight test suites²² that share the same problem.

Paired degradation/acceleration

Correlated changes in time series

What is the maximum acceptable duration of a single test?

What is the maximum acceptable duration of the whole test suite?

Check out the durations of tests in your project. What is the typical duration of the whole test suite? Find the slowest test (or a group of the slowest tests). Is it possible to test the same thing in less time?

Run tests in parallel if possible

If you are care only about the total build time, you should try to run tests in parallel. Be careful: in this case, you will not get reliable performance results. Also, it’s not always possible to run arbitrary tests in parallel because they can work with the same static class or share resources (e.g., files on a disk).

Replace integration tests by unit tests if possibleIf you have a ready framework for integration tests, it’s usually much simpler to write an integration test instead of a unit test. Unit tests require some effort: you have to isolate a part of the system correctly, mock other parts, generate synthetic data, and so on. You typically shouldn’t do it in integration tests: the whole system with real data is ready for your checks. However, if you want to check only a single feature, a unit test is a recommended way. If you run the unit tests before the integration tests, the increased feature covering by additional unit tests can also improve the build time: in case of failed unit tests, you can skip the integration test phase .

Too many outliers

Sometimes you make some changes (for example, change API for reading data from the disk) and accidentally increase the number of outliers (e.g., 40–50 instead of 10–15). In this case, the standard deviation is also increased, so you have an additional way to detect the problem.

Extremely huge outliers

Outliers are always bigger than the mean value. It’s usually OK if the difference between the maximum outlier and the mean value is huge (e.g., mean = 300 ms, max = 2600 ms). However, sometimes these values are extremely high (e.g., mean = 300 ms, max = 650000 ms). Such a situation can be a sign of a serious bug that can hurt your users .

Changes in tests

This is one of the most common false anomalies. If you make any changes in a test (add or remove some logic), it’s obvious that the test duration can be changed. Thus, if you have a performance anomaly like degradation in a test, the first thing that you should check is if there are any changes in the test. The second thing for checking is any changes that spoil the performance on purpose (e.g., you can sacrifice performance for the sake of correctness).

Changes in the test order

Changes in CI agent hardware

It’s great if you can run performance tests on the same CI agent (a physical machine) all the time. However, the agent can break down, and it can be hard to find an identical replacement. Any changes in the environment can affect performance: from a minor change in the processor model number to the RAM memory size. It’s always hard to compare measurements from different machines because the actual changes are unpredictable. If you want to perform nanobenchmarks, you typically need a set of identical physical CI agents.

Changes in CI agent software

You can get some trouble with the same agent without hardware replacement. It’s a common practice when admins install operating system updates from time to time. They can be minor security updates or major OS updates (e.g., Ubuntu 16.04 → Ubuntu 18.04). Any environment change can affect performance. This leads to a situation when you see a suspicious degradation or acceleration on performance plots without any changes in the source code.

Changes in CI agent pool

Only the luckiest have an ability to run tests on a CI agent pool with dedicated identical machines. A much more frequent situation is a dynamic pool of CI agents: you can’t predict which hardware/software environment will be used for the next test suite run. Something is constantly changing in such a pool: some machines are turned off, some machines are put into operation, some machines get updates, some machines are occupied by developers who do performance investigations, and so on. Such a situation means increased variance (because of the constant jumping between) and performance anomalies based on the changes in the pool. In Figure 5-8, you can see a performance anomaly for MonoCecil test in Rider for macOS agents around October 20. Nothing was changed in the source code; the degradation was caused by a planned update of all macOS agents. The updating process consumes CPU and disk resources and affects the performance of tests (it wasn’t a special performance test; it was a regular test that runs on regular agents from the pool). As soon as the update finished, the performance returned to the “normal level” (if you can say “normal” for a test with such variance).

Changes in the external world

If you have any external dependencies, they can be a persistent source of performance anomalies. Unfortunately, it’s not always possible to get rid of these dependencies. Once a dependency becomes a part of your tested logic, you start to share the performance space with it. The classic example of such a dependency is an external web service. You can download something from the web or test an authentication method. For example, I had such a problem with NuGet Restore tests in Rider. These tests checked that we could restore packages correctly and fast. The first version of these tests used nuget.​org as a source feed for all NuGet packages. Unfortunately, these tests were very unstable. Once a day, there was such a situation in which one of the tests was failing because of slow nuget.​org responses. On the next iteration, we created a mirror of nuget.​org and deployed it on our local server. We (almost) didn’t have fails any more, but the variance was still huge for these tests. On the final iteration, we started to use a local package source (all the packages were downloaded on the disk before the test suite is started). We got (almost) stable tests with low variance. It should be noted that it’s not an honest test refactoring. We sacrificed a part of the logic (downloading packages from a remote server) for the sake of the false anomaly rate.

Any other changes

Our world is constantly changing. Anything can happen at any minute. You should always be ready to meet false performance anomalies. A performance engineer who is responsible for the processing of the anomalies should know what kinds of false anomalies are frequent for the project infrastructure. Checking if an anomaly is false should be the first thing that you should do before a performance investigation. This simple check helps to save time and prevent a situation in which a false anomaly becomes a Type I (false positive) error.

Performance degradation

It may sound obvious, but the biggest problem with this anomaly is the degradation of the performance. Usually, people start to do performance testing because they want to prevent degradations.

Hidden bugs

Missed asserts are bugs in tests, but you can have similar bugs in the production code. If a test has a huge variance, the first thing that you should ask is the following: “why do we have such variance here?” In most cases, you have a nondeterministic bug behind it. For example, it can be a race condition or a deadlock (with termination on timeout but without assert).

Slow build process

You have to wait too long before all tests are passed on a CI server. It’s a typical requirement that all tests should pass before an installer will be available, or a web service will be deployed. When the whole test suite takes 30 minutes or even 1 hour to run, it’s acceptable. However, if it takes many hours, it slows down your development process.

Slow development process

If a test is red and you are trying to fix it, you have to run the test locally again and again after each fix attempt. If a test takes 1 hour, you have only eight attempts with a standard 8-hour working day. Moreover, it doesn’t make any sense to wait for the test result without any actions, so developers often switch to another problem. The developer context switch is always painful. Also, the huge test duration implies huge errors. When a test takes 1 hour, you are usually OK with an error of a few minutes. In such a situation, it’s hard to set up strict performance asserts (we will talk about this later).

Unpredictably huge duration

We already talked about a huge test duration: this is not a good thing. When you have an unpredictably huge test duration, it’s much worse. In such case, it’s hard to work on the performance of such tests. If you have timeouts (which are popular solutions because tests may hang), the test can be flaky because the total duration can sometimes exceed the timeout.

It’s hard to specify performance asserts

Let’s look again at Figure 5-6. You can see a performance history plot of a concurrency test from the IntelliJ IDEA test suite. Some of the runs can take 100 seconds (especially on Windows), and others can take 4000 seconds (especially on macOS). We can observe both kinds of values on the same revision without any changes. Imagine that you introduce a performance degradation. How do you catch it? Even if you have a performance degradation of 1000 seconds, you can miss it because the variance is too huge.

Missed asserts

Many times, I have seen tests with green performance history as follows: 12.6 sec, 15.4 sec, 300.0 sec, 14.3 sec, 300.0 sec, 16.1 sec, … . A typical example: we send a request and wait for a response. The waiting timeout is 5 minutes, but there is no assert that we got the response. After 5 minutes, we just terminate waiting and finish the test with the green status. It may sound like a stupid bug, but there are a lot of such bugs in real life. Such tests can be easily detected if we look for the tests with extremely high outliers.

Surprising delays in production

Have you ever had a situation when you do an operation that is usually performed instantly, but it hangs an application for a few seconds? Such situations are always annoying users. There are many different reasons for such behavior. Usually, it’s hard to fix them because you typically don’t have a stable repro. However, some of them can also be a cause of outliers on your performance plot. If you systematically have outliers on a CI server, you can add some logs, find the problem, and fix it.

Hacks in test logic

Have you ever had flaky tests with race conditions? What is the best way to fix such tests? There is an incorrect but popular hotfix: putting Thread.Sleep here and there. Usually, it fixes the flakiness; the test is always green again. However, it fixes only symptoms of a problem, but not the problem. Once such fix is committed, it’s hard to reproduce this problem again. And it’s hard to find tests with such “smart fixes.”²⁵ Fortunately, such hacks can be seen with the naked eye on the performance plots. Any Thread.Sleep calls or other hacks that prevent race conditions or similar problems can’t be hidden from a good performance engineer.

False anomalies

The main problem with a false anomaly is obvious: you spend time on investigations, but you do not get a useful result.

Systematic monitoring

This is the most important recommendation: you should monitor performance anomalies all the time. Since, a real application can have hundreds of them, you can use the dashboard-oriented approach: for each anomaly, we can sort all tests by the corresponding metrics and look at the top. Look at the tests with the highest duration, the highest variance, the highest outliers, the highest modal values, and so on. Try to understand why you have these anomalies. Do you have any problems behind them? Could you fix these problems? You can look at such a dashboard one time at month, but it will be much better if you will do it every day: in this case you can track new anomalies as soon as they are introduced.

Serious anomalies should be investigated

If you systematically track anomalies, you can find a lot of serious problems in your code. Sometimes, you can find performance problems that are not covered by performance tests. Sometimes, you can find problems in business logic that are not covered by functional or unit tests. Sometimes, it turns out that there are not any problems: an anomaly can be a false anomaly or a natural anomaly (which is caused by “natural” factors you can’t control like network performance). If you don’t know why you have a particular anomaly, it’s a good practice to investigate it. If you can’t do it right now, you can create an issue in your bug tracker or add the anomaly to a “performance investigation list.” If you ignore found anomalies, you can miss some serious problems, which will be discovered only in the production stage.

Beware of high false anomaly rates

Beware of alert fatigue

It’s great when you can track down all your performance problems. However, you should understand how many issues can be handled by your team. If there are too many performance anomalies in the queue, the investigation process becomes an endless and boring activity. You can’t fix performance issues all the time: you also have to develop new features and fix bugs.

Detection time: on time.

The best thing about this approach is simple: we detect all performance degradations in advance automatically. There is no need to solve any new performance problems because we don’t have any of those (in theory, of course).

Analysis duration: short.

Since we won’t wait too long before our changes will be merged, the precommit tests should work quickly. It’s great if a typical precommit test suite run doesn’t take more than a few hours.

Degree of degradation: huge.

Of course, there are some limitations. We don’t have any possibility of doing a lot of iterations (because we have to run all the tests very quickly). Thus, we can catch only huge degradations (e.g., 50% or 100%); it’s almost impossible to detect small degradations (e.g., 5% or 10%). If we try to do this, it will increase the total run duration or Type I (false positive) error rate.

Process: automatic.

I just want to repeat one of my favorite parts about this way: it’s completely automatic, meaning that no human actions are required.

Detection time: 1 day late.

With daily tests, we detect performance degradations when they are already in master.

Analysis duration: up to 1 day.

Daily tests don’t have “a few hours run” limitation; we can use up to 24 hours. If that’s not enough, we can try weekly tests and spend up to 7 days per a test suit.

Degree of degradation: medium.

Since we have a lot of time, we can do many iterations and detect medium performance degradation (like 5% or 10%).

Process: semiautomatic.

Daily tests should be a part of your CI pipeline; the build server should run them every day automatically. However, if some tests are red (we have a performance degradation), the incident should be investigated manually. Typically there are a few team members who monitor the status of daily tests all the time and notify a team in case of any trouble .

Detection time: late.

Unfortunately, some degradations will be detected late (probably after a week or after a month). However, it’s better to detect such cases after a month inside the team than to let customers detect them after a few months.

Analysis duration: it depends.

We don’t have any duration limitations; we can spend as much time as we want. If we don’t have enough historical data, we can even take specific commits, build them, and run some additional iterations. Everything is possible in the retrospective analysis!

Degree of degradation: small.

We can detect any kind of performance degradations (even less than 1%)! In fact, the main limitation here is how much we are ready to allocate in terms of resources.

Process: semiautomatic.

The same situation as in the case of daily tests: we can run retrospective analysis automatically, but all issues found should be investigated manually.

Detection time: on time.

This approach allows preventing performance degradations before they will be merged into master.

Analysis duration: it depends.

In fact, the merge deadline is our only limitation. We can do as many tests as we want before we are sure that it’s safe to merge it.

Degree of degradation: small.

Since we have a lot of time, we also can do a ridiculous number of iterations, and find even very small degradations.

Process: almost completely manual.

It’s the developer’s responsibility to check dangerous changes; it’s not possible to automate this. If you suspect that you can have some performance problems in your branch, you should run tests manually. If you find any problems, you should investigate them manually. There is no automation here (except for running tests and branch comparison) .

Detection time: very late.

Usually, developers run prerelease performance tests before the release. And they hope that there are not any problems; it’s an additional check just to be sure. However, if you discover a serious performance problem a few days before the release, it can be a huge problem (especially if you have strict deadlines).

Analysis duration: it depends.

Well, it’s up to you: it depends on your release cycle. How much time do you typically have between the release candidate and the actual release? Some teams spend only a few days for the final stage of testing, while others spend months. You should find an acceptable trade-off between how fast you want to deliver your product and how critical performance degradation can be.

Degree of degradation: it depends.

It depends on the duration of analysis. The rule is simple: the more time you spend, the more minor degradations can be found.

Process: almost completely manual.

The same situation as in the usual checkpoint case. You should manually run tests before release, and you should manually check the report and investigate all the issues.

Detection time: late.

This approach allows checking changes that are already merged. Typically, the manual testing is a part of your workflow: you can check your daily builds,³⁰ you can check some internal milestone builds, you can check “checkpoints,” you can check preview versions, and you have to check the release candidate.

Analysis duration: it depends.

It always takes too much time. The exact number of spent hours depends on the target product quality and capabilities of the QA team.

Degree of degradation: huge.

Usually, manual testing allows detecting only huge performance degradations because it’s hard to detect a small performance regression with the human eye.

Process: completely manual.

You start to test software manually, you test it manually, and you investigate it manually. There is no automation here.

Detection time: too late for the current release, but not too late for the next one.

Analysis duration, Degree of degradation, Process: it depends.

It’s up to you how you collect, analyze, and process performance issues after a release.

Wall-clock time

This is an honest test duration. It can be measured via Stopwatch or be fetched from a CI server.

Throughput

How many operations can we process per second?

Asymptotic complexity

What is the asymptotic complexity of your algorithm? O(N)? O(N*log(N))? O(N^3)?

Hardware counters

There are plenty of them. You can use “general” counters for all cases (e.g., “Retired Instructions”) or “specific” counters for specific tests (e.g., “Branch mispredict rate” or “L2 Cache Misses”). We will talk about hardware counters in detail in Chapter 7.

I/O metrics

You can collect all the metrics provided by OS for network and disk operations. It often helps to locate a real bottleneck correctly.

GC.CollectionCount

Single iteration

This is the most popular and simple case: we always do exactly one iteration of a test. On the one hand, it’s great because it’s a very simple situation: we have only one measurement per revision. Performance history looks simple as well; it’s just a function from a commit to a single number (for each metric). On the other hand, we have limited data: we don’t know any information about the performance distribution for the test. Imagine that you have the following measurements for two subsequential commits: 50 ms and 60 ms. Do we have a problem? You can’t say anything about it because you don’t know the distribution.

Many iterations

If you do many iterations, you have much more data! On the one hand, that’s great because you can run many cool analyses. On the other hand, now you kind of have to do these analyses. Additional iterations are not free: you pay for them with time and machine resources. If you decide to do many iterations, you should understand how you are going to use this data (it also helps you to choose the best number of iterations). For example, it allows comparing commits. If you have a (50ms) vs. (60ms) situation, you can’t say for sure that there is a performance degradation here. If you have a (50ms;51ms;49ms;50ms;52ms) vs. (60ms;63ms;61ms;49ms;61ms) situation, you can say that it’s most likely a degradation. If you have a (50ms;65ms;56ms;61ms;58ms) vs. (60ms;48ms;64ms;53ms;50ms) situation, you can say that most likely nothing is changed .

Whole test

This is probably the most common way. You write a test that measures only one target case. Such testing may require a preparation (e.g., you should set an initial state up and warm the target logic up), but one test measures only one thing.

Test stage

In some cases, an honest test separation can be expensive. Imagine that you have a huge desktop application and you want to measure the “shutdown” time: the interval between a moment when a user clicks the close button and the moment when the application process is finished. Such tests require a lot of preparatory work. For example, you can spend 5 minutes for initialization (emulation of active work in the application) and only 1.5 seconds on the shutdown logic. If we perform 12 iterations inside the test, the whole test will take more than 1 hour. A whole hour of testing for a single test that takes 1.5 seconds! That looks like a waste of our time and machine resources.

Unfortunately, we can’t significantly improve the situation for the shutdown test. However, we can something else: we can use these 5 initialization minutes to our advantage! In fact, we have an integration test that takes a lot of time and performs a lot of different operations. Let’s introduce “test stages” and measure each test separately. We can measure the application load time and duration of some typical operations in the same tests. On the one hand, this move looks dirty and breaks the rules of classic unit testing: instead of measuring each feature in a separate test, we measure all kinds of different stuff in the same test. On the other hand, we have no choice (don’t hate the player; hate the game!). Tests should be fast. In the case of performance tests, it’s impossible to run them really fast, but the whole performance testing suite should take a reasonable amount of time. Test stage is a powerful technique that can save you a lot of time.

Test suite

When we analyze many tests together, we can do a lot of additional analysis. It’s very important to perform a correlation analysis. For example, if you have a performance degradation after some changes, it’s useful to find the whole scope of tests that have this degradation .

Spatial clustering

When you have metrics for the same test from several agent, you can try to find factors that affect performance. It can be the operating system, the processor model, or any other parameter of your environment.

Temporal anomalies

If you are investigating the performance history of a single test, it can be useful to compare durations of the test runs on different CI agents. If a performance degradation or another anomaly appeared at that moment when the CI agent was changed, the first thing that you should check is the difference between the CI agent environments .

Nontrivial dependencies

Let’s say that we have a test that processes many requests. The requests can be processed in several threads. How does the performance depend on the degree of parallelization? You may get a 2x performance boost when a single-thread implementation is replaced by a two-thread solution. However, switching from four threads to eight may slow down the benchmark because of inefficient and heavy locking. You can find the best parallelization degree only if you check several possible values.

Asymptotic complexity

Let’s say that we have a test that checks whether a given string of length M is contained in a text of length N. The time complexity depends on the underlying algorithm. For example, it can be O(N · M) for a trivial implementation or O(N + M) for a smarter algorithm. You can easily miss some important degradation if the test works only for short search patterns and doesn’t check the larger cases. The knowledge of the complexity allows you to extrapolate results on huge inputs without actually having to test them.

Corner cases

Let’s say that we have a test with the quicksort algorithm . In the best and average case, the complexity is O(N ·  log N), but it becomes O(N²) in the worst case. The knowledge of the worst-case performance also may be very important (especially if we have a risk on a performance attack on the program). The worst possible performance is another valuable metric that we can collect during testing.

Duration range

Let’s say that we have a test that parses text with a regular expression. In this case, the test duration may vary in a huge range depending on the expression complexity and the text. It’s not enough to just check a few input cases to get reliable performance metrics. Good performance coverage for such a test requires hundreds of inputs that correspond to different real-life situations and corner cases. Speaking of corner cases: there are regular expression denial of service (ReDoS) attacks that can significantly slow down your code. One of the most famous .NET Framework 4.5 ReDoS exploits against MVC web applications is described in [Malerisch 2015]: the EmailAddressAttribute, PhoneAttribute, UrlAttribute classes contained regular expressions that can be forced to calculate an exponential number of states on special inputs. The vulnerability was fixed in Microsoft Security Bulletin MS15-101.³² As you can see, the subspaces can be analyzed together: here we have an interesting performance issue that involves the environment and performance subspaces .

History moment (single revision)

If you only have a single revision, you can look for spatial anomalies: there are plenty of them. You can’t find any performance degradations here, but you still can find a lot of problems that can be critical for your production environment.

Linear history (single branch)

If you have several revisions, you can look for spatial anomalies like degradation/acceleration. If you find a problem that is introduced in the latest release, you can bisect the history and find a commit with relevant changes.

Treelike history (selected branches or whole repository)

Sometimes, it makes sense to analyze several branches or even the whole repository. The number of performance measurements are always limited. If you are looking for anomalies like “Huge variance” or “Huge outliers,” you can join performance history of the master branch and all feature branches. Analysis of this “mixed” history can produce a lot of false positive results, but it usually easily finds serious problems that are hard to detect based on a single branch because you don’t have enough measurements .

NUnit, MSTest: both frameworks provide [Timeout] attributes, which all allow you to set timeout in milliseconds.

xUnit: As of xUnit 2.0 (and in subsequent versions like 2.1, 2.2, 2.3), the framework doesn’t support timeouts³⁴ because it’s pretty hard to achieve stable time measurements with parallelization enabled by default in xUnit 2.x. Thus, you have to implement the timeouts manually like in the preceding example. In this case, it’s highly recommended to disable parallelization.

Simple implementation

You can implement it with a few lines of code. In case of NUnit or MSTest, a single [Timeout] attribute is usually enough. In case of xUnit or a complicated check, you need two lines with Stopwatch (Start/Stop) and a single line with assert.

Portability

Not all computers are equally fast. A test can satisfy a 2000ms timeout on your machine in 100% runs, but it can fail on a slow machine of your colleague or in a virtual environment on a CI server.

Flakiness

When a timeout is close to actual test duration, the test can be red sometimes depending on the duration variance and other resource-consuming processes in OS, which can slow down this test.

Maintainability

When I see a test with a hardcoded absolute timeout, I always look at the test history. Typically, it looks like in Table 5-16. You can see that developers change the hardcoded value in the source code all the time. This is not a healthy thing. If such commits are a common practice in your team, it’s always easier to increase the timeout of a red test instead of doing an investigation in case of real performance problems .

Test is hanged because of a deadlock. The timeout helped us to save time on a CI agent.

Test takes 1.5 minutes instead of a few seconds because of a bug. Hooray, performance asserts helped to find a performance degradation.

The variance is huge, a test takes from 1 second to 5 minutes (probably because of the moon phase). Typically, this means a serious bug in the source code; such anomalies should be investigated.

Relative method performance

You can introduce a Baseline and measure the relative performance of all methods to the baseline. When you are marking changes in the source code, you can calculate relative performance against the baseline instead of analyzing the absolute numbers.

Relative machine or environment performance

The baseline approach can also be used for comparing performance between different machines.³⁵ The same trick can be used to compare performance between several runtimes on the same hardware. For example, Mono and .NET Core have different startup time overheads. In theory, the relative threshold is not a correct approach because the performance ratio between different methods can be different for each machine/environment. In practice , this approach usually works for most simple cases.

Handling portability issues

You should understand that this is not the perfect solution, but it usually works pretty well for simple cases.

Flakiness

The same as in the absolute threshold case: sometimes you will get false alarms.

Maintainability

Relative thresholds are still hardcoded; you should manually change it in case of important changes like changes in the test .

No hardcoded values

You shouldn’t keep many magic numbers in the source code anymore. You even shouldn’t think about how fast the code should be. An algorithm will check automatically that you don’t have any performance degradations or other anomalies.

Slow reaction to changes in the test

If you change the logic of the test (for example, add a few heavy asserts), you should retrain your algorithm and wait while the algorithm “learns” the new baseline. Meanwhile, you will get false alerts. Of course, you can introduce a way to mark a test as “changed” or clear the performance history, but it’s usually not as simple as changing a hardcoded threshold.

Smart algorithm is required

You should manually implement an algorithm that compares the performance history and the current state. Unfortunately, no universal algorithm solves this problem in general or works for all projects. There are some ready solutions, but you should check which one works for you. Don’t forget about possible pitfalls like the optional stopping problem (which we discussed in Chapter 4).

Handle even supertricky cases

You can detect very tricky problems that are almost impossible to cover by a smart algorithm. Typically, an experienced developer can instantly say if you have a performance problem or not with a quick glance at the performance plot.

Complete lack of automation

You should manually check most suspicious tests every day.

Codename: “Optimizations”

Task: Optimize ineffective code

Goal: We should achieve “better” performance

It’s not a good idea to blindly optimize different parts of your code. A performance test can help you to verify that you actually optimized something and evaluate the performance boost.

Codename: “Feature”

Task: Implement a new feature

Goal: The feature should be fast

When a feature is already implemented, there is always a temptation to say something like “It seems that it works fast enough.” A proper performance test helps to set your business requirements in advance. This case is pretty similar to a situation in classic TDD.

Codename: “Refactoring”

Task: Refactoring in performance-sensitive code

Goal: We should keep the same level of performance (or make it better)

It’s pretty hard to say that you didn’t introduce any performance degradations if you don’t have a baseline. A baseline helps you to verify that everything is OK.

Step 1: Write target method

Just write a method that covers the target case. Imagine that you are writing a functional test that covers your code. As in the case of ordinary tests, you should try to isolate logic and measure only logic that matters to you. In the “Optimizations” case, you should cover only logic that you are going to optimize and nothing else. In the “Feature” case, you should cover the feature (and only the feature) in advance (as you usually do in typical TDD). In the “Refactoring” case, you should cover only the performance-critical part of the architecture that you are going to refactor. It’s always better to have several performance tests. If you came up only with a single one, try to parametrize it. If you read a file, try files with different sizes. If you process a dataset, try different datasets.

Step 2: Collect metrics

As a minimum, you have to measure raw test duration. However, it’s better to collect some additional metrics like hardware counters, GC collections, and so on. Do many iterations, accumulate them, and calculate statistics numbers. Run tests not only on your developer machine but also on machines of your colleagues and on a server.

Step 3: Look at the performance space

It’s not enough to just collect raw metrics; you should carefully look at them. Check out how the distribution looks. Does it have one mode or several modes? What about the variance? How does the performance depend on the test parameters? Is the dependency linear or not? What’s the maximum parameter value that produces a reasonable duration for the performance test? If you practice PDD on a regular basis, you will come up with your own checklist soon. Looking at the performance space doesn’t require too much time (especially if it’s not your first time), but it can save a lot of time later. Knowledge about some “features” of the test performance space will help you to find tricky places in your source code that you should be aware of.

Step 4: Write performance asserts

Now it’s time to transform your business goals into performance asserts. Remember that the test should be red for “Optimizations” cases. Many developers skip this step. You may be tempted to say: “OK, I know how much time it takes now. I can optimize my code and check how much it takes after that. Next, I will write performance asserts.” This is a bad practice: it can destroy your business goal. If you want to optimize a method twice, write a corresponding assert. If you discover new things during optimizations (like “Hey, I can optimize it ten times!” or “It’s just impossible to optimize more than 50%”), you always can change the assert later. But you still have to express your original intention in the form of performance asserts. I have seen many times when developers say something like “After these crazy hacks I get 5% speedup, now I’m going to commit it” (whereas 5% speedup doesn’t have business value and crazy hacks mutilate the code and move it to the “impossible to maintain” state). Original performance asserts don’t protect you from all such cases, but they will make you think twice before committing code that doesn’t solve the original problem.

Step 5: Play with the test status

Next, you should check that you wrote good performance asserts. In the “Optimizations” case, try to transform the red test to a green one by commenting the “heaviest” part of your code. In the “Refactoring” case, try to add a few Thread.Sleep calls here and there and make sure that the test is red now. In the “Feature” case, check empty and Thread.Sleep implementations. You should be sure that you wrote performance asserts correctly (at the end, tests should be green in case of success or red in case of failure).

All tests should be green

If you constantly have some red or flaky tests, nobody will care about “one more test” with some performance problems.

Type I (false positive) error rate should be low

If you get false alarms about performance problems all the time, you will probably start to ignore them because you will spend your time on the investigation without any benefits from it.

It should be easy to write a performance test

Writing performance tests is usually an optional task. If such tests require complicated routine work, developers will be tempted to skip it.

Zero tolerance for performance problems

If you have a new performance problem, it should be investigated on the spot. Try to forget about backlog lists and thoughts like “I’m too busy right now, will take a look at the next week.” It will be much harder to investigate the issue a week later: other problems can be introduced and “the rest of the windows will soon be broken.” Of course, it’s ideal when you instantly fix any performance problems. In many cases, though, this can be impossible because you have many other higher-priority issues that can’t be postponed. But, in terms of zero tolerance for performance problems, it doesn’t matter that you can’t always achieve this ideal situation.³⁷

Regular checking of the performance anomaly list

I should say it again: it’s pretty hard to catch all problems automatically. New problems that are not covered by performance tests with strict asserts can be introduced at any moment. Thus, it’s a very good practice to have some performance alarms and dashboards and to check them regularly.