Our #1 Solution to Playwright Flakiness - WaitForResponse, WaitForRequest, Promises

‍

We are huge advocates of Playwright here at Loop. However, as with any test automation tool, it is not immune to flakiness. In our experience, the number one cause of flakiness when running Playwright automation is the UI changing faster than the API can keep up with during a test run. This phenomenon, known as asynchronous testing, can lead to problems such as UI elements being clicked before the API is ready to show the relevant data, resulting in inconsistent or failed test results.

‍

One strategy for addressing asynchronous testing in the context of Playwright automation is to use a promise with a WaitForResposne. A promise is an object that represents the eventual completion or failure of an asynchronous operation. By combing these two methods, you can synchronize the execution of your test steps with the availability of data from the API, ensuring that the UI is not interacting with data that is not yet ready or fully loaded.

‍

Despite the challenges of asynchronous testing, Playwright is generally a very stable and robust test automation tool. With careful planning and the use of synchronization techniques such as promises, it is possible to mitigate the risk of flakiness and ensure reliable and consistent test results. In this blog post, we will explore the problem of asynchronous testing in more detail and discuss the use of promises as a strategy for addressing it in the context of Playwright automation.

‍

‍

Why do APIs Matter?

Interfaces) are critical in the world of web applications. APIs serve as a bridge between the front-end user interface (UI) and the back-end data and services that power the application. When a user interacts with a web app, the UI sends requests for data to the back-end via the API. The API then retrieves the requested data from the back-end and returns it to the UI, which displays it to the user.

‍

APIs are critical to the functionality of web apps because they provide a way for the UI to access the data and services it needs to operate. Without APIs, the UI would be unable to communicate with the back-end and the web app would not be able to function properly.

‍

There are countless other reasons why APIs are critical in the world of web applications, but that is a topic for a different article.

‍

Diagnosing your API is Getting Outpaced

If you suspect that your API is being outpaced by the UI in your Playwright test automation, there are a few ways you can diagnose the problem. The first is to open the Playwright debugger and manually click through the test steps slowly. This will allow you to see if the test is able to complete successfully when given more time. If the test passes when you go through it manually, it is highly likely that the issue was related to performance and that the API was simply unable to keep up with the UI.

‍

Another option is to run the test in slow motion. This can be done by launching the browser with a slowMo option set to a value in milliseconds. For example:

‍

const browser = await chromium.launch({slowMo: 50});

‍

This will slow down all of the actions performed by Playwright, giving the API more time to catch up with the UI and potentially resolving issues related to asynchronous testing. For more detail on slowMo, see our article here.

‍

Using Chrome Developer Tools “Network” to Watch API Calls

‍If you have identified that your Playwright tests are suffering from flakiness due to the UI changing faster than the API can keep up, you can use Chrome DevTools to help diagnose the problem and identify potential solutions.

‍

To begin, open Chrome DevTools and go to the Network tab. Then open your test in debug mode and be ready to manually click through the test steps. As you click through the test, you should see a series of API calls being made in the Network tab. Pay attention to the timing of these calls in relation to the actions you are taking in the UI.

‍

You should see a pattern of UI actions being followed by API calls, as the UI requests data from the back-end. Then, you should see the UI update to show the data returned by the API, and the user takes another action. This pattern should repeat as the test progresses.

‍

To help narrow down the problem, you can use the filter option in the Network tab to filter out tracking APIs and focus only on your own APIs or the APIs serving up the data for your test. This will make it easier to see what is happening behind the scenes as your test runs.

‍

It is important to start to identify which API responses are critical for future UI actions in your Playwright tests. For example, if the UI clicks the "View Company X" button, and then clicks another button, "View More Details," it is likely that there was only one API GET call triggered by the first click. This call would have retrieved the data needed to populate the "View Company X" view, and it would be critical for subsequent UI actions such as the "View More Details" button click.

‍

WaitForRequest - WaitForResponse - Promises. 

Now that we have identified the API calls that are critical for the UI actions in our Playwright tests, we can use a variety of techniques to ensure that they are properly synchronized with the UI and avoid flaky test results. One option is to use the waitForRequest and waitForResponse methods in Playwright, which allow you to pause the execution of your test script until a specific network request or response is received. This can help to ensure that the UI is not interacting with data that is not yet ready or fully loaded.

‍

Another tool is to use promises to avoid race conditions and ensure that your test steps are executed in the correct order. A promise is an object that represents the eventual completion or failure of an asynchronous operation. By using promises, you can synchronize the execution of your test steps with the availability of data from the API.

‍

This will ensure that the script waits for the API response to be received before clicking the button, reducing the risk of flaky test results.

‍

It is important to note that if you use waitForResponse without a promise, as in the following example:

‍

Example Code

‍

There is a risk that the API may have already triggered and received the response by the time Playwright starts waiting for it. This can occur if the API is particularly fast, or if there is a delay in the execution of the test script due to other factors. If this happens, the waitForResponse will time out and the test will fail.

‍

To mitigate this risk, you should always use promises to ensure that the waitForResponse is executed concurrently with the click action, as shown in the following example:

‍

Example Code

‍

This allows Playwright to send the API request and start waiting for the response at the same time, reducing the risk of the waitForResponse timing out. By using promises in this way, you can improve the reliability of your Playwright tests and reduce the risk of flaky test results.

‍

‍

Do not use Network Idle

You may be tempted to use waitForLoadState with the 'network idle' state to ensure that all network activity has ceased before continuing with your test.

‍

However, it is important to note that waitForLoadState can sometimes produce inconsistent results, particularly when used with the 'network idle' state. This is because waitForLoadState('network idle') is intended to pause the script until the network activity for the page or frame has reached a quiescent state, meaning that there are no more network requests being made or responses being received.

‍

Unfortunately, predicting when the network activity of a page or frame will reach a quiescent state can be difficult, as it may vary widely depending on the complexity and behavior of the application. If the application makes frequent or long-running network requests, or if there are delays or errors in the network communication, waitForLoadState('network idle') may not accurately determine when the network activity has reached a quiescent state, leading to inconsistent test results.

‍

In these cases, it may be more reliable to use waitForRequest and waitForResponse to synchronize your test steps with specific network requests and responses. This can allow you to be more surgical in your synchronization, as you can pause the script until the exact requests and responses you are interested in have been made or received.

‍

Conclusion

In conclusion, asynchronous testing is a common problem when running test automation with Playwright. It occurs when the UI changes faster than the API can keep up, leading to inconsistent or failed test results. To address this problem, it is important to synchronize your test steps with the availability of data from the API. One way to do this is to use promises, which allow you to synchronize the execution of your test steps with the completion of an asynchronous operation.

‍

By using promises, you can ensure that the UI is not interacting with data that is not yet ready or fully loaded. While asynchronous testing can be challenging, with careful planning and the use of synchronization techniques such as promises, it is possible to mitigate the risk of flakiness and ensure reliable and consistent test results with Playwright.

‍