Use Playwright and AskUI Together

April 22, 2024
Left side AskUI logo with subtitle AskUI. Right side Playwright logo with title Playwright on the top. Background is left side teal colored right side is rose colored.
linkedin icontwitter iconfacebook iconmail icon

Playwright is the de facto standard when it comes to reliably testing browser applications across browsers and operating systems.

But what if your workflow under test leaves your browser and requires you to automate on a desktop application?

AskUI is not confined to a specific application because it uses visual selectors and real human-user-like input with keypresses and natural mouse movement. This makes it an ideal tool to accompany Playwright.

In this post you will learn to setup Playwright together with AskUI to define workflows that can leave the browser.


Install Playwright

Playwright for JavaScript/TypeScript consists of two components.

1. The playwright node package

2. A browser installed through Playwright library

First you have to install playwright as a dev-dependency. Execute the following command in your terminal:

-- CODE language-bash line-numbers -- npm install --save-dev playwright

For this blog you will use Chromium as a browser. The installation through `playwright` is done like this:

-- CODE language-bash line-numbers -- npx playwright install --with-deps chromium

Configure Playwright

You will configure Playwright and the Chromium browser in the file helpers/askui-helper.ts, so it will work correctly together with Jest.

-- CODE language-ts line-numbers -- import { UiControlClient, UiController } from 'askui'; import { AskUIAllureStepReporter } from '@askui/askui-reporters'; // Server for controlling the operating system let uiController: UiController; // Client is necessary to use the askui API let aui: UiControlClient; // 1. Playwright setup import { chromium, devices, Browser, BrowserContext } from 'playwright'; let pwBrowser: Browser; let pwContext: BrowserContext; jest.setTimeout(60 * 1000 * 60); beforeAll(async () => { // 2. Playwright setup pwBrowser = await chromium.launch({ headless: false, slowMo: 100 }); pwContext = await pwBrowser.newContext(devices['Desktop Chrome']); uiController = new UiController({ /** * Select the display you want to run your tests on, display 0 is your main display; * ignore if you have only one display */ display: 0, }); await uiController.start(); aui = await{ credentials: { workspaceId: '', token: '', }, reporter: new AskUIAllureStepReporter(), }); await aui.connect(); }); ... afterAll(async () => { // 3. Playwright setup await pwContext.close(); await pwBrowser.close(); aui.disconnect(); await uiController.stop(); }); export { aui }; // 4. Playwright setup export { pwContext };
  1. Import the necessary classes from playwright and define the variables
  2. Configure chromium to be not headless and also slow down its execution. Also we want a Desktop Chrome as we are on desktop.
  3. Close the BrowserContext and the Browser after the execution finished.
  4. Export the Context so you can use it in your workflow

Write a Workflow

With the BrowserContext set up, hop over to the file my-first-askui-test-seuite.test.ts and import it there. Add this to the start of the file:

-- CODE language-ts line-numbers -- import { pwContext } from './helpers/askui-helper';

Now you can implement a simple workflow:

  1. Open a new tab
  2. Navigate to
  3. Click away the cookie banner
  4. Fill the search field with Playwright framework
  5. Press Enter on keyboard
  6. Switch to image search by clicking on the Button architecture
  7. Move the mouse cursor to the element that has an image with a red and a green mask in it (Playwright logo description)
-- CODE language-ts line-numbers -- describe('jest with askui', () => { it('should open in Chrome with playwright', async () => { let page = await pwContext.newPage(); await page.goto(''); await page.getByRole('button', { name: 'Reject all' }).click(); await page.locator('//textarea[@name="q"]').fill('Playwright framework'); await'Enter'); await page.waitForTimeout(1000); // AskUI workflow // Make sure you have the focus on the browser await aui.mouseLeftClick().exec(); await'Images').exec(); await'architecture').exec(); await aui.moveMouseTo().element().matching('image with a red and a green mask in it').exec() await page.waitForTimeout(5000); }); });


To run this you need to run this command in your terminal:

-- CODE language-bash line-numbers -- // Linux + macOS npm run askui // Windows AskUI-RunProject

The following gif shows the execution of the whole workflow:

Playwright and AskUI workflow in action (2 times speedup).


Combining AskUI with Playwright enhances automation capabilities to extend beyond web browsers. While Playwright is a solid choice for browser automation, AskUI takes the reins when dealing with desktop apps and operating systems.

This integration offers a practical toolkit for workflows across various applications for efficient automation.

Johannes Dienst
April 22, 2024
On this page