Use Playwright and AskUI Together

Playwright is the de facto standard when it comes to reliably testing browser applications across browsers and operating systems.

But what if your workflow under test leaves your browser and requires you to automate on a desktop application?

AskUI is not confined to a specific application because it uses visual selectors and real human-user-like input with keypresses and natural mouse movement. This makes it an ideal tool to accompany Playwright.

In this post you will learn to setup Playwright together with AskUI to define workflows that can leave the browser.

Prerequisites

AskUI installed and configured on your system (Windows, Linux, macOS)

Install Playwright

Playwright for JavaScript/TypeScript consists of two components.

‍

1. The playwright node package

2. A browser installed through Playwright library

First you have to install playwright as a dev-dependency. Execute the following command in your terminal:

-- CODE language-bash line-numbers -- npm install --save-dev playwright

For this blog you will use Chromium as a browser. The installation through `playwright` is done like this:

-- CODE language-bash line-numbers -- npx playwright install --with-deps chromium

Configure Playwright

You will configure Playwright and the Chromium browser in the file helpers/askui-helper.ts, so it will work correctly together with Jest.

-- CODE language-ts line-numbers -- import { UiControlClient, UiController } from 'askui'; import { AskUIAllureStepReporter } from '@askui/askui-reporters'; // Server for controlling the operating system let uiController: UiController; // Client is necessary to use the askui API let aui: UiControlClient; // 1. Playwright setup import { chromium, devices, Browser, BrowserContext } from 'playwright'; let pwBrowser: Browser; let pwContext: BrowserContext; jest.setTimeout(60 * 1000 * 60); beforeAll(async () => { // 2. Playwright setup pwBrowser = await chromium.launch({ headless: false, slowMo: 100 }); pwContext = await pwBrowser.newContext(devices['Desktop Chrome']); uiController = new UiController({ /** * Select the display you want to run your tests on, display 0 is your main display; * ignore if you have only one display */ display: 0, }); await uiController.start(); aui = await UiControlClient.build({ credentials: { workspaceId: '', token: '', }, reporter: new AskUIAllureStepReporter(), }); await aui.connect(); }); ... afterAll(async () => { // 3. Playwright setup await pwContext.close(); await pwBrowser.close(); aui.disconnect(); await uiController.stop(); }); export { aui }; // 4. Playwright setup export { pwContext };

Import the necessary classes from playwright and define the variables
Configure chromium to be not headless and also slow down its execution. Also we want a Desktop Chrome as we are on desktop.
Close the BrowserContext and the Browser after the execution finished.
Export the Context so you can use it in your workflow

Write a Workflow

With the BrowserContext set up, hop over to the file my-first-askui-test-seuite.test.ts and import it there. Add this to the start of the file:

-- CODE language-ts line-numbers -- import { pwContext } from './helpers/askui-helper';

Now you can implement a simple workflow:

Open a new tab
Navigate to https://google.com/
Click away the cookie banner
Fill the search field with Playwright framework
Press Enter on keyboard
Switch to image search by clicking on the Button architecture
Move the mouse cursor to the element that has an image with a red and a green mask in it (Playwright logo description)

-- CODE language-ts line-numbers -- describe('jest with askui', () => { it('should open google.com in Chrome with playwright', async () => { let page = await pwContext.newPage(); await page.goto('https://google.com/'); await page.getByRole('button', { name: 'Reject all' }).click(); await page.locator('//textarea[@name="q"]').fill('Playwright framework'); await page.keyboard.press('Enter'); await page.waitForTimeout(1000); // AskUI workflow // Make sure you have the focus on the browser await aui.mouseLeftClick().exec(); await aui.click().button().withText('Images').exec(); await aui.click().text('architecture').exec(); await aui.moveMouseTo().element().matching('image with a red and a green mask in it').exec() await page.waitForTimeout(5000); }); });

Testrun

To run this you need to run this command in your terminal:

-- CODE language-bash line-numbers -- // Linux + macOS npm run askui // Windows AskUI-RunProject

The following gif shows the execution of the whole workflow:

Playwright and AskUI workflow in action (2 times speedup).

Conclusion

Combining AskUI with Playwright enhances automation capabilities to extend beyond web browsers. While Playwright is a solid choice for browser automation, AskUI takes the reins when dealing with desktop apps and operating systems.

This integration offers a practical toolkit for workflows across various applications for efficient automation.

More to explore

academy

Use AskUI and Cucumber Together

By defining the behavior of a system in a structured format like Gherkin, Behavior-Driven Development (BDD) enables teams to bridge the gap between stakeholders, testers, and developers, avoiding misunderstandings and reducing rework. In this blog post, we'll show you how to set up Cucumber in conjunction with AskUI to define AskUI workflows using BDD principles.

academy

Better AskUI Workflows with Convenience Methods in AskUI SDK (ADK)

AskUI interacts with your operating system like a real human user. This directly leads to code that specifies WHAT you have to do in a specific sequence to achieve a workflow. Unfortunately this also means a lot of repetitive code. This is why we implemented some convenience methods to help you ease these pain points.

academy

Use Selenium and AskUI Together

When you automate browser applications there is no way around an automation tool as Selenium. It is optimized for browser interaction and executes blazingly fast. However sometimes you need to leave the confines of your browser to interact with another application or your operating system. This is not possible with Selenium alone.AskUI shines in this regard as it is not limited to a browser with its visual selectors and real input commands like keypresses and mouse movement.

Get in touch

For media queries, drop us a message at info@askui.com

Download Press Kit