Use Selenium and AskUI Together

April 10, 2024

When you automate browser applications there is no way around an automation tool as Selenium. It is optimized for browser interaction and executes blazingly fast.

However sometimes you need to leave the confines of your browser to interact with another application or your operating system. This is simply not possible with Selenium alone.

AskUI shines in this regard as it is not limited to a browser with its visual selectors and real input commands like keypresses and mouse movement.

In this post you will learn to setup Selenium together with AskUI to define workflows that can leave the browser.

Prerequisites

AskUI installed and configured on your system (Windows, Linux, macOS)

Install Selenium

The first thing you have to do is to install the selenium-webdriver. And since we are using TypeScript, let's install the type definitions also:

-- CODE language-bash line-numbers -- npm install --save-dev selenium-webdriver @types/selenium-webdriver

Configure Selenium

With Selenium ready to go, you have to head over to the file helpers/askui-helpers.ts, configure the Selenium Webdriver there. Here is how the the content of the file should look like:

-- CODE language-ts line-numbers -- import { UiControlClient, UiController } from 'askui'; import { AskUIAllureStepReporter } from '@askui/askui-reporters'; // Server for controlling the operating system let uiController: UiController; // Client is necessary to use the askui API let aui: UiControlClient; // 1. Selenium setup import { Builder, Browser, WebDriver } from 'selenium-webdriver'; let seleniumDriver: WebDriver; jest.setTimeout(60 * 1000 * 60); beforeAll(async () => { // 2. Selenium setup seleniumDriver = await new Builder().forBrowser(Browser.CHROME).build(); uiController = new UiController({ /** * Select the display you want to run your tests on, display 0 is your main display; * ignore if you have only one display */ display: 0, }); await uiController.start(); aui = await UiControlClient.build({ credentials: { workspaceId: '', token: '', }, reporter: new AskUIAllureStepReporter(), }); await aui.connect(); }); ... afterAll(async () => { // 3. Selenium setup await seleniumDriver.quit(); aui.disconnect(); await uiController.stop(); }); export { aui }; // 4. Selenium setup export { seleniumDriver };

Import the necessary classes to initialize a WebDriver for the Chrome browser
Initialize the WebDriver for Chrome browser
Make sure the browser gets closed after execution
Export the WebDriver for use in a AskUI Workflow

Write a Sample AskUI Workflow

With the WebDriver set up, hop over to the file my-first-askui-test-seuite.test.ts and import it there alongside a few more classes you need to write Selenium code. Add this to the start of the file:

-- CODE language-ts line-numbers -- import { seleniumDriver } from './helpers/askui-helper'; import { By, Key, until } from 'selenium-webdriver';

Now you can implement a very simple workflow:

Maximize the browser window
Open the website https://google.com
Dismiss the cookie popup by clicking the Reject all button
Search for webdriver and wait for the search results to appear
Then with AskUI: Click on the button with text Images to get to the image search
Then refine your search by clicking on architecture
Move your mouse to the image where a Chrome and a Firefox logo in it

-- CODE language-ts line-numbers -- describe('jest with askui', () => { it('should open askui.com in Chrome with selenium', async () => { // Selenium await seleniumDriver.manage().window().maximize(); await seleniumDriver.get('https://www.google.com'); await seleniumDriver.findElement(By.xpath('//button/div[text()="Reject all"]')).click(); await seleniumDriver.findElement(By.name('q')).sendKeys('webdriver', Key.RETURN); await seleniumDriver.wait(until.titleIs('webdriver - Google Search'), 1000); await seleniumDriver.sleep(1000); // AskUI await aui.mouseLeftClick().exec(); await aui.click().button().withText('Images').exec(); await aui.click().text('architecture').exec(); await aui.moveMouseTo().element().matching('image with a firefox and chrome logo in it').exec() await seleniumDriver.sleep(5000); }); });

Testrun

-- CODE language-bash line-numbers -- // Linux + macOS npm run askui // Windows AskUI-RunProject

The following gif shows the execution of the whole workflow:

Gif showing the Selenium and AskUI workflow in action (doubled speed).

Conclusion

Integrating AskUI with a framework like Selenium comes down to configuring Selenium Webdriver in Jest. If you are solely automating in a web browser Selenium is a solid choice. Once you also want to manipulate a desktop app or interact with your operating system AskUI can take over.

Combining Selenium with AskUI gives you all the options to automate workflows over multiple applications.

‍

Johannes Dienst

April 10, 2024

Use Selenium and AskUI Together

Prerequisites

Install Selenium

Configure Selenium

Write a Sample AskUI Workflow

Testrun

Conclusion

More to explore

Automating the User Experience: A Look at End-to-End and UI Testing

Automating Software Quality: A Look at Test Automation

Welcome: Automation Testing with Machine Learning