Use Selenium and AskUI Together

April 10, 2024
Left side AskUI logo with subtitle AskUI. Right side Selenium logo with title Selenium on the top. Background is left side teal colored right side is rose colored.
linkedin icontwitter iconfacebook iconmail icon

When you automate browser applications there is no way around an automation tool as Selenium. It is optimized for browser interaction and executes blazingly fast.

However sometimes you need to leave the confines of your browser to interact with another application or your operating system. This is simply not possible with Selenium alone.

AskUI shines in this regard as it is not limited to a browser with its visual selectors and real input commands like keypresses and mouse movement.

In this post you will learn to setup Selenium together with AskUI to define workflows that can leave the browser.


Install Selenium

The first thing you have to do is to install the selenium-webdriver. And since we are using TypeScript, let's install the type definitions also:

-- CODE language-bash line-numbers -- npm install --save-dev selenium-webdriver @types/selenium-webdriver

Configure Selenium

With Selenium ready to go, you have to head over to the file helpers/askui-helpers.ts, configure the Selenium Webdriver there. Here is how the the content of the file should look like:

-- CODE language-ts line-numbers -- import { UiControlClient, UiController } from 'askui'; import { AskUIAllureStepReporter } from '@askui/askui-reporters'; // Server for controlling the operating system let uiController: UiController; // Client is necessary to use the askui API let aui: UiControlClient; // 1. Selenium setup import { Builder, Browser, WebDriver } from 'selenium-webdriver'; let seleniumDriver: WebDriver; jest.setTimeout(60 * 1000 * 60); beforeAll(async () => { // 2. Selenium setup seleniumDriver = await new Builder().forBrowser(Browser.CHROME).build(); uiController = new UiController({ /** * Select the display you want to run your tests on, display 0 is your main display; * ignore if you have only one display */ display: 0, }); await uiController.start(); aui = await{ credentials: { workspaceId: '', token: '', }, reporter: new AskUIAllureStepReporter(), }); await aui.connect(); }); ... afterAll(async () => { // 3. Selenium setup await seleniumDriver.quit(); aui.disconnect(); await uiController.stop(); }); export { aui }; // 4. Selenium setup export { seleniumDriver };
  1. Import the necessary classes to initialize a WebDriver for the Chrome browser
  2. Initialize the WebDriver for Chrome browser
  3. Make sure the browser gets closed after execution
  4. Export the WebDriver for use in a AskUI Workflow

Write a Sample AskUI Workflow

With the WebDriver set up, hop over to the file my-first-askui-test-seuite.test.ts and import it there alongside a few more classes you need to write Selenium code. Add this to the start of the file:

-- CODE language-ts line-numbers -- import { seleniumDriver } from './helpers/askui-helper'; import { By, Key, until } from 'selenium-webdriver';

Now you can implement a very simple workflow:

  1. Maximize the browser window
  2. Open the website
  3. Dismiss the cookie popup by clicking the Reject all button
  4. Search for webdriver and wait for the search results to appear
  5. Then with AskUI: Click on the button with text Images to get to the image search
  6. Then refine your search by clicking on architecture
  7. Move your mouse to the image where a Chrome and a Firefox logo in it
-- CODE language-ts line-numbers -- describe('jest with askui', () => { it('should open in Chrome with selenium', async () => { // Selenium await seleniumDriver.manage().window().maximize(); await seleniumDriver.get(''); await seleniumDriver.findElement(By.xpath('//button/div[text()="Reject all"]')).click(); await seleniumDriver.findElement('q')).sendKeys('webdriver', Key.RETURN); await seleniumDriver.wait(until.titleIs('webdriver - Google Search'), 1000); await seleniumDriver.sleep(1000); // AskUI await aui.mouseLeftClick().exec(); await'Images').exec(); await'architecture').exec(); await aui.moveMouseTo().element().matching('image with a firefox and chrome logo in it').exec() await seleniumDriver.sleep(5000); }); });


-- CODE language-bash line-numbers -- // Linux + macOS npm run askui // Windows AskUI-RunProject

The following gif shows the execution of the whole workflow:

Gif showing the Selenium and AskUI workflow in action (doubled speed).


Integrating AskUI with a framework like Selenium comes down to configuring Selenium Webdriver in Jest. If you are solely automating in a web browser Selenium is a solid choice. Once you also want to manipulate a desktop app or interact with your operating system AskUI can take over.

Combining Selenium with AskUI gives you all the options to automate workflows over multiple applications.

Johannes Dienst
April 10, 2024
On this page