Use Selenium and AskUI Together

When you automate browser applications there is no way around an automation tool as Selenium. It is optimized for browser interaction and executes blazingly fast.

However sometimes you need to leave the confines of your browser to interact with another application or your operating system. This is simply not possible with Selenium alone.

AskUI shines in this regard as it is not limited to a browser with its visual selectors and real input commands like keypresses and mouse movement.

In this post you will learn to setup Selenium together with AskUI to define workflows that can leave the browser.

Prerequisites

AskUI installed and configured on your system (Windows, Linux, macOS)

Install Selenium

The first thing you have to do is to install the selenium-webdriver. And since we are using TypeScript, let's install the type definitions also:

-- CODE language-bash line-numbers -- npm install --save-dev selenium-webdriver @types/selenium-webdriver

Configure Selenium

With Selenium ready to go, you have to head over to the file helpers/askui-helpers.ts, configure the Selenium Webdriver there. Here is how the the content of the file should look like:

-- CODE language-ts line-numbers -- import { UiControlClient, UiController } from 'askui'; import { AskUIAllureStepReporter } from '@askui/askui-reporters'; // Server for controlling the operating system let uiController: UiController; // Client is necessary to use the askui API let aui: UiControlClient; // 1. Selenium setup import { Builder, Browser, WebDriver } from 'selenium-webdriver'; let seleniumDriver: WebDriver; jest.setTimeout(60 * 1000 * 60); beforeAll(async () => { // 2. Selenium setup seleniumDriver = await new Builder().forBrowser(Browser.CHROME).build(); uiController = new UiController({ /** * Select the display you want to run your tests on, display 0 is your main display; * ignore if you have only one display */ display: 0, }); await uiController.start(); aui = await UiControlClient.build({ credentials: { workspaceId: '', token: '', }, reporter: new AskUIAllureStepReporter(), }); await aui.connect(); }); ... afterAll(async () => { // 3. Selenium setup await seleniumDriver.quit(); aui.disconnect(); await uiController.stop(); }); export { aui }; // 4. Selenium setup export { seleniumDriver };

Import the necessary classes to initialize a WebDriver for the Chrome browser
Initialize the WebDriver for Chrome browser
Make sure the browser gets closed after execution
Export the WebDriver for use in a AskUI Workflow

Write a Sample AskUI Workflow

With the WebDriver set up, hop over to the file my-first-askui-test-seuite.test.ts and import it there alongside a few more classes you need to write Selenium code. Add this to the start of the file:

-- CODE language-ts line-numbers -- import { seleniumDriver } from './helpers/askui-helper'; import { By, Key, until } from 'selenium-webdriver';

Now you can implement a very simple workflow:

Maximize the browser window
Open the website https://google.com
Dismiss the cookie popup by clicking the Reject all button
Search for webdriver and wait for the search results to appear
Then with AskUI: Click on the button with text Images to get to the image search
Then refine your search by clicking on architecture
Move your mouse to the image where a Chrome and a Firefox logo in it

-- CODE language-ts line-numbers -- describe('jest with askui', () => { it('should open askui.com in Chrome with selenium', async () => { // Selenium await seleniumDriver.manage().window().maximize(); await seleniumDriver.get('https://www.google.com'); await seleniumDriver.findElement(By.xpath('//button/div[text()="Reject all"]')).click(); await seleniumDriver.findElement(By.name('q')).sendKeys('webdriver', Key.RETURN); await seleniumDriver.wait(until.titleIs('webdriver - Google Search'), 1000); await seleniumDriver.sleep(1000); // AskUI await aui.mouseLeftClick().exec(); await aui.click().button().withText('Images').exec(); await aui.click().text('architecture').exec(); await aui.moveMouseTo().element().matching('image with a firefox and chrome logo in it').exec() await seleniumDriver.sleep(5000); }); });

Testrun

-- CODE language-bash line-numbers -- // Linux + macOS npm run askui // Windows AskUI-RunProject

The following gif shows the execution of the whole workflow:

Gif showing the Selenium and AskUI workflow in action (doubled speed).

Conclusion

Integrating AskUI with a framework like Selenium comes down to configuring Selenium Webdriver in Jest. If you are solely automating in a web browser Selenium is a solid choice. Once you also want to manipulate a desktop app or interact with your operating system AskUI can take over.

Combining Selenium with AskUI gives you all the options to automate workflows over multiple applications.

More to explore

academy

Use AskUI and Cucumber Together

By defining the behavior of a system in a structured format like Gherkin, Behavior-Driven Development (BDD) enables teams to bridge the gap between stakeholders, testers, and developers, avoiding misunderstandings and reducing rework. In this blog post, we'll show you how to set up Cucumber in conjunction with AskUI to define AskUI workflows using BDD principles.

academy

Better AskUI Workflows with Convenience Methods in AskUI SDK (ADK)

AskUI interacts with your operating system like a real human user. This directly leads to code that specifies WHAT you have to do in a specific sequence to achieve a workflow. Unfortunately this also means a lot of repetitive code. This is why we implemented some convenience methods to help you ease these pain points.

academy

Use Playwright and AskUI Together

Playwright is the de facto standard when it comes to reliably testing browser applications across browsers and operating systems. But what if your workflow under test leaves your browser and requires you to automate on a desktop application? AskUI is not confined to a specific application because it uses visual selectors and real human-user-like input with keypresses and natural mouse movement. This makes it an ideal tool to accompany Playwright.

Get in touch

For media queries, drop us a message at info@askui.com

Download Press Kit