askui Automation Example with Flutter

This tutorial shows how to use askui to automate an Android app built with Flutter. We provide the source code for the Flutter demo app used in this tutorial. You can find the source code on our GitHub repository. Set up the demo app by following the instructions below. This tutorial assumes that you already have your Android device prepared. It can be a real Android device or an Android Emulator.

If you haven't set up your Android device or Emulator yet, follow this post.

This tutorial includes:

• Build and Run the Flutter Demo App
• Set up the ADBKeyboard
• Click/Touch Automation
• Type Automation
• Swipe Automation

Live Demo in Action (playback speed x3)

‍

‍

Setup

The source code for the Flutter demo app used in this tutorial is provided in this repository.

‍

1. Build and Run Flutter Demo App

1) Install Flutter

2) Clone this repository and run flutter create demo_app within the directory:

3) Install dependencies for the Flutter demo app:

4) To use the camera, we need to set the minSdkVersion in android/app/build.gradle:

5) (optional) The app is ready to be built but will throw deprecation warnings. If you want to clear the deprecation warnings, follow this step. See this issue

6) Run the Android Emulator.

7) Run the demo app:

Now you should see the demo app running on your Android device.

2. Setup ADBKeyboard

In this example, we are going to automate the typing on the Android device. To let askui fluently type as desired, we will use a virtual keyboard that handles the keyboard input via adb: ADBKeyboard.apk

‍

1) Download the ADB-Keyboard package (Important: Version 2.0): Link to GitHub Repo

2) Unzip it.

3) Find your device:

4) Install the ADBkeyboard on the device:

5) Configure the ADB Keyboard:

6) Enable the ADB Keyboard:

7) To check if it is enabled:

    Click on a textfield in an app and see if the ADB Keyboard {ON} notification is shown at the bottom of the screen.

3. Setup askui

1) Setup askui by following the Getting Started Guide.

2) We need to run the UiController directly with an extra argument to specify the runtime mode, as the current version of askui(ver. 0.6) doesn't provide the API for running it with the runtime argument yet.
From within your npm project path, go to the directory that contains the askui-ui-controller binary:

If you got them both(emulator and UiController) running, then we are ready to go for the UI automation.

‍

3) If you are working with the test code from our official docs, then you need to deactivate a few lines of the code in test/helper/jest.setup.ts that is running the UiController, because we are already running it manually in the previous step:

Breaking Down the Askui Test Code

This chapter will walk you through the provided askui-test/demo-automation.ts step by step.
The test is divided into three parts, and each test is run for each tab within the demo app:

• Outline tab: Find a textfield and type in characters.

• Datepicker tab: Select a desired date within the date picker widget.

• Camera tab: Open the camera and push the record button.

0. General tips for using askui as a more friendly tool:

1) Try to annotate : Use await aui.annotateInteractively() or await aui.annotate() in order to see how askui is understanding the visible elements on your screen. By using await aui.annotate(), the result of the annotation will be saved in report/ as an HTML file.

2) Be aware of the screen size of your device : askui understands your application based on the screen shown and captured. Therefore, on some occasions, you may want to know your screen size to e.g. properly scroll or swipe within your application. You may need to change the numbers for the input swipe command within the provided test code so that it suits the screen size of your device.

3) Try to select the elements by their text

• tip: If you are using a device with a bigger screen e.g. Tablet, then the screen of your test device (real Android device or emulator) might be big enough to see the whole page without scrolling.

1. Click and Type

The test code is within the askui-test/demo-automation.ts. Copy and paste the code into your askui test code.

• We start the test automation from the very first tab of our demo app.

‍

‍

‍

• To type into a textfield, we first need to get focus on the desired textfield. We can achieve it by running the code below:

• As we have multiple of textfields in our demo app, we can iterate the same procedure for each of them:

• After filling up the textfields, we can push the buttons at the bottom of the page:

2. Datepicker

• After running the test code above, we should see the demo app swiped to the Datepicker tab.

‍

‍

• First, we select and type characters into two different textfields:
  

• Thereafter, we interact with two different date picker widgets that are represented with edit buttons:

• Let's go further below to the bottom of the page, and then interact with more interfaces:

3. Take a Picture with the Camera

• In the final tab Camera, we can launch the device's camera and take a picture by pressing the record button.

‍

‍

4. Complete Test Code

This is the complete test code that runs askui to automate our test:

5. Conclusion

After following through this example, you should be able to automate the interaction with the provided demo app. Although this example specifically provides a demo app built with Flutter, the overall method of using askui should also work with any mobile app running on an Android device.

If you got an issue while following this example, or in case you would like to share your test case, don't hesitate to join our community on Discord!

‍