Five ways to automate
a test suite.
A fair comparison. And which one fits when the screen has no DOM.
Side by side
Five approaches. Ten questions.
Architectures, not adjectives. Where an alternative is strong, we say so.
| Question | Scripted UI frameworksPlaywright, Selenium, Appium | Record/replay desktop toolsRanorex, TestComplete, Tosca | DIY computer useA frontier model + your own harness | Manual or outsourced QAIn-house testers, crowd, vendors | AskUIRuns your test cases as written |
|---|---|---|---|---|---|
| Works without a DOM or accessibility tree | NoLocators need a DOM or a native element tree. | PartlyObject recognition first, image matching as a fallback. | YesThe model acts on screenshots. | YesA person reads the screen. | YesDesktop, head units, HMIs, Citrix and canvas. |
| Runs your existing manual test cases as written | NoEach case is rewritten as code. | NoEach case is rebuilt as a recording, script or model. | PartlyA prompt can take the text. The rest is yours to build. | YesThat is what they were written for. | YesMarkdown, CSV or PDF, from your Git repository. |
| Maintenance when the UI changes | Code editsLocators and page objects follow the UI. | Object upkeepObject maps and recordings follow the UI. | In-housePrompts and harness are yours to maintain. | Re-briefingPeople adapt. Every run still costs the same hours. | Text editsNo selectors or recordings. Edit the step if the flow changed. |
| Defined verdicts, a screenshot per step, a transcript | PartlyPass or fail per test. Traces and screenshots when configured. | PartlyMature per-step reports and screenshots. Deterministic, so no transcript. | NoOnly the verdicts, reports and logs you build. | PartlyAs good as each tester's notes. | YesPASSED, FAILED, WARN, SKIPPED or BROKEN. Screenshot per step. Full transcript. |
| Runs air-gapped on your own model | YesRuns offline. No model involved. | YesInstalls on-premise. Check which AI add-ons need a cloud. | PartlyPossible if you host a capable model yourself. | PartlyIn-house, yes. Outsourced teams need access to the lab. | YesSelf-hosted model via vLLM. Offline licence key. |
| Drives hardware with nothing installed on it | NoNeeds a browser, driver or debug connection. | NoRuns on the machine under test. | PartlyOnly if you build the capture and input path. | YesA person with the device. | YesCompanion Mode: HDMI capture plus USB or Bluetooth HID. |
| Checks beyond the screen: APIs, databases, files | YesAnything a library can reach, written in code. | YesAPI, database and scripting steps, depending on the product. | In-houseEvery tool is yours to build, secure and maintain. | PartlyBy hand, in a second tool, next to the test. | YesTool Store: SQL over ODBC, HTTP, shell, file checks, QR and zoom. Off until enabled, recorded in the project. |
| Bench tools and your own systems in the same test | PartlyPossible in code; the test becomes a program. | PartlyThrough plug-ins or scripting, per product. | PartlyIf you write and maintain every integration. | NoCoordinated by hand across tools and people. | YesCustom C# tools and MCP servers, called from plain-language steps: e.g. Vector CANoe, dSPACE, CanTools. One report. |
| Who can author tests | DevelopersTypeScript, Python, Java, C# and more. | Trained testersRecorders and keywords. Scripting for complex cases. | AI engineersWhoever builds and runs the harness. | TestersThe people who know the product. | TestersPlain language. Developers extend it with C# tools and MCP. |
| Cost as the suite grows | Free licenceEngineering time grows with the suite. | Licences + upkeepCommercial licences plus maintenance time. | Tokens + teamInference on every run, plus the team that owns the harness. | LinearEvery run costs people time, or a fee per delivery. | Per agentPer concurrent agent. No per-test fees. Own model, no inference fee. |
Categories describe typical architectures as of September 2026. Individual products and configurations vary; check each vendor's documentation.
One by one
Pick what you use today.
See where AskUI fits.
Each one is good at something. Here's when to choose it, and when to choose AskUI.
vs. Playwright.
Excellent and free for web apps with a DOM. AskUI drives the web through Playwright itself.
Choose Playwright when
A pure web app, a stable DOM and developers who write the tests.
Choose AskUI when
The same suite must also reach desktop apps, head units, HMIs, Citrix or canvas.
What AskUI adds
- Desktop apps, head units, machine HMIs, Citrix and canvas UIs, in the same suite as your web tests.
- Test cases in plain language, so the testers who wrote them can run them. No code.
- A harness that builds the same layered context for every test, caches it between turns, and applies fixed rules: two attempts per step, then one of five verdicts.
vs. Ranorex, TestComplete and Tosca.
Mature, capable tools where objects can be recognised. Teams told us the cost is keeping tests in step with the UI.
Choose a desktop tool when
Objects can be recognised, and your team keeps the recordings and object maps in step.
Choose AskUI when
Keeping tests in step with the UI costs more than writing them did.
What AskUI adds
- No selectors, recordings or object maps. Your Markdown, CSV or PDF test cases, versioned in Git.
- AgentOS as a Windows SYSTEM service: survives RDP disconnect and drives the logon screen.
- Dreaming studies your runs and proposes rule fixes. You approve every line.
vs. building it with Claude or OpenAI.
A model can click through a screen in an afternoon. You'd still have to build everything that sits next to it.
Build it yourself when
You have a team to build and own the harness, verdicts, reports and security.
Choose AskUI when
You want that harness already built, on your own model, and can keep your agent through MCP.
What AskUI adds
- Five defined verdicts, a screenshot per step, reports and CI via GitHub Actions, GitLab or Azure DevOps.
- Your model: Anthropic, Azure AI Foundry or self-hosted vLLM. Offline licence key, secrets redacted, ISO 27001.
- Keep your agent: AskUI also runs as an MCP server for Claude or any other MCP client.
vs. manual and outsourced testing.
People catch what nobody wrote down. But every regression run costs the same hours again.
Stay manual when
You need a person to catch what nobody wrote down.
Choose AskUI when
The same regression run costs the same hours, release after release.
What AskUI adds
- The same test cases, run every night in CI instead of once per release.
- Evidence for every step: a verdict, a screenshot and the full transcript.
- Head units and devices you cannot install on, through HDMI capture and USB or Bluetooth HID.
When not to use AskUI
AskUI still helps when the same suite must also reach desktop apps or HMIs.
Send us ten test cases.
We'll run them on your system.
Or download AskUI Desktop. No sign-up.