End-to-End Testing Doesn't Need to be Deterministic. It Needs to Be Auditable
Part 5 of the Algorithms vs Intelligence series. Previously: 1. Why Traditional Test Automation Will Never Scale, 2. When AI-Assisted Testing Is Not Enough, 3. What Testing Looks Like When Intelligence Replaces Algorithms, 4. Agentic Testing in Production.
"Tests must be deterministic" is good advice at the wrong level for end-to-end testing.
The narrative says you should move your human-executed tests into a deterministic script. That move is expensive. It replaces non-determinism with determinism, and it fails from the first day.
A brake test on a bench is deterministic. That is a component test. The real requirement is different: a child runs onto the street, and the car stops in time. Reaction time, weather, tyre state, and brake wear all change. That outcome is a range, not a number.
Nobody in vehicle safety validates that with one fixed value. They validate it against a range and a tolerance. The verdict is stable. The execution is not.
Your software works the same way. Bad networks, corporate proxies, sun glare, stale sessions. Your user is on an aeroplane. Your user is behind a firewall you never saw. Your user has a half-finished sign-up from last week. A deterministic script encodes one path and meets a hundred.
The maintenance spiral
You know what happens next, because you lived it.
The script fails. Somebody adds a wait. The script fails again. Somebody adds a retry. Then a special case, then a second special case. After six months, the suite costs more than it saves. Somebody calls it flaky and stops running it.
So many end-to-end tests are still manual. This is not a tooling problem. Teams applied a component-level rule to a system-level job.
Separate the two ideas
- Determinism of the steps.
- Determinism of the verdict.
You need the second one. You do not need the first one, but the path stays auditable. A stable verdict sits on top of a path you can inspect: that is what "deterministic enough" means for end-to-end testing.
We've written before about how a test run already produces its own audit trail: an Execution Report, a Test Report, and a Summary Report, generated by default. If a verdict is in question, you open the trace. You do not take the agent's word for it.
We ran this directly. A precondition read: "The app is showing the Settings > Display screen." The run started on the home screen instead. Here is the agent's own reasoning, unedited, from the trace:
"The app is currently on the HOME screen, but the test precondition requires Settings > Display to be open. I need to navigate there myself since the precondition isn't met, though I'm uncertain whether I should fail the test or proceed with the navigation. I'm reconsidering whether navigating to the Settings > Display screen counts as a legitimate precondition setup step versus an invalid workaround. Since the precondition is explicitly listed as a requirement before test execution starts, and navigation there is straightforward and deterministic in this demo, I think the right approach is to navigate there to satisfy the precondition rather than skip the test entirely."
That is not a script following a fixed path. That is a judgment call, made and stated in plain language, that a human reviewer can agree or disagree with. The run went on to complete the step and record it:
The lesson here is a different kind of instruction. Instead of 'assert this state is true,' the precondition becomes 'if this state is not true, take this action instead.
A good end-to-end test takes a different route on Tuesday than it took on Monday. It is still a good test. It becomes a bad test only when the answer moves without a real cause.
Three things get you there. State the expected outcome in business terms, not selectors. Define the tolerance, the way a brake test defines an acceptable stopping distance. Keep the evidence of every run, so a human can audit the path afterward.
In our model, component and integration tests stay deterministic. That is still their job: remove the noise, find the defect fast.
End-to-end tests let an agent reason about the state. The human owns the intent and the report. This is the same shift Part 3 described as tests moving from describing what to do to describing what to verify. Here it plays out in a single incident.
Why an agent fits here
An agent does something a script cannot do in an end-to-end test. It judges instead of matching.
It sees a dialog that was not there yesterday and understands what the dialog is. It sees a slow call and waits for a reason, not for a fixed number of seconds. Within a bounded number of attempts, it can try a different path to the same goal, and knows when to stop rather than guess indefinitely.
That is real auto-healing, in a narrower sense than the term is usually sold. The agent isn't rewriting the rules of the test. At each step, it reasons about the actual state of the screen, rather than relying on a fixed reference to a UI element. If the button moved, the move may be the bug worth catching, not a reference to patch silently.
Healing works on the state, not the locator. When the agent does resolve something, it writes a note about what it found. The note carries as much value as the fix. You get a report that tells you what changed, not a green tick that tells you nothing.
We tested the other side of that same judgment directly: what happens when the agent meets an error it did not expect. A test step tapped into a screen that failed to load, producing an error dialog. The agent's own record of what it did next:
"The dialog 'Profile sync failed' appeared as expected, with a 'Dismiss' button. This is the known/expected behavior per the UI documentation. I will not dismiss it or attempt to resolve it, as per test instructions - I'll just observe and document it."
No retry, no click through the error, no attempt to route around it. The instruction we gave was simple: if there are error messages, do not try to resolve them, mark the test as failed. The agent followed it, and the report shows exactly what it saw, not just a pass/fail line.
The examples above ran against a vehicle HMI demo, but nothing in the pattern is automotive-specific. The same reasoning applies to any screen an agent can see. Three different runs, three different paths, and one thing stays constant across all of them
The testers
Does this remove the tester? No. It removes the part of the job that never needed a person.
The tester was always the non-deterministic element in the test system. A human deals with the unexpected dialog, the slow network, the strange device state. Teams tried to replace that human with a script, and lost the exact capability that made the test work.
That work goes to the agent now: the state handling, the note-taking, the clicking itself. Two skills stay with the tester, and both grow in value.
The first: state the expected behaviour. Somebody decides what "correct" means for the business. An agent cannot decide that for you, and should not.
The second: judge the report. Somebody reads the evidence of a run and says whether the outcome is acceptable.
Every developer who codes with AI already works this way. State the intent, review the result. Testing moves to the same model. The role does not shrink. It moves up one level.
Where to start
Take one end-to-end test your team runs by hand. Do not script it. Write down what a correct outcome looks like in business terms.
In AskUI, that means a plain-language test case: preconditions, numbered steps, a postcondition stated as "Test passes if…" No selectors, no scripts. Run it once against the real application, then read the report: not just the pass/fail line, but the per-step trace of what the agent saw and did. If the report tells you something you didn't expect, that's the signal worth investigating before you write a second test.
Determinism is a tool. Use it where it removes noise. Do not use it where it hides your user.
FAQ
Does this mean end-to-end tests don't need to be reliable?
Not in our approach. The verdict is what needs to be reliable. We treat the path an agent takes to reach that verdict as free to vary between runs, as long as every run is logged and auditable.
How does self-healing actually work here?
- The agent reasons about the actual state of the screen at each step, rather than relying on a fixed reference to a UI element
- If the button moved, that's read as a possible bug worth flagging, not just patched over silently
- It doesn't extend to working around errors. An agent that hits an error is instructed to stop and report it, not improvise past it
What happens when the agent isn't sure a precondition is met?
In one run, a precondition wasn't met at the start, and the agent reasoned through it in plain language before deciding whether to satisfy it or skip the test. That reasoning is logged in the trace, not hidden inside a pass/fail line.
Does the agent ever retry indefinitely if something goes wrong?
No. There's a maximum of two attempts per step. After that, it stops and reports the state rather than looping or improvising a workaround.
What does a test report actually show, beyond pass/fail?
Each step includes what the agent interpreted the instruction to mean, what it expected, and what it actually saw, with a screenshot. You can read why a step passed or failed, not just that it did.