Mutation Benchmark
50 scenarios · 6 UI patterns · realistic refactors plus real regressions that must NOT turn green
AgentQA + LLM (Levels 2, 5): measured, not estimated
One browser run with a configured LLM backend. Costs below are estimates from the backend configuration, not billing receipts.
Compare repair rate and wrong-element attempts separately. A clicked look-alike can still be rejected by the downstream assertions. These synthetic cases do not prove universal bug detection.
Results by mutation category
Positive rows: scenarios passing. Negative rows: scenarios wrongly passing (must be 0).
| Category | n | Standard Playwright | AgentQA deterministic (L1·3·4) | AgentQA strict | AgentQA + LLM (L2·5) |
|---|---|---|---|---|---|
| rename-id | 6 | 0/6 | 6/6 | 6/6 | 6/6 |
| rename-testid | 6 | 0/6 | 6/6 | 6/6 | 6/6 |
| restructure | 6 | 0/6 | 6/6 | 6/6 | 6/6 |
| class-scramble | 6 | 0/6 | 6/6 | 6/6 | 6/6 |
| text-variation | 6 | 0/6 | 6/6 | 6/6 | 6/6 |
| combined-hard | 6 | 0/6 | 2/6 | 0/6 | 5/6 |
| delayed-render | 2 | 0/2 | 2/2 | 2/2 | 2/2 |
| negative-removed | 4 | 0/4 | 0/4 | 0/4 | 0/4 |
| negative-decoy | 4 | 0/4 | 0/4 | 0/4 | 0/4 |
| negative-broken | 4 | 0/4 | 0/4 | 0/4 | 0/4 |
Which level fixed what
| Level | Name | Validated heals | Cost |
|---|---|---|---|
| 1 | Smart Retry | 2 | $0.00, offline |
| 3 | Multi-Selector Fallback | 57 | $0.00, offline |
| 4 | DOM Geometry & Layout Matcher | 2 | $0.00, offline |
| 2 | LLM DOM Repair | 0 | 3 heals in the LLM run (estimated costs above) |
| 5 | LLM Decompose & Rebuild | 0 | 0 heals in the LLM run (estimated costs above) |
Mean ladder latency per heal: 901 ms (median 921 ms), including Level 1's wait for late renders.
Real bugs: what AgentQA did
| Scenario | Result | What happened |
|---|---|---|
checkout/negative-removed Place order was removed (Cancel remains) | failed (correct) | refused #place (low_confidence) |
login/negative-removed Sign in was removed | failed (correct) | refused #submit (low_confidence) |
modal/negative-removed Save was removed (Close remains) | failed (correct) | refused #save (ambiguous) |
table/negative-removed Archive was removed (Delete remains) | failed (correct) | refused #archive2 (ambiguous) |
checkout/negative-decoy Place order replaced by a look-alike "Save for later" | failed (correct) | clicked a look-alike [data-testid="t6514c4"]; the assertion failed and the heal was rejected |
login/negative-decoy Sign in replaced by "Sign up" (different action) | failed (correct) | clicked a look-alike [data-testid="t6514c4"]; the assertion failed and the heal was rejected |
modal/negative-decoy Save replaced by "Apply preview" (does not persist) | failed (correct) | clicked a look-alike [data-testid="t7a8589"]; the assertion failed and the heal was rejected |
table/negative-decoy Archive replaced by a second Delete-looking control | failed (correct) | refused #archive2 (ambiguous) |
checkout/negative-broken refactored UI, but placing an order silently does nothing | failed (correct) | healed the right control; the app is broken, so the assertion failed and the heal was rejected |
login/negative-broken refactored UI, but signing in silently does nothing | failed (correct) | healed the right control; the app is broken, so the assertion failed and the heal was rejected |
list/negative-broken refactored UI, but Add silently does nothing | failed (correct) | healed the right control; the app is broken, so the assertion failed and the heal was rejected |
modal/negative-broken refactored UI, but Save silently does nothing | failed (correct) | healed the right control; the app is broken, so the assertion failed and the heal was rejected |
Copied from `python demo.py` (artifacts/demo) and `python scripts/run_benchmark.py` (BENCHMARK_RESULTS.json). Not hand-written. Generated 10/16/2026, 12:31:55 PM. See BENCHMARK_RESULTS.md for the methodology and limits.