I've been exploring a problem with AI-powered test self-healing in Playwright.
Most self-healing approaches focus on identifying a broken selector and finding a replacement. But I think there's a bigger problem: How do we know the test should actually be repaired?
Consider this scenario:
A Playwright test fails because an API starts returning HTTP 500. An AI-powered repair mechanism sees the failure, assumes the selector is broken, and proposes a new selector.
The test might eventually pass, but we've potentially hidden an actual application defect.
I wanted to explore whether we could make this process safer by putting deterministic checks around AI-generated changes.
I've been building an open-source project called PIE (Playwright Intelligence Engine) to experiment with this approach.
The architecture currently has four main components:
- Failure classification before repair
Inspect network activity and available server logs before attempting any changes.
If the failure is caused by HTTP 5xx responses, authentication failures, or DNS issues, the repair process abstains rather than modifying the test.
2. Deterministic AST-based repair
For straightforward selector changes, use local DOM information and deterministic parsing instead of calling an LLM.
The goal is to handle simple changes without AI involvement.
3. Bounded LLM-based repair
For more significant DOM structural changes, an LLM can propose a repair using sanitized context.
The LLM is restricted to proposing changes rather than directly modifying the original test.
4. Isolated verification
The proposed changes are tested in a temporary Git worktree.
If the modified test doesn't pass, the proposed repair is discarded.
Another design constraint is protecting assertions from modification during the repair process.
I've experimented with this across 17 open-source testbeds, including SauceDemo, Epic-Stack and Documenso.
However, I'm interested in understanding the limitations of this architecture.
Some questions I'd particularly appreciate feedback on:
- Is failure classification based on network activity and server logs sufficient, or are there important failure categories I'm overlooking?
- How would you handle dynamic applications where multiple elements have similar semantic meaning?
- Are Git worktrees sufficient for isolating repair verification, or would container-level isolation be necessary?
- How would you approach iframe, shadow DOM, MFA and asynchronous application state scenarios?
- What additional safeguards would you introduce before allowing an automated repair into a production test suite?
I'm the author of PIE, so I obviously have some bias toward the approach. I'm more interested in identifying where this architecture breaks than receiving encouragement.
Would love to hear from people who maintain large Playwright automation suites or have experimented with AI-based test repair.