The question.
UI automation breaks when selectors move, sessions expire, or a task completes only partially. A single successful click does not establish task completion.
The design combines deterministic actions with agent planning, preferring service APIs where available and verifying visible results when a browser is required.
The proposed approach.
- Represent tasks with explicit inputs and success criteria
- Prefer stable locators and verify outcomes after meaningful steps
- Checkpoint progress and detect stale sessions
- Pause before irreversible actions and record completed side effects
Target architecture.
Task specificationAction plannerBrowser toolsVisible-state verificationCheckpoint store
The difficult parts.
- Changing page layouts and ambiguous controls
- Retries that accidentally duplicate actions
- Distinguishing an intermediate screen from a completed task
How to evaluate it.
- Completion rate across controlled UI variations
- Recovery from interruptions and expired sessions
- Duplicate-action rate and intervention frequency
Planned deliverables.
- A reusable browser task specification
- A failure-and-recovery test collection
- An auditable workflow execution design
Design principles.
Verify the resulting state, not just the attempted action.
Use an API when it makes the workflow more dependable.