Slack’s Agent-Driven End-to-End Testing: Making UI Automation More Resilient
Slack did not present agentic testing as a replacement for CI. Its 200-plus-run experiment treats browser agents as an exploratory layer for brittle UI flows, with clear costs, boundaries and evidence requirements.
An end-to-end test can be green while missing the real problem. Or it can fail because a button moved even though the user journey still works. Slack’s “agentic testing” experiment addresses that gap by asking an agent to verify a goal instead of forcing one exact sequence of selectors: can a user send a thread message in a test workspace, and is it visible in the expected place?
This is not a new product category or a reason to put every test under an LLM. Slack compared more than 200 agent-driven E2E runs using Playwright MCP, Playwright CLI and agent-generated Playwright tests. The experiments used non-production data. The useful question is therefore not “Will an agent replace our tests?” but “Where does a flexible path add value inside the testing stack?”
From journeys to goals
A traditional test describes a journey: click, click, type, assert. That is valuable because it is fast, repeatable and easy to budget in CI. Its weakness is that it is tightly coupled to the journey. A selector can break after a UI change even when the business outcome remains available.
An agent-driven test describes a goal and an observable result. The agent sees the current state, chooses an action and checks the next state. In Slack’s runs, the broad flow stayed consistent while the exact action sequence changed. A search was confirmed through a suggestion in one run and with Enter in another; an existing state was reused or rebuilt.
That freedom is useful, but it is not free. An agent may take a valid detour, make a needless exploration or misread a plausible result. The final claim must still come from explicit assertions in a controlled test workspace. “The agent found a path” is not, by itself, a quality signal.
What Slack actually measured
Slack compared three execution models: an agent using Playwright MCP, an agent using Playwright CLI, and generated Playwright tests that then ran deterministically. The scenarios included a simple thread reply and a more complex search-and-navigation flow. Agent instructions were tested both as natural language and as structured YAML.
In the simple thread flow, the MCP agent had a zero percent failure rate in this experiment; in the more complex search-discovery flow it was around twelve percent. The CLI approach was around twelve and twenty percent respectively. Generated Playwright tests were reasonable on the simple flow but degraded substantially on the complex one. These are results from Slack’s setup, not a universal ranking for every application.
Runtime is part of the decision. MCP runs averaged roughly five to eight minutes, CLI runs about nine to eleven minutes. Generated tests took about three minutes including generation, and repeated execution makes them cheaper still. Slack reports roughly 15 to 30 dollars per agentic run in this experiment. That is hard to justify for every commit, but it can be reasonable for a difficult flaky flow.
Why the execution layer matters
The MCP-versus-CLI difference is not merely a tooling preference. MCP gives the agent browser actions and observed state in a tighter loop. With CLI, actions, waits, snapshots and extra commands more easily become separate conversation turns. Slack observed more context growth and more authentication, timing and navigation failures in that configuration.
That does not make MCP a universal answer. A long flow can become expensive with MCP too, because every new snapshot adds context. CLI can be adequate with disciplined session handling and sparse observation. The practical test is: how much information is genuinely new per step, how is state preserved, and can a failure be replayed with the same inputs?
A safer starting point is a division of labor. Playwright or Cypress remain the tools for short, repeatable checks. An agent first handles cases where the UI drifts often, a failure is hard to reproduce, or a fixed click path creates too much maintenance. A successful agentic run can then be preserved as a deterministic test. Exploration flows back into a cheap regression instead of becoming an expensive permanent loop.
Guardrails are part of the test
An agent should not “just try things” until the screen looks green. Every run needs four pieces:
- Test system: an isolated environment with synthetic accounts and data, never an unrestricted production account.
- Goal and assertion: an observable end state, such as a message in the correct thread, not merely “find out whether it works”.
- Action space: allowed pages, tools, writes and maximum steps; risky actions need a block or human approval.
- Stop and trace: time and step limits, state captures at important points, tool calls, result and termination reason.
These rules determine whether a team can investigate a red run. A useful trace answers which state the agent saw, why it chose an action, which alternative it rejected and which assertion failed. Without that chain, resilience becomes an impressive but irreproducible snapshot.
A practical pilot
For a team with one flaky UI flow, a small pilot is enough. Collect ten to twenty real but non-sensitive failure cases. Define one goal with a clear assertion. Run it outside normal CI with a fixed budget and a recording of every decision.
Then separate three measures: did the agent reach the goal, how often did a human need to intervene, and what were runtime and cost? Also check whether flexibility absorbs real UI changes or merely creates random paths. Only then should the team consider agentic pull-request exploration or automatic conversion of a successful run into a stable Playwright test.
Conclusion: complement, do not replace
Slack’s work is a useful, deliberately limited contribution to the agent debate. Agentic E2E tests can explore UI flows more flexibly and provide clues for difficult failures. They are also slower, more expensive and less naturally reproducible than deterministic tests. Their first practical home is an additional layer for exploration, debugging and production-bug reproduction in a safe environment.
The strongest architecture is not either-or. Scripts check the known path in CI. Agents examine whether the goal remains reachable when the path changes. When a flexible run discovers a stable, frequently needed journey, the team can turn it back into a small, transparent test. That feedback loop is what turns an interesting experiment into engineering practice.
Frequently asked questions
FAQ: Agent-driven E2E testing
Does an agent replace classic UI tests?
No. Deterministic tests remain essential for fast regressions, contracts and CI. Agentic runs complement them where UI paths vary or failures are difficult to investigate.
Why not run every test agentically?
Slack’s experiment showed higher runtime and much higher cost per run. That is usually unnecessary for every commit; targeted debugging and exploration fit better.
What is the most important safety measure?
An isolated test system with synthetic data and restricted actions. Every run also needs a clear goal, an assertion and a stop limit.
MCP or CLI?
It depends on the execution layer. MCP was more stable and compact in Slack’s experiment, but that is not a universal promise. State handling, observation frequency and replayable traces matter more.