Agentic Testing: How It Differs from AI-Assisted Automation

The vast majority of QA teams already use AI in some form or another. A model will generate test cases from a user story. A tool will suggest a new locator when a script breaks. A chatbot will generate test data. The test suite itself, however, still works the way it did years ago. An engineer decides every step in advance, and the machine just runs those steps every time.

Agentic testing is a game changer. You give an AI agent a goal, and it plans the steps it needs to take, does the work in the app, checks what happened, and then decides what to do next. Playwright currently has three built-in test agents: a planner, which explores the app and produces a test plan; a generator, which takes that plan and turns it into actual tests; and a healer, which re-runs the failing tests and fixes them.

The World Quality Report 2025-26 from Capgemini and Sogeti surveyed 2,000 senior execs and found that 43% of organizations are still in the early stages of experimenting with Gen AI in QA, and only 15% are using it at scale across the organization. Gen AI is at the very top of the list of skills that quality engineers need (63% of respondents).

This article covers what agentic testing is, how it differs from AI-assisted automation, where it works and where it doesn’t, what tools are available, and how to run a pilot while retaining control over release quality.

What Is Agentic Testing?

Agentic testing is a form of testing in which an AI agent works to achieve a test objective on its own. It plans its actions, carries them out in the application via the UI or API, observes the result, and then decides what to do next. That cycle of observing, deciding, and acting is what makes a tool agentic. A tool that merely writes and edits test scripts is not agentic.

Think of a recorded script as a route that’s been saved in your navigation app. It’ll work until the road closes, and then it’s all up in the air. An agent is more akin to a driver who knows where they’re going and will find another route. The tester provides the objective, e.g., “a returning customer can apply a discount code and complete checkout,” and the agent works out the path. If a promotional popup appears in the middle of the flow, a script that hasn’t been trained to handle it will just fail. The agent can close that popup and carry on.

Two things are very easily confused. First up is agentic testing, which uses agents to test software. Testing AI products (i.e., a chatbot, a recommendation model) is a completely different proposition. ISTQB has separate certs for that: CT-GenAI is the cert for using generative AI for testing, and CT-AI is the cert for testing AI-based systems.

Secondly, agents don’t have to work completely on their own. Most teams will have the AI agent do the work, with someone else in charge of the goals, limits, and final decision.

​

Why Agentic Testing Is Getting Attention

• Testing is a drag on teams. The World Quality Report still rates testing as one of the most manual and slowest parts of the software delivery process. AI has now accelerated the coding process, so there’s even more to test.
• Very few tests are automated. On average, only around a third of the test cases are automated. About a quarter of the new automated scripts we’ve written over the past year were created using Gen AI tools.
• Maintenance takes time. Any tests based on selectors will break if a button, a label, or the layout changes. The team ends up spending time fixing tests rather than actually finding bugs.
• Results are good but very uneven. Companies using Gen AI in quality engineering report an average productivity uplift of 19%. Around a third report very modest gains. The report ascribes this to a lack of skills, unclear ownership, and poor governance.
• Skills and certifications are on the rise. Gen AI topped the list for quality engineers (63%), just edging out core quality engineering skills (60%). The ISTQB General Assembly approved the CT-GenAI syllabus on 25th July 2025. Version 1.1 went live on 27th April 2026.
• Analysts expect to see even more agents. Gartner forecasts that by 2028, around a third of enterprise software applications will have agentic AI. That was less than 1% in 2024.
• Buyers are asking about it. In 2026, a number of testing vendors published buyers’ guides and ‘best agentic QA companies’ lists. That’s evidence the term is now in use when people compare tools and partners. All of these are vendor-produced, so take their claims with a pinch of salt.

How Agentic Testing Differs from AI-Assisted Automation

The key difference is who picks the next action during the test run.

In AI-assisted automation, a person still runs the process. The AI helps write the script, suggest a new locator, generate data, or explain a failure. The end result is still a normal script that does the same thing every time. In agentic testing, the agent makes decisions as the test runs. A manual tester makes each of those decisions live. A script has those decisions baked into the code. The agent also makes decisions as the test runs; the difference is that, instead of a person, a model makes them.

​

​

Self-healing has levels. Replacing a broken selector after a CSS class has changed is a minor fix. That’s firmly in the AI-assisted automation camp. Finding a new path after a settings page shifts from one screen to several tabs is a bigger change. That’s much closer to agentic behavior. When you’re looking at a tool, you need to ask yourself which of the two it actually does.

A lot of teams use both. Playwright’s agents create the standard Markdown plans and test files that stay in your repo. The agents write the tests, and the subsequent runs are just plain old scripted runs. That’s how a lot of teams operate. The agents will explore, draft, and suggest fixes, and the scripts then do the actual checks to determine whether a release is approved.

Three Levels of AI in Testing

AI-assisted authoring

What it is: a person using an LLM to write test cases, scripts, and test data, or to explain a failure.

When to use it? You already have an automation framework and just want to build and fix your tests quicker.

Trade-off. The generated tests can pass whilst checking very little, indeed. A large number of tests does not necessarily equate to good coverage.

Agentic test creation

What it is: given a requirement, an agent reads the existing tests, explores the app, writes a plan, writes the tests, runs them, and then gradually improves them. That’s how the Playwright planner, generator, and healer work. The same idea is described by enterprise vendors like Tricentis in their own products.

When to use it? New features with a very small test suite, or old areas that have never been automated.

Trade-off. You still have to go and look at the assertions. A healer that edits an expected value to make a test pass can be hiding a real bug. Playwright’s healer is documented to just skip a test if it thinks the feature itself is broken. This is the behavior you want from any tool.

Agentic execution

What it is: the agent works in the application during the run, with no fixed script. Some tools work by visually reading the screen. That’s useful if there are no selectors available (embedded interfaces, HMI’s or games).

When to use it? Critical user flows, screens that change quite often, and smoke checks after a deployment.

Trade-off. The results can be very different between runs; runs are more expensive than a script, and you need the step-by-step logs to see why the agent did what it did.

How an Agentic Test Runs

1. Set the goal. The tester writes the goal, the starting conditions, and what success looks like (i.e., a user story).
2. The agent creates a plan. It takes the goal and turns it into a series of steps in the UI or API. It can use the documentation and previous tickets, and it has access to a test environment that’s already set up.
3. The agent works and watches. It clicks, types, calls endpoints, and reads the responses.
4. The agent will adjust. If something moves or an unexpected element appears, it will retry or take a different path within its permissions.
5. A separate check confirms the result. Critical results (e.g., an order total, a permission rule) are verified via fixed assertions or via another system (e.g., an API response, a database record). The agent’s own opinion is not sufficient.
6. A person does the review. The run leaves its mark in the form of steps, screenshots, logs, and the agent’s reasoning. The person then decides which of the findings are genuine bugs and which changes they’re going to accept.

Where Agentic Testing Works Best, and Where Scripts Are Better

Good fits

• Critical user flows. Checkout, onboarding, and subscription flows all matter to the business, change quite frequently, and are all expensive to script.
• Interfaces that change all the time. Screens behind feature flags and settings pages break selector-based tests most often.
• Smoke checks after a deployment. Agents can quite quickly cover paths that nobody’s actually scripted, even outside of working hours. See also exploratory testing and smoke testing.
• Screens lacking stable selectors. Embedded UIs and canvas screens are a real pain for tools that rely on locators.
• Reproducing bugs and a first look at failures. An agent can attempt to reproduce a user’s report from its description. Bugs that only manifest occasionally are not good candidates.

Scripts are the better choice

• Money, access rights, and data. Totals, permissions, and stored data all need to produce the same result every time.
• Fast at the low level. Unit and API tests are trivial and stable. The agents don’t really add anything there.
• Audit-style checks. Regulated flows will typically require a clear pass/fail record.
• Race conditions and deep backend problems. Practitioners have reported that agents don’t reliably find these.

Tools for Agentic Testing

Tools change all the time, so check each one against its current documentation. The list below is just a selection of tools, not a ranking in any way. Some of the descriptions are taken directly from the vendor’s own documentation.

• Playwright Test Agents. Planner, Generator, and Healer agents. Plans and tests are normal files in your repo, so you can review every step.
• QA.tech is a commercial platform. The agents test goals via the UI (visually), build up a knowledge graph of the application, and then post the results as a PR on GitHub.
• AskUI. Computer-use agents that work directly with the screen. They’re for settings with no accessible elements.
• BarkoAgent (TestDevLab) is a chat-style LLM that can connect to tools such as Selenium and Jira. It can create tests from documentation and also review pull requests.
• Tricentis qTest and UiPath (both enterprise platforms). Tricentis talks about agentic test creation in qTest. UiPath has agentic testing capabilities in the form of Autopilot, Agent Builder, and Test Suite.

Best Practices for Adopting Agentic Testing

• Start with one critical flow. Pick a flow where a failure would negatively impact revenue or confidence in the release. Run the agent alongside your current suite before you start to rely on it.
• Write your goals as specifications. The World Quality Report recommends treating AI instructions as technical specifications. Vague goals lead to a mess of runs and rework.
• Check the results outside the agent. Verify the key outcomes with fixed assertions or against another system. The same model shouldn’t be making the decision as to whether the test has passed and also be the one to choose the path.
• Keep people in control of the changes. The agent can propose a new test or a locator fix. People approve changes to expected values, skipped tests, and release rules.
• Limit access and isolate the data. Use test accounts and test data that you can reset. Give the agent no access to production and no way to do anything destructive.
• Record every run. Keep the steps, screenshots, logs, and model/tool versions. A model update can change the behavior even if your code is exactly the same.
• Measure the actual results. Track confirmed bugs, false alarms, review time, maintenance time, and cost per useful finding. The number of tests generated tells you nothing.
• Build skills and ownership. The World Quality Report states that organizations that have scaled AI have clear roadmaps, ownership, and governance. It also recommends that you check that the teams are able to challenge the AI output rather than just use the tools.

Lots of vendors and consultants will tell you a 30-day pilot is the way to go. A fairly straightforward plan would be to establish your baseline of bugs, flaky tests, and maintenance effort. Run the agent in shadow mode (so it doesn’t block any releases) and then compare the actual findings and total cost against your current process.

From there, you can decide whether to scale it up, tweak it, or walk away. You could also go so far as to introduce a few known bugs to see if the agent actually finds the real problems or just sits there doing nothing.

​

Risks and Limits of Agentic Testing

• Fake or flaky tests. An agent can write tests for features that don’t exist at all, or can report a test as having passed when it hasn’t. That’s where separate checks and human review come in.
• Unreliable results. The same task can pass one go and fail the next. Errors accumulate across a chain of tool calls. Repeat the critical flows and have the fixed checks as the final gate.
• Tests that quietly get weaker. A repair that alters an expected value can turn a genuine regression into a green build. A person has to approve any changes to assertions.
• Model updates can change behavior. The agent you checked last month may behave differently after an update. Always record the version and verify again after any changes.
• Privacy and integration issues. The top impediments to scaling Gen AI in QE, according to the World Quality Report, were data privacy risk (67%), integration complexity (64%) and concerns around hallucinations and reliability (60%). Anonymized data, appropriate enterprise models, and audit trails will help address data protection concerns.
• Too much access. In July 2025, an AI coding agent from Replit deleted a live production database during a code freeze. Big news in the press. It was a coding agent rather than a test agent, but the issue is the same: too much access and not enough oversight. You have to give agents the minimum access they need.
• Cost and hype. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027 for cost, lack of clear business value, or poor risk controls. A pilot that’s tied to definite outcomes is the way to avoid that.

How a QA Partner Can Help

If you don’t want to build this yourself, then an independent QA provider can help in a few areas. They can establish the baseline and success metrics for the pilot, review the agent’s output from the outside, maintain fixed automation as the release gate, and test the AI/LLM features in your product. An outside perspective is useful, as neither your dev team nor the tool vendor will be a completely neutral judge of the agent’s output.

The White Test Lab team combines this kind of review with hands-on automation in Playwright and Cypress, API testing, and AI-based product testing. Agents are added on top of the working QA process rather than replacing it.

Conclusion

With agentic testing, the agent decides how to get to the goal as the test runs. That’s the fundamental difference from a script. AI-assisted automation makes the existing approach much faster and cheaper to maintain. Agentic testing can cover user flows that scripts are poor at and reduce some rework. Both have their place. The best outcome is when the agent goes out and explores, reviews the failures, and then the fixed checks and the people ultimately decide whether the software is good enough.

The World Quality Report states that the companies that scaled AI in QE did so by investing in skills, governance, and alignment rather than simply buying tools. For most teams, the next step is to run a small pilot on one flow and benchmark it against the existing process. If you need help planning that pilot or to reviewing the results of it, White Test Lab can help.

FAQ

What is agentic testing in simple terms?

Agentic testing is testing where an AI agent is given a goal (e.g., complete a checkout) and then decides for itself how to achieve it. It plans the steps, performs them in the app, checks the outcome, and alters its course if things are different. A human sets the goal, the boundaries, and the success criteria.

What’s the difference between agentic testing and AI-assisted test automation?

In AI-assisted automation, a human still runs the process. The AI helps write or fix the scripts, and the scripts then run in exactly the same way each time. In agentic testing, the agent chooses the next action during the run, so results can vary from run to run. The easiest way to tell them apart is to ask who decides the next step while the test is running.

Will agentic testing replace QA engineers or test automation?

No. Agents can take over the repetitive work of writing test paths, running them, and doing a first pass at the failures. People still call the shots on what is risky, design the independent checks, judge the ambiguous behavior, and call the release. Scripted automation will of course still be in place for the stable regression and release gates. The QA role shifts to supervising and designing the test system.

Is agentic testing the same as testing AI agents or LLM-based products?

No. Agentic testing uses AI agents to test any software, including normal web, mobile, and API applications. Testing an AI product, for example, checking a chatbot for hallucinations or running prompt regression, is a separate task. ISTQB has different certifications for the two: CT-GenAI for using generative AI in testing and CT-AI for testing AI-based systems.

What are the biggest risks of agentic testing?

The main risks are made-up or weak tests, unstable results, quiet changes to expected values, behavior changes after model updates, data privacy, and over-access. In the World Quality Report, data privacy risk (67%) and integration complexity (64%) were the most common barriers to scaling Gen AI in QE. Isolated test data, minimum access, full run logs, and human approval of assertion changes reduce most of these risks.

How should a team start with agentic testing?

Start with one important user flow. Write down your current bugs, maintenance effort, and cost. Run the agent next to your existing suite without blocking releases. Compare confirmed findings, false alarms, review time, and total cost, and then decide whether to scale. A small, measured pilot tells you more than a vendor demo.