The big story out there is that we have a handful of frontier companies that are very rapidly pushing the envelope of what’s possible in automating the software development life cycle by introducing AI tools like Codex or Claude into all of this. And they are promising that we can get rid of a lot of things that we’re used to, including people, software engineers, and testing.
Other companies are really looking at that right now—and that’s probably 90 or 95% of the market—with some sort of fear of missing out, with an urge to jump on the bandwagon. And, of course, also with the promise of losing people and reducing your payroll and getting things more controllable and efficient and run by machines. You really can’t blame CIOs for wanting to do that.
But in testing, we are especially vulnerable to this kind of idea.
And there are two reasons for that.
#1 Testing is easy
One is that testing is the kind of thing that seems like everybody can do it. Like, my mom can do it. It’s that easy. You just try things out, you find problems, and then you fix them.
For that reason, it doesn’t really have the recognition for being a highly skilled job, which it is (it may have that in common with things like UX).
#2 Testing is automatable
The second thing that makes us vulnerable is that we have gone, for the last 10, 15 years or so, massively in the direction of “test automation,” or what I call automated checking.
And of course, this is something that the new AI tools are really, really good at.
What about Quality?
Meanwhile, software is not getting any better. It’s still mediocre, and failure is probably a daily experience for users of software of any kind.
AI is not making this any better. In fact, I would say that it introduces entirely new problems into the mix.
So testing stays relevant. It is necessary. But AI also promises to replace it.
So where is AI actually strong? Where does it deliver value? And where do humans deliver value? In other words, where are the safe zones? Where do we win with what we can do? And where does AI win with what it can do?
To get a handle on things, I propose that we look at it in three different batches, three areas that we work in that are very different in their methods and where we engage.
They are: defect prevention, defect detection, and giving confidence.
Preventing Issues
This is everything that happens before you start construction or before you start coding. And to set yourself up to achieve quality in principle, there are really two things that matter here.
#1 Repeatability
One is structured, repeatable process, so you get predictability and things are not just happening in a chaotic way. That’s important.
#2 Risk
Number two is risk identification—the ability to see where things are going to break and where issues are going to be impactful, and then to be able to determine what to do about that.
Now of these two things, the first one, the process, is AI-able. AI can help with this. It can generate artifacts, and it can automate processes.
But the second one, the risk identification, is super dependent on detailed knowledge of your individual context, of your domain, of the technology you’re using, and in bringing all of that together to see how it’s going to break.
This is not something that AI is good at, and if you’re good at that, AI is not going to come for your lunch. Defect prevention is the domain of humans. AI can help you a little bit, but this will not go away.
Why not? Can’t AI just take all the Risk out of Software Development?
Can’t AI just generate all the requirements and the designs and respond to exactly what you want it to do and make it perfect?
The answer to that is no.
I think where this comes from, this sort of expectation, is from the idea of vibe coding and getting the impression that the AI tools exactly what you want them to do and how you describe it.
And in a small shop, if you’re working alone, this might kind of work, but at a business level, this does not work. Business software needs to live in a complex environment of the technology it sits on, the human users it serves, and ultimately the business that tries to benefit from that. There’s an enormous amount of context that doesn’t typically live in the head of a single person or in your documentation.
And so really, it’s a simplistic view.
This is not something AI is good at. AI doesn’t have a worldview. It doesn’t understand your business. It doesn’t care about your users. And it is often not able to exercise your technology properly either in most cases for reasons I will mention further on.
So no, the answer is, risk identification is not something that AI is going to take away. Risk is going to be alive and kicking, and it’s your job to design testing and quality assurance methods for that to mitigate it.
Finding Issues
In order to understand how that works, imagine what a test pilot does. A test pilot gets into something that was built as brand new and takes it up into the air and pushes the envelope, tries to find issues, tries to find things that break, that make this thing less safe, less efficient, dangerous even. That’s testing.
To do that, you need deep experience in the technology you’re exercising, the business world you’re operating in, the expectations of users, and really a holistic context of how the whole thing needs to work in the real world to be able to find issues that matter.
Can’t AI just find and fix all the defects?
Now, there’s a story out there that because AI is fairly good at generating code, of doing its own reviews, of generating tests, running the tests, finding issues, fixing the issues, deploying and running things all the way down to production, it is good at testing and it is good at finding issues.
That, I think, is not right at all.
I think that story is just plain wrong. And it comes really from a misunderstanding of what testing is and what test automated checking does, because that’s really what’s happening here.
What you get is an AI that checks its own output. Is this something that exercises and challenges the envelope, that challenges assumptions, that tries to break something? Or is this really just a self-confirming cycle that’s happening here?
And so this is problematic for several reasons.
#1 AI finds the easy issues
This kind of automatic check-in is not what a test pilot does. This is what a guy with a clipboard does. And this is not something that will help you improve quality. It will, at best, find and fix the easy issues, things that take the cognitive load away from the real testing.
#2 AI’s reach is limited.
If you are in a fully cloud native environment where everything is automatable and AI can reach every single part of the chain, this maybe work. But most companies are not in this position. Most companies operate on mixed environments with legacy technology, with things that are not in the cloud, manual processes, things that are bespoke and simply not easily automated, so that the agents can’t even get to it.
#3 New bugs
And number three is the fact that AI is introducing entirely new bugs, things that a human engineer would probably not break. So, all of this means that testing—the real testing, what the test pilot does—is not something that you can abolish or get rid of.
In fact, there’s probably a need for more of it.
Continuous Confidence
So now we get to our last area, which is giving confidence. This is the domain of phase gating, reporting, and first and foremost, automated checking.
Because in most companies, we have teams of developers working, code is constantly being changed, merged, and migrated through environments, and there is this continuous need for reassuring ourselves that we haven’t broken anything. This is what we call regression.
The best way to do that is to have an automated suite of checks that runs constantly and should always be green.
Here’s where the bad news is.
If that is what you do, if your specialization is in creating these automated checks, then AI unfortunately is going to come for your lunch.
The tools that we have today are far too good at this.
My guess is that 9 out of 10 people that specialize in automated checking will probably not do this anymore in a few years, and those that are remaining in the job will use AI and agents to orchestrate the process.
So what if test automation is your thing?
What if you specialize in this and you are being threatened now by all the AI tooling coming for your job?
Well, you really only have two choices here.
#1: Master AI
Your first option is to be the one in ten. To be that one person left standing on the ice floe as it is shrinking, to be simply better than everybody else at using and understanding the new tooling and being able to orchestrate agents and automate the whole process and sit on top of it.
#2: Master QA and Testing
Your other option is to retool and move left towards the areas of issue prevention and issue detection. So classic QA and testing.
That means building up your domain knowledge, understanding your business, your users, and understanding the technology that you work with. Abandon the black box approach (if you ever had it), look under the hood and see how things are really working, so you develop an intuition what is fit for purpose and how quality can be improved.
All of this will make you a better tester. It will help you find issues and it will improve quality in the product.
And that need does not go away.
In fact, I think it will probably grow stronger with the additional problems that AI is introducing into all of this.
Your Thoughts?
So let me know what you think in the comments. I am really curious how this impacts you and where you want to go.
And, of course, if you like this kind of content hit that Subscribe button and I’ll see you next time on Testing Gold.