AI and Software Testing
How does AI alter your software testing strategy? Does it make testing more or less beneficial? Does it change where testing occurs in your SDLC?
How Did This Pass QA? Quality in the World of AI #
Coming into the CTO club for the first time as a notetaker, I expected the conversations around me to be extremely corporate. However, the days when CTOs actually sat in front of a laptop and coded are far from behind us. With the advent of AI, it seemed like everyone in the room was intimately familiar with the tools. I was even more surprised when I heard one of the “cardinal rules” of classic software engineering broken: “we don’t need unit tests.” Needless to say, it was an eye-opening experience and I learned a ton. Thanks for the opportunity to hang out in a room I wasn’t supposed to be in. Here are the highlights.
The Topic of the Night: The Unit Test Is Dead #
We opened the night with the lightning topic: “the unit test is dead.” Almost everyone’s CLAUDE.md file has some version of “make sure all your unit tests pass” in it. The problem is that you quickly lose track of which unit tests are actually correct. Different tests passed at different times, written by different runs of the same model, and now you have a suite that nobody trusts and nobody wrote.
The group decided the pyramid should flip. Where you used to have a wide base of unit tests and a thin cap of UI tests, the room described the opposite: business logic still belongs in unit tests, but everything above that moves to E2E coverage with tools like Playwright. When you are writing specs that execute for hours at a time and code is being generated faster than anyone can read it, the unit test stops being the thing that tells you the system works.
Which raised the question: if the same engine wrote the code and the test, who decides which one is broken? Left alone, the model will happily fix the test. That is exactly backwards, and it is why the room kept saying QA is back, just not the QA we’re used to. Adversarial testing. Continuous discovery testing. Point Codex at Claude’s output and you will get five or six P0s and P1s before lunch.
Manual, Automated, or Simulated? #
If agents write the code, do you still need a human clicking through the app? Maybe, or maybe a simulated human instead.
For example, you can record how real users actually move through your system, train a model on that behavior, and run farms of agents that take on user identities. End users always find some random way to use a feature that you never intended. Rather than waiting for feedback to come back, you ask the model to predict how people will break the new thing.
We were given a live example of this. A platform digitizes debt collection, an industry still running on phone calls and physical mail, in a space where compliance is state by state plus federal and you have to deal with the lawyers. In an agentic system, agents act as debtors: paying, settling, defaulting, opting out. Because you know what actually happened yesterday, you can replay last night’s real data every time you make a change and see whether the system still produces faithful outcomes. Others pointed to GitHub’s Scientist framework, which runs two versions of a service side by side and compares outputs, and to Anthropic’s experiments running a coffee shop with many agents to find the best margin.
The catch is that each agent can still hallucinate, and a simulation is only useful if it stays faithful over time. Everyone agreed it works today. Nobody had a good answer for preventing degradation, and the consensus was that it only works if you massively decompose the system so you are testing one stream of work at a time. Maybe one day.
What Skills For QA in a New World? #
If regression testing is too expensive and E2E is the new regression test, what is the job? The group decided on discovery testing, adversarial testing, and business intent analysis. The new QA is a very creative role. Someone who is trained to break things and gets brought in at the beginning rather than the end.
Several people also argued the role does not shift to developers, it shifts left to product. Hire a product owner as your QA person. Define success criteria in Gherkin, tag each feature, add feature flags, and when a change touches a flagged feature it gets surfaced automatically. All of the expected things can be built today. It is the unexpected ones that need a downstream owner, because the biggest complaint about AI is that it has no taste.
The counterweight came from the more risk-averse side of the table: clients still want to receive code plus tests and have their own QA spin it up. Who watches the watchers? And if you have no QA function at all, how does one engineer know what another engineer is working on? Often, QA is a culture and a communication method.
The Agent Factory Floor Model #
The framing that stuck with me most: software is becoming a factory, and the agents are the workers on the floor. The warehouse model, but for code.
Does it hold up with N agents working at once? Merge conflicts are solvable with worktrees, but there is no consensus on how to orchestrate at the agent level when your marketing team and your finance team are both touching the same component. A CTO gave an example of aircraft design (the fuselage should touch the engine?): a design structure matrix, features on both axes, mapping which components touch each other the way a toilet touches the floor and not the seats. Sequence the build so you never blow up the scope of a feature that depends on three others.
Roughly 60% of the time now goes to PRD generation and sequencing stories, which is exactly the part people want automated. The group even cracked jokes about agents on the org chart. Unit tests, in this model, mostly get generated because customers ask for them.
Testing in Orgs that Move in Real-Life Speed #
Everything above assumes a streamlined shop. Most companies are not that. The most grounded comment of the night was that nothing about our processes has actually changed with LLMs, we are just moving faster through the same ones. Which is why lean management is coming back to the front: which value streams actually deserve the attention?
Two examples anchored the extremes. A Finance company run some of the most complex codebases ever built with a very low defect rate and no QA function. Weekly builds, 1,700 developers. If every developer introduced one bug it would break several times a day, and it does not, because the distance between developer and user is short and every developer owns what they ship. They get the 2am call, they flip the runtime switch, they fix it in the morning. At a medtech company, building cardiovascular systems, QA was mandatory and non negotiable. Same discipline, opposite structure, both correct for their context.
The through line was cycle time. When QA is a downstream function, developers throw code over the wall, QA takes a day, and the feedback lands a week later. That delay is the actual problem. Spending that time on regression is waste. Spending it on discovery and edge cases is the whole point.
Takeaways #
- The testing pyramid is flipping. Business logic stays in unit tests, everything else moves to E2E.
- QA came back, but meaner. Models fix their own tests to pass, so point one AI at another’s output.
- Simulated users work until they drift. Agent farms replay real data fine, but nobody stopped the degradation problem.
- QA is moving to product, not devs. Hire someone creative who breaks things early, because humans have “taste.”
- Cycle time beats coverage. One company has 1,700 devs and no QA. Another requires it. Both work.
Paarth Jamdagneya is the founder of Promptster, focused on monitoring and optimizing engineer AI-enablement
