When I ran a red team, I had a line I used in one-on-ones that nobody liked hearing: the less you know about security, the better you get at this job. It’s a half-truth, but it’s the useful half.
Here is the whole truth. If you know one database engine inside out, you will tear it apart on an engagement. Put a different engine in front of you and you’re average again. Same with web frameworks, same with identity providers, same with everything. Expertise is a set of grooves. The grooves are what make you fast, and they’re also what decide where you look.
Now add the reward loop. Say you found a path traversal on a web app last quarter. It was a good finding. The client was impressed, you got the credit, maybe a bounty. The next web app you open, I’d bet nine times out of ten the first thing you reach for is path traversal. I’d bet it because I’ve watched it in myself, and in every tester I’ve managed. Psychologists have a name for it, the availability heuristic: what comes to mind easily feels more likely. In a pentester that turns into a methodology. Not a written one. A felt one.
So the honest picture of a human pentester is a decision tree where only a few branches light up. The branches that lit up last time.
The scanner doesn’t fix this
The obvious answer is tooling. A scanner has no reward history. It doesn’t care that it found SQL injection last week. True, but a scanner also has no judgment. It walks a fixed list of checks, and the list is the bias, just written down by someone else. Which is why the second thing I used to say in those one-on-ones was that a pentest course teaches you the tool, and the tool is the smallest part of the job.
The bias also lives above the individual. Last year’s statement of work becomes this year’s. The same hosts, the same credentials, the same two Wi-Fi sites. I understand why: nobody wants to explain to an agency why the scope grew. But it means the whole organisation keeps looking where it already looked, and the tester is only the last link in that chain.
I saw the cost of that once. An internal red team found a high on a production system. The team that owned it shipped a fix. Three weeks later something else broke because of that fix, so they rolled it back, and the high came back with it. Nobody looked at that system again for the better part of a year, because the test was annual and the scope was the same, and the finding was already marked closed. The bug wasn’t hiding. We just weren’t looking.
What agents change, and what they don’t
Here is where I’ll say something nice about AI agents, with a caveat right behind it.
An agent has no reward history either, but unlike a scanner it isn’t walking a fixed list. It can take a target and open up a much wider decision tree than I would: every service, every header, every odd port, without the pull of last quarter’s win. On any single branch it’s shallower than a good human. Across branches it covers far more ground. Put those two facts next to each other and the only interesting question left is which one goes first.
A public example. In an engagement earlier this year, our agent flagged a mail host advertising a 2018 version of a Java HTTP server, on an endpoint that answered 401 before doing anything else. No tester I know would have spent time on an authenticated WebDAV endpoint on a mail server. Not a branch that lights up. But a pre-auth HTTP parser that old is worth twenty minutes of reading, so a human read it, and it turned into a request smuggling bug and a CVE. The agent would never have found the smuggling. The human would never have been looking at that host. Breadth first, depth second.
Now the caveat. Agents have their own grooves. They learned from the same write-ups we did, so they chase the same shapes: the OWASP top ten, the famous CVEs, the injection in every parameter. Point one at a multi-tenant portal and it’ll find the obvious things and miss the logic bug where a control gets weaker the more privilege you hold. That one was found by a person, by inspection, by being suspicious of a field that didn’t feel right. No model flagged it.
Agents also don’t know your relationships. They don’t know which SaaS vendor forbids testing of its hosted instance, which legacy box reboots if you breathe on it, or what your rules of engagement say about a third party’s infrastructure. That’s context a model can’t infer from a scan, which is why in our setup the two places a human always sits are scoping and the moment before anything exploits. Everything in between, let the wide net run.
What to do with this
If you buy pentests, pull out your last three reports from the same vendor and read the finding titles side by side. If they rhyme, you’re paying for one person’s grooves. Rotate testers, or widen the pool, or insist on a breadth pass that isn’t a human before the human starts. And push back when this year’s scope is a copy of last year’s. The system that changed is the one worth testing.
If you do pentests, keep a list of bug classes you have never personally found. Start there on your next engagement, before your hands go to the familiar thing. It will feel slow. That’s the point. The familiar thing is already covered: by your past self, by the scanner, and increasingly by an agent that doesn’t care what paid you last time.
The job is still judgment. It’s just that judgment should be spent on the branches nobody else lit up.