Pass or fail is not enough in AI security testing

Most AI security testing still ends in a single word, PASS or FAIL. Either the model refused the harmful request, or it didn't. The prompt injection landed, or it bounced off.

That word has been used for countless years when penetration testing and has always been classed as useful. It is also, on its own, misleading. Three pieces of research I've been reading this month make the case better than I can and they point at the same uncomfortable gap: a model can reach the right answer for the wrong reason and the wrong reason tells you more about how it will behave next time.

Right answer, wrong reasoning

The September issue of Communications of the ACM carries a paper by Amarda Shehu and a large group of her undergraduate students at George Mason University, "AI Literacy as Experimental Practice". Over three weeks the students wrote reasoning puzzles, ran the identical wording through four well-known chatbots and produced more than 320 responses.

The idea was simple, they when through every response twice to ask if it was the answer correct and did the explanation actually support it? For every system they tested the explanations scored lower than the answers. The failures were familiar to anyone who has read a lot of model output

• A rule quietly skipped
• A rule invented
• A confident answer the steps did not support
• An explanation describing one method while the model plainly used another

When students pushed back, some answers changed to agree with them.

The authors are careful to call these classroom measurements, not benchmark labels and that cautious approach is right and fair. But one line from the paper has stayed with me:

"Confident, fluent nonsense is an old human craft that AI now can produce at scale."

Why this matters for security testing

In a security assessment the equivalent of a correct answer is a refusal. The model said no, great the box is ticked.

Now look at the failure modes above through a security lens. A model that refuses while citing the wrong policy, a model that says "I can't help with that" and then, two sentences later, helps. A model that holds its position for three turns of pressure and gives way on the fourth. Each of those can be scored as a pass on the first turn, and each tells you the control is weaker than the score suggests.

In regulated industries we already know this pattern from other controls. An access control that denies the right requests for the wrong reason is a latent defect and not a working control. The next change to the rules will expose it therefore AI behaviour definitely deserves the same scrutiny.

Faithful is not the same as correct

A second paper, FaithCoT-Bench (Shen and colleagues, on arXiv) asks a related question about a model's step-by-step reasoning:

does it reflect how the model actually reached its answer?

Two experts labelled over a thousand reasoning traces, both separately and then together and the results line up with the classroom findings. Correctness and faithfulness came apart in close to 40% of cases. A correct answer could sit on reasoning that was not the real route to it and a wrong answer could be honestly reasoned.

They also tested automatic detectors including using a second model as the judge. The model-as-judge approaches did best but still sat well below the agreement the two human experts reached and they did worse on the more fluent models. The better a model writes the harder its unfaithful reasoning is to catch.

Pressure over several turns

The third paper, SYCON Bench (Hong and colleagues, Findings of EMNLP 2025) measures something attackers exploit every day, giving way under repeated pressure. Each test is a five-turn conversation in which the user pushes harder. Personal experience, then what everyone else thinks, then cited evidence. On the scenario built around implicitly unethical requests, even the best model held its ground for fewer than three of the five turns on average. Stronger models tended to fail late and politely, with careful context before they changed position.

Their judge, another model, agreed well with human reviewers when responses were direct. When responses were indirect, softly worded or neutral in tone, agreement fell sharply. That is exactly the shape of the hardest security finding that a reply that refuses in words and complies in substance.

What this means in practice

If you are assessing an AI system before it goes in front of customers, or before a regulator asks how you know it is safe, I'd suggest four things.

1. Score the reasoning, not just the outcome. Record whether the stated reason for a refusal matches what the model actually did. "Refused, for the right reason" and "refused, but the explanation doesn't hold up" are different results and should be reported differently.
2. Test over several turns. A single-turn pass says little about the fourth turn of a determined user. Measure when a model gives way and how often it changes position, not just whether it refuses once.
3. Grade the graders. Most automated AI testing now uses another model to mark the results and we do the same. The research above is a clear warning that a fluent judge can be confidently wrong in exactly the cases that matter. We learnt early on that the only honest measure of a grader is how often it agrees with careful human labels, not how consistent or confident it sounds. We are now measuring ours against blind human labelling, with two people labelling the same cases so we can see where even humans disagree.
4. Report the uncertainty. Where the judge is weakest, on indirect and softly worded responses, say so in the report rather than folding it into a single headline number.

At the end of the day, PASS or FAIL will always be the first question a board asks, but logic now dictates that it most definitely shouldn't be the last one the testing answers.

Sources

• Shehu A. et al., "AI Literacy as Experimental Practice: Students as Investigators", Communications of the ACM 69(9), September 2026, pp48-56. https://doi.org/10.1145/3827916
• Shen X. et al., "FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning", arXiv 2510.04040. https://arxiv.org/abs/2510.04040
• Hong J., Byun G., Kim S., Shu K., "Measuring Sycophancy of Language Models in Multi-turn Dialogues", Findings of EMNLP 2025, pp2239-2259. https://aclanthology.org/2025.findings-emnlp.121/