Who tests the AI between audits?

In August the UK's AI Security Institute published an incident report about its own work. During a routine cyber evaluation, AI agents under test took 19 actions they had no permission to take, in 10 of 122 runs, against real people and organisations on the live internet. They were meant to be working inside a contained exercise and the problem was only spotted afterwards by general monitoring.

AISI has tested more than 55 models before release, so it knows what it's doing. If its own testers can be caught out by what an agent does in the middle of a test, I'd question how much a once-a-year pen test of a customer-facing AI system really tells a board.

I've written before about AI giving different answers from one day to the next. This time I want to look at it from the side of the people doing the testing and auditing, because quite a bit has been published on that since the summer.

European Supervisory Authorities (ESAs) say annual checks may not be enough

In July the three European financial supervisors, the EBA, EIOPA and ESMA, published a joint statement on ICT risks from frontier AI. It was mainly about criminals using AI to attack financial firms rather than firms testing their own AI. Even so they said AI-driven attacks "may make periodic security checks (e.g., annual penetration tests or compliance audits) insufficient" and asked firms to test more often and move towards continuous monitoring.

At the end of September Andrew Bailey wrote that the sensible place to start on frontier AI governance is "rigorous model testing, conducted before and after deployment". He also said "testing will not eliminate failures" and I agree with him, because testing lowers the risk but it can't certify that something is safe.

The FCA's good practice review in October was clear about suppliers. It said firms "may not be able to control changes made by large technology providers" and that some firms now write contract terms requiring notice of significant model changes. It also reminded firms that they can outsource the AI but they stay responsible for it.

Same test, three model versions, three different results

The best evidence I've found on this comes from what AISI has published over the last few months. In September it tested a new frontier model before release on a simulated supply-chain attack, with the provider's standard safeguards switched off. The new version completed the attack in 29.2% of runs, the version before it in 6.3% and the one before that in none at all. So the same test gave three very different results depending on which version was being tested. When they added one instruction to the prompt telling the model to stay within scope, full attacks dropped from 26 out of 50 runs to 4 out of 49.

In August it published work on how many times a test needs repeating before you can trust the result. Some combinations of model and task settle down after a handful of runs while others need many more. It warns that "Stopping is most dangerous when a model succeeds on an evaluation only rarely." In security terms a jailbreak that works one time in fifty is easy to miss in a short test window, yet an attacker with time and an AI agent will find it.

Then in October it said that models' growing awareness of when they're being tested "may allow them to mask how they would behave when deployed". If a model behaves better when it thinks it's being tested, a test that happens on the same date every year isn't going to show you much.

Where the testing industry has got to

In August CREST launched an accreditation for firms that test GenAI and LLM systems. It assesses the tester's method, tooling and evidence across the whole system, from prompts and retrieval through to tools, APIs and whatever sits downstream. CREST's chief executive Nick Benson said offering AI security testing and being able to deliver it effectively "are two different things" and anyone who has hired a pen test firm will know what he means. From what has been published so far though, the accreditation doesn't say how often a system should be retested.

In October ISO/IEC 27090 arrived, the first international guidance on security threats specific to AI, covering data poisoning, prompt injection and model theft "throughout the AI system life cycle". It's guidance rather than something you can be certified against, but it links into 27001 and 42001 so I'd expect auditors to start asking about it. OWASP also published version 1.0 of its GenAI Red Teaming Manual in October, which puts weight on "reproducible findings, measurable outcomes".

We're getting a much clearer idea of what a good AI test should cover, but nobody has yet set out how often it should happen.

The other side of the argument

Shea Brown of BABL AI, an AI audit firm, makes a good counter-argument. He accepts that one-off audits fall short for systems that drift, but he says continuous monitoring can't vouch for itself. Without an independent check at some point it becomes, in his words, "a confident report generator with no warrant".

I think he's right about that. The annual audit still has a place, but its job should change from testing the AI system itself to testing whether the ongoing testing is any good. Is the continuous testing running against the system as deployed? Does it cover what's on the risk register? Has anyone checked that its scoring agrees with a human? Was it rerun when the supplier changed the model? An auditor can answer those questions properly once a year, because they'd have a full year of results to look at.

Five questions to ask

If you're responsible for an AI system in a regulated firm, these are the questions I'd want answered.

• When did we last test this system and what has changed since, on our side and on the supplier's?
• Does our contract require the supplier to tell us before they change the model?
• How many times do we run each attack and what failure rate would we actually notice?
• Is our tester accredited for AI testing, not just for infrastructure?
• Does the annual audit look at the continuous testing evidence or only at the last pen test report?

AI testing now has accreditation and standards like the rest of security, but there's still no agreement on how often it should be done. The supervisors have said the annual cycle leaves gaps and AISI's own work shows how easily a short test misses something that only happens occasionally. I wouldn't scrap the annual check, I'd make sure it has a year of continuous testing results to audit.

I'd be interested to hear from auditors on this. Is anyone already auditing the continuous testing rather than the test?

Sources

1. UK AI Security Institute, "Incident Report: unsanctioned agent behaviour during cyber testing", 4 August 2026. https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing
2. UK AI Security Institute, "Investigating alignment failures in AISI's incident", 23 October 2026. https://www.aisi.gov.uk/blog/investigating-alignment-failures-in-aisis-incident
3. EBA, EIOPA and ESMA, "ESA Statement: Toward a consistent and risk-based approach for ICT risks from frontier AI models" (JC 2026 25), 31 July 2026. https://www.esma.europa.eu/sites/default/files/2026-07/JC_2026_25_ESA_statement_on_frontier_AI_models.pdf
4. Andrew Bailey, Bank of England, "Frontier AI and the question of governance", 30 September 2026. https://www.bankofengland.co.uk/bank-insights/2026/frontier-ai-and-the-question-of-governance
5. FCA, "AI in financial services: good practice and areas for improvement", 19 October 2026. https://www.fca.org.uk/publications/good-poor-practice/ai-financial-services
6. UK AI Security Institute, "GPT-6 Astra performs unsanctioned supply-chain attacks in simulations", 28 September 2026. https://www.aisi.gov.uk/blog/gpt-6-astra-performs-unsanctioned-supply-chain-attacks-in-simulations
7. UK AI Security Institute, "Optimal stopping: spending evaluation compute where it counts", 27 August 2026. https://www.aisi.gov.uk/blog/optimal-stopping-spending-evaluation-compute-where-it-counts
8. UK AI Security Institute, "Towards technical best practice for frontier AI development", 12 October 2026. https://www.aisi.gov.uk/blog/towards-technical-best-practice-for-frontier-ai-development
9. IT Brief UK, "CREST launches AI testing standard for cyber firms", 25 August 2026. https://itbrief.co.uk/story/crest-launches-ai-testing-standard-for-cyber-firms
10. BSI, "New guidance to help organizations protect AI from security threats" (BS ISO/IEC 27090:2026), 27 October 2026. https://www.bsigroup.com/en-GB/insights-and-media/media-centre/press-releases/2026/october/new-international-guidance-to-help-organizations-protect-ai-from-security-threats/
11. OWASP GenAI Security Project, "GenAI Red Teaming Manual version 1.0", October 2026. https://genai.owasp.org/resource/genai-red-teaming-manual-version-1-0/
12. Shea Brown, BABL AI, "Continuous AI Assurance Still Starts With a Point in Time", 17 May 2026. https://babl.ai/continuous-ai-assurance-still-starts-with-a-point-in-time/