Tested once, trusted for ever? What the Bank of England's AI survey says about assurance
Allen Clark
On Monday 26th October the Bank of England and the FCA published their fourth survey of AI in UK financial services, covering 196 banks, insurers and other regulated firms. Across all respondents, 86% now use AI compared with 75% in the 2024 survey. Among respondents to the full questionnaire, 80% already have AI agents deployed and another 14% plan to deploy them in the next twelve months.
The survey included 105 full-questionnaire responses and 91 short-form responses. Unless otherwise stated, its detailed findings use valid responses to each question in the full survey. The 2024 and 2026 respondent populations differ, so comparisons should be read with that in mind.
Much of the survey is reassuring, but read it alongside the speech the FCA gave the week before and it shows a gap between how carefully firms check their AI before it goes live and how little they check it afterwards. That's where I think the risk now sits.
Cautious with agents, governed on paper
Firms are being careful with agents. Most of the agents in production today support decisions rather than make them, with 80% of firms describing agents that recommend an action while a human decides and executes. Only 2% reported agents with broad autonomy. Even looking a year ahead, no more than 9% of firms expect agents to be approving or declining customer outcomes, placing trades, starting or blocking payments or changing limits and controls.
Governance is in place on paper too, with 94% of firms having firmwide AI policies or standards and 84% running a central AI or model-risk forum.
That's sensible as far as it goes. An agent that only recommends still shapes the decision though, especially when a busy person is approving its recommendations all day long. Agent use is also growing in an area exposed to the outside world, with 16% of firms using agents for automated customer communications today and 34% once planned deployments are included. Anything that talks to customers can be talked to by anyone, including people trying to manipulate it.
Firms are relying on other people's AI and on their word
Third-party AI now accounts for 39% of all the AI systems the firms reported, up from 33%. For the systems firms rated as most important to their business it's 53%, up from 35% in 2024, so more than half of the AI that matters most is now built by someone else.
Firms also understand those bought-in systems far less well. For their most important AI built in-house, 77% say they know in depth how it works. For AI bought from a supplier that falls to 24%, with 81% relying on what the vendor tells them and what's in the contract. Their most common governance headache, named by 54% of firms, was applying their usual model risk checks to AI they hadn't built themselves, while 34% said not being able to see how third-party AI works was a major reason for holding back from using it.
Most of that bought-in AI also comes from the same few suppliers. Just three model providers account for 54% of the supplier relationships firms reported, while three cloud providers account for 79%. The firms rated that dependence on a few critical suppliers as one of the two biggest risks to the financial system as a whole, both now and over the next three years.
A contract tells you what a supplier has promised, not how their model behaves inside your system with your prompts, your data and your customers. You can't see inside someone else's model but you can test how it behaves from the outside. That gives firms independent evidence alongside the supplier's assurances.
Most of the checking happens before go-live
When it comes to managing data risk, more than 90% of firms have a person approve their AI systems, review them before they go live and check out the vendor. Only around half keep monitoring them once they're running, test whether they leak data or review them again at regular intervals.
So most firms check carefully at the door and far fewer keep checking once the system is inside. Checking at the door matters, but it can't establish that a system will remain safe once it's running. Traditional software also needs monitoring and retesting as dependencies, data and threats change. With AI, behaviour can vary between runs, and a supplier can change the underlying model outside the firm's control.
In her speech on 19th October, Jessica Rusu, the FCA's Chief Data, Information and Intelligence Officer, said
"A system that is safe at deployment may behave differently at scale, under stress or as the models beneath it change. Agentic systems therefore require continuous, conditional assurance."
I'd add that it can behave differently from one day to the next even when nothing has changed at all. Full Fact found exactly that this month when they asked five chatbots the same questions every day for ten weeks, which I wrote about in Wrong on Sunday, right on Monday. For third-party models the bit "as the models beneath it change" is largely outside the firm's control, because the vendor can update the model whenever they like.
The firms already know what the risks are
When asked to rank the risks AI poses to them today, firms put data protection and security first and third-party and supply chain risk second. Third came the risks specific to generative AI, which the report spells out as hallucinations, prompt injection or manipulation and data leaking out through prompts and answers. They expect every category of risk to rise over the next one to three years and 88% expect the growing hacking abilities of the most advanced AI models to have a moderate or high impact on their risk over the next two years.
So prompt injection and data leakage are on the risk register, yet only around half of firms are doing ongoing leakage testing, which is the first thing I'd want fixed.
What the regulators now expect
The FCA speech also sets out where supervision is heading. There are three things in it that anyone responsible for AI in a regulated firm should take note of.
The first is incident reporting, where the FCA says its existing requirement to report major incidents
"must extend to significant AI failures and near-misses".
It also notes that the AI Minister has described near-miss reporting for agentic AI as becoming a national security priority, but you can only report a near-miss you've noticed. Security testing can reveal whether a prompt injection could succeed. Detecting an attempted injection in live use also depends on suitable logging, monitoring and investigation. A firm that tests once before launch and then relies on customers to complain is missing both ongoing evidence of its defences and a reliable way to identify near-misses.
The second is accountability, where the FCA wants "senior managers personally accountable for AI systems", with clear controls, human oversight and escalation triggers. If you're the senior manager, you'll want evidence that the system still behaves the way it did when you signed it off, not just a record that it was reviewed once.
The third is resilience, where the speech tells firms to stress-test AI systems, plan for failure, keep fallback options and understand their dependencies on suppliers, including reliance on a single model. With so much of their AI coming from a handful of suppliers, that last point applies to a lot of firms.
Firms with EU operations will hear the same message from Brussels. Under the EU AI Act, banks and insurers can use the governance they already have to cover part of what the Act asks of high-risk AI, so the policies and forums the survey found are the first thing an assessor will look at. The Act also expects high-risk systems to stand up to people deliberately trying to manipulate them, while DORA requires in-scope financial entities to maintain a risk-based digital operational resilience testing programme that covers relevant ICT systems and applications, including those using AI. A policy can't prove either of those things, but testing the system can. For uses like credit scoring and life and health insurance pricing the high-risk rules apply from 2 December 2027, which isn't long when the evidence takes time to build.
What should you do now
These are the questions you should be asking of any firm in this survey.
- Which of our most important AI systems come from a third party and what independent evidence do we have about how they behave? If the honest answer is the vendor's assurance and the contract, start testing them from the outside.
- Are we testing after go-live as well as before? Put the systems that matter on a regular cycle of security testing and test again whenever the vendor changes the model, the prompts change or the system is connected to something new.
- Are we testing for the risks we've put on our own register? If prompt injection and data leakage are in your top three, they should be in your test plan, against the system as it's actually deployed.
- Could we spot a near-miss? Decide what an AI near-miss looks like for each system. Use security testing to check whether attacks could succeed, and make sure live logging, monitoring and investigation can identify attempted misuse and near-misses.
- Can the accountable senior manager show their working? Keep the test evidence, mapped to the controls and frameworks you report against, so the person who signs off can show what was tested, when and with what result.
- What happens to our customer-facing agents when someone deliberately tries to misuse them? Customer communications is a growing use of agents in the survey, so that's where adversarial testing should start.
Where ProbeSix fits
ProbeSix's standard scan targets prompt injection and attempts to leak sensitive information, which sit first and second in the OWASP Top 10 for LLM applications. It runs those attacks against the system a firm has actually deployed rather than against a model in isolation. After go-live you can re-run the same attacks and compare the results with the original scan, so you'll know if a system that passed at launch has started to give way. Every result is mapped to the frameworks regulated firms report against, including the EU AI Act and DORA, which gives the senior manager who signs off something firmer than a one-off review.
The survey shows UK finance taking AI seriously, with careful use of agents and governance in place. The next step is making sure the checking doesn't stop on launch day, because the AI keeps changing after that whether we test it or not.
Sources
- Bank of England and FCA, "Artificial intelligence in UK financial services - 2026", published 26 October 2026: https://www.bankofengland.co.uk/report/2026/artificial-intelligence-in-uk-financial-services-2026
- FCA, Jessica Rusu, "Building the future of finance in the age of AI", speech, 19 October 2026: https://www.fca.org.uk/news/speeches/building-future-finance-age-ai
- Full Fact, "Wrong on Sunday, right on Monday", Public Interest AI Trust Benchmark report one: https://fullfact.org/ai-benchmark/report-one/