Banks have handed life-changing decisions to machines. Now the hard part: figuring out who’s accountable when those machines get it wrong
A few years ago, a loan officer would sit across from you, ask a few questions, maybe raise an eyebrow at your spending habits, and make a call. You could argue with that person. You could explain the rough patch on your credit report. Today, that officer is often a model, software trained on millions of past decisions, quietly deciding whether you’re worth the risk. You can’t argue with it. Most of the time, you don’t even know it exists.
ChatGPT Generated Image
That’s the backdrop to a conversation building steam across boardrooms and regulators alike: how do you govern something that decides faster than any human can review it, at a scale nobody can fully audit?
Finance Isn’t Like Other Industries
Plenty of sectors are wrestling with AI oversight right now. But finance carries a particular kind of risk, because money doesn’t stay in one place. It moves, it connects, it cascades. A recommendation algorithm messing up on a shopping app is annoying. A risk model messing up inside a major bank can spread through markets before anyone notices the source.
A few things make finance especially tricky:
Institutions are deeply interconnected, so one bad model at one firm can ripple outward fast.Customers rarely get a real explanation when they’re denied a loan or flagged for fraud, which makes it hard to hold anyone accountable.Trading and underwriting now happen at speeds no human reviewer can realistically keep pace with.
Regulators clearly see this coming. The EU’s AI Act, fresh guidance from the Federal Reserve and OCC, and ongoing work from bodies like the Financial Stability Board are all starting to treat AI as its own category of risk, not an IT upgrade, but something that belongs alongside credit risk and market risk.
So What Does Governance Actually Involve?
It’s not one policy or checklist. It’s closer to a set of habits a bank builds into how it treats every model, from the day it’s built to the day it’s retired. Institutions doing this well tend to focus on a handful of things:
Keeping a real inventory of every model in production, what it does, who owns it, and what it’s allowed to decide.Making sure there’s a human-readable reason behind a rejected loan or a flagged transaction, not just a score nobody can explain.Running regular audits to check whether a model is quietly discriminating against certain groups, even unintentionally.Keeping humans in the loop for decisions with real legal or reputational weight.Watching models after launch, since a system accurate on day one can drift as behavior and markets change.
None of this sounds exciting. It’s closer to fire codes than innovation. But that’s kind of the point, the boring stuff is usually what stops a small glitch from becoming a front-page problem.
The Part Nobody Really Talks About
The trickier issue might not be technical at all. It’s cultural. Data science teams are rewarded for accuracy and speed. Risk teams are trained to ask what could go wrong. For a long time, these were basically two departments that rarely spoke the same language. Good AI governance forces them into the same room, and that’s where things get uncomfortable.
Deployment slows down. A fraud model that looked ready to ship might sit in review for months while someone stress-tests it against every weird scenario they can think of. It’s frustrating short-term. But banks that lean into that friction, instead of fighting it, tend to end up ahead, not behind.
A few examples make it concrete:
A bank that can prove its lending model treats people fairly builds real trust with regulators and customers.A trading firm that can walk an auditor through its algorithm’s decisions avoids a longer, costlier investigation later.An insurer upfront about a model’s blind spots is in better shape if a claim ends up in court.
Seen this way, governance isn’t fighting innovation. It’s the scaffolding that keeps innovation from collapsing on itself.
Where This Goes Next
A few things seem likely to shape the next stretch of this story.
Different countries are writing their own AI rules, and global banks will push for common ground so they’re not juggling a dozen conflicting rule-books.Banks are leaning more on outside vendors and foundation models, so governance can’t stop at the bank’s own walls anymore, it has to stretch across the whole chain of tools involved.Generative AI is creeping from drafting memos into shaping investment advice, blurring the line between “helpful tool” and “thing making real decisions” in ways current rules weren’t built for.
None of this gets solved with one law or one audit. It gets solved the slow, unglamorous way most financial infrastructure gets built, through mistakes, corrections, and repetition.
Where This Leaves Us
AI in finance isn’t going anywhere, and neither is the risk that comes with it. The banks that come out ahead won’t be the ones shipping the flashiest models first. They’ll be the ones that can explain what their models did, defend those decisions, and fix them fast when something goes sideways. Governance isn’t slowing this industry down, it’s the seat-belt that lets it move fast without everything falling apart.
The rules are still being figured out, in real time, by people who don’t have all the answers yet. But the institutions taking this seriously now will probably be the ones still standing the next time something breaks.
Who’s Watching the Algorithm That’s Watching Your Money? was originally published in Coinmonks on Medium, where people are continuing the conversation by highlighting and responding to this story.