By Chuck Gallagher — Business Ethics Keynote Speaker and Trainer
TL;DR: Chuck Gallagher, AI ethics speaker and author, argues that most companies can already name their AI risks and still cannot stop them — because a named nightmare without resources and training is just a better-informed way to fail.
The agent had been running six weeks and nobody had looked at it, which was more or less the point. Requisitions came in, it checked them against the approved vendor list, matched budget codes, and pushed them through at about eleven hundred a week.
Then a vendor changed its remit-to address. The agent updated the record — updating records was inside its authority — and it kept paying.
Four weeks of payments to an address that belonged to a man in another country who had sent one very well-written email.
That company had named this nightmare. It was on a slide, presented in March.
If Companies Can Name the Risk, Why Can’t They Stop It?
Part One of this series covered the first of Reid Blackman’s three questions, from his 2026 book and his May 2026 Harvard Business Review article: what are the ethical nightmares of your organization as they pertain to AI? The other two are where companies fall down. What resources will you build to avoid those nightmares? How will you train your people to use those resources effectively?
In McKinsey’s 2026 AI Trust Maturity Survey, 74 percent of the roughly 500 respondents flagged inaccuracy as a highly relevant AI risk and 72 percent flagged cybersecurity. So it isn’t ignorance.
The slide was right. Somebody built it, somebody presented it, and the payments went out anyway.
What Counts as a Resource, and Why Isn’t a Policy One?
Blackman’s word for what most companies have instead is compliance theater — policies that gather dust. His prescription is to stop writing them and start building cross-functional teams doing collaborative problem-solving around real outcomes, working the same method at the enterprise level, the department level, and the project level, in one language at every tier.
Not long ago, a participant in a presentation I made to the Montana Society of CPAs asked: other than you just not making the choice you did, what one thing could have prevented you from making the unethical, illegal choice you made?
My response: “Two signatures on the trust check!”
I had full access to trust funds with no oversight. Stop! You may want to reread that last sentence. I had full access to trust funds with no oversight. If the trust checks had required two signatures to be valid — most will recognize that as a simple internal control — then I would have had no access to the funds and hence been denied the opportunity.
That agent could change a payment address by itself. Nobody had to sign anything. Perhaps that sounds like a technology problem, but it’s the same problem I had, and mine involved a checkbook.
Why Does Training Keep Showing Up as the Bottleneck?
In the same survey, nearly 60 percent named knowledge and training gaps as the primary barrier to implementing responsible AI, up from about 50 percent the year before.
Removing opportunity isn’t rocket science! All it requires, for the most part, is the use of God-given common sense. Yet, you have no idea the number of organizations I’m blessed to consult with and speak to who pay me with a check requiring only one signature.
They fly me in. I stand up in front of their people and tell them how I stole from a children’s education trust because nobody ever made me get a second signature on anything. Then somebody in accounting cuts a check for the engagement, and one person signs it.
Who Owns This When It Goes Sideways?
Organizations with explicit ownership for responsible AI — an actual accountable function, an AI governance role or internal audit and ethics — averaged 2.6 on McKinsey’s four-level maturity scale. Organizations without one averaged 1.8.
Some time back I was brought in to speak to a construction company, and before the event I asked the CEO, more or less as small talk, whether they did any business in China. You’d think that I’d have done my homework better. “We decided long ago that it wasn’t ethical,” he said. To do what they do in China would require paying off the officials who issue the permits, and that’s a violation of the Foreign Corrupt Practices Act. “And we don’t want to break the law!”
Then he told me the other half of it. If he asked a person to go win contracts in China, where you have to pay to play, he would be asking that person to break the law and violate the company’s ethical principles. That creates undue pressure for an employee and places them in the untenable position of having to do something wrong to meet the company demands. “That’s just not right!” he said. So that was why I was there — to reinforce that message.
He was the first CEO I’d met who was that articulate about his beliefs and the extent he was willing to go to keep his employees between the ethical lines, even at the cost of the company’s business abroad. It seems that’s what 2.6 looks like when there’s a person attached to it.
What Does Skipping This Actually Cost?
Blackman names three ways this ends badly once generative and agentic systems start making decisions: the deliberation problem, automation bias, and what he calls the ethical flash crash.
The man who approved those payments would tell you he reviewed them. He’d pass a polygraph. He looked at a screen where the answer was already filled in, and his brain did what brains do.
So more human review isn’t the resource. Review with something to push against is — a second signature, a dollar threshold, a change type that nothing can self-approve, human or otherwise.
Frequently Asked Questions
What are the three questions in Reid Blackman’s Ethical Nightmare Challenge?
What are the ethical nightmares of your organization as they pertain to AI? What resources will you build to avoid those nightmares? How will you train your people to use those resources effectively? Blackman, founder and CEO of the AI ethical risk consultancy Virtue, laid the framework out in his 2026 book and in Harvard Business Review in May 2026. His word for what most companies have instead is compliance theater.
What is the difference between an AI policy and an AI resource?
When someone at the Montana Society of CPAs asked me what one thing could have prevented my crime, my answer was two signatures on the trust check. I was a CPA and I knew the code of professional conduct, and the code did not require a second signature on anything. An authority ceiling on an agent is the same idea, and so is a threshold that trips, or a named human who gets the call.
Why is AI training failing inside companies?
Organizations bring me in to talk about internal controls and then pay me with a check requiring one signature. That happens after the talk. McKinsey’s 2026 AI Trust Maturity Survey found nearly 60 percent of respondents cite knowledge and training gaps as the top barrier to implementing responsible AI, up from roughly 50 percent the year before.
Does having someone accountable for AI governance actually matter?
The McKinsey data says yes, and by a wide margin. Organizations with explicit ownership for responsible AI averaged 2.6 on the firm’s four-level maturity scale, while organizations without a clearly accountable function averaged 1.8. The construction company CEO above had decided the China question long before I ever showed up, and he could tell me exactly why.
What is automation bias and why does it matter for AI agents?
The man who approved four weeks of payments to a stranger would tell you he reviewed them. That is automation bias — deferring to a machine’s output while sincerely believing you evaluated it independently. It is one of three failure modes Blackman flags for generative and agentic AI, alongside the deliberation problem and the ethical flash crash. Human-in-the-loop review is the control most companies rely on.
Before You Go
Most organizations I meet aren’t ignorant. They’re stalled. They can tell me exactly what could go wrong and they can’t tell me who would catch it, what would stop it, or whether anybody has practiced. That’s the work I do with boards and leadership teams — sitting in the room and working the scenario until the resource and the training are real, rather than documented. It’s worth having before the four weeks of payments. Start the conversation at ChuckGallagher.com.
Five Questions for Reflection
- If you had to name the one control that would stop your organization’s worst AI scenario — the actual thing, not the policy — what is it, and does it exist yet?
- Who is the identifiable human accountable for AI risk in your company?
- Has anyone in your organization ever practiced responding to an AI incident, or have they only read about it?
- Where in your operation does a human sign off on something a machine already decided? How would you know if that review is real?
- You have named a risk and not built the control. If that risk lands, how does that decision look in a deposition?
