What if the greatest risk in your enterprise AI strategy isn't hallucinations? What if it's success? Because the moment your employees stop checking AI outputs... the moment managers begin trusting the recommendations... the moment an AI-generated decision becomes part of how your company actually operates... you've crossed a line. And according to COSO's newest 2026 guidance, everything changes after that. Tonight we're discussing the death of stochastic serendipity and why enterprise AI governance just entered a completely new era. Welcome to the show, everyone! I'm Simon Carver, and joining me from his backyard shed in Sydney is Lachlan Reed. G'day, Simon! Great to be here. And we've got a full house today, because we are joined by two absolutely stellar minds to help us unpack this massive compliance shift. First, representing Barcelona, Spain, we have the brilliant workforce strategist and journalist, Sofía Navarro! Hola, Lachlan! It is wonderful to be here. This 2026 COSO guidance is perhaps the most significant governance milestone since ChatGPT launched, and I am thrilled to dive into the human and organizational implications. And we also have Jack Burns, our resident systems thinker, physics mind, and overall voice of calm reason. How's it going, Jack? Going well, Lachlan. This guidance is a long-overdue reality check. Before we pull this apart, a quick reminder to our listeners: if you find this breakdown valuable, make sure to like, share, and subscribe to the podcast on whatever platform you're listening on right now. It really helps us keep these deep dives coming. Yes, hit that subscribe button! Now, let's get into it. Jack, you've been looking at how traditional enterprise systems work versus what Generative AI actually does. What is the fundamental mismatch here? It comes down to basic physics and system architecture, Simon. Traditional enterprise software, like an ERP system, is deterministic. You write a rule, you execute the code, and you get the exact same output every single time. It's a static machine. Generative AI, however, is probabilistic. It doesn't execute rules; it predicts the most likely next word, token, or data point. Yes, and from a strategic workforce perspective, that variability is terrifying for traditional risk officers. Under deterministic systems, you audit the process once or twice a year because the code doesn't magically change its behavior on Tuesday afternoon. But with a probabilistic LLM, you can feed it the exact same contract prompt five times and get five slightly different summaries. Exactly! It's like my old trail bikes. You kick the starter on a modern bike, electronic fuel injection, boom, works every time. You kick an old carbureted dirt bike, and depending on the humidity, the temperature, or how it's feeling, you get a completely different reaction. Traditional controls were built for the fuel injection; they don't know how to handle the carburetor. That is a brilliant analogy, Lachlan. And that brings us to what the 2026 COSO guidance calls the Reliance Threshold. Sofía, how do you define that transition? The Reliance Threshold is the exact moment an AI tool crosses over from being a convenient novelty to an essential operational control. Imagine a financial analyst who uses AI to draft quarterly reports. In month one, they check every line. But by month six, under tight deadlines, they notice the AI has been ninety-five percent accurate. So they start skimming. They trust it. They approve it. Suddenly, the corporate decision-making process is entirely dependent on the model. And once you cross that threshold, COSO is very clear: that AI system is no longer a personal productivity sandbox. It is now part of your internal control environment. That means it must be documented, evaluated, and continually monitored just like any other financial control. You can't just hide behind "oh, it's just a pilot project." Which brings us to what Simon called the "death of stochastic serendipity." Say that three times fast after a couple of beers. But seriously, it means the days of just throwing prompts at a wall, getting lucky with a cool result, and calling it "innovation" are officially dead. We have to move to capability-based governance. Right. Many boards are still asking the wrong questions, like "Do we allow our employees to use ChatGPT or Claude?" COSO says that's irrelevant. The model name doesn't matter. What matters is the capability being performed. Are you using AI for data extraction? Transaction processing? Decision support? Because models change every month, but the business capability—and the risk associated with it—remains. And we have to talk about shadow AI here. Executives think they're safe because they blocked public domains on the office network. Meanwhile, Microsoft, Salesforce, Workday, and SAP have quietly embedded AI features directly into the core tools the company already pays for. These features often activate by default. You are likely already operating outside your stated risk appetite without even knowing it. So how do we prove we're in control? The guidance introduces this fantastic concept called Minimum Viable Evidence. Jack, what does that actually look like in an audit? It means trust is not a control, Simon. If an auditor walks in, you must be able to prove four things: Which specific version of the model was used? Which exact prompt was executed? What data source did it access? And which human ultimately approved the output? If you cannot produce that audit trail, you fail the control. It is a shift from retroactive checking to continuous, automated monitoring. We cannot rely on annual reviews when the underlying LLM updates its weights overnight. Organizations must build systems that validate AI outputs in real time, catching drift and bias as they happen. It's a massive shift, but honestly, it's the only way we make this whole AI transition sustainable. If you want to move fast, you've gotta have brakes you can actually trust. Absolutely. And on that note, we want to hear from you. Have your organizations started mapping these AI Reliance Thresholds yet? Drop us a comment, share this episode with your compliance and tech teams, and don't forget to subscribe. Huge thanks to Sofía Navarro and Jack Burns for bringing their incredible insights today. Thank you, Simon. It was an absolute pleasure. Stay grounded, everyone. Thanks for having us. Catch you all next time! Keep those gears turning.
Audio Coursesby Jellypod