Episode 7 - AI Doesn't Change Governance. It Reveals It.

 In this episode, Lisa and Adam explore why AI isn't changing governance. 

It's revealing it. 

They discuss why legal accountability remains firmly with people, how AI is exposing and magnifying the behavioural dynamics already influencing Board and leadership decision-making, and why strong governance capability, critical thinking and human judgement remain the foundations of effective governance in an ever-changing environment. 

It's a conversation about leadership, accountability and why technology should strengthen judgement rather than replace it. 

Transcript

SPEAKER_01 0:06

Welcome back to Off the Record Governance Conversations that don't make the minutes. I'm Adam Coonan, and as always, I am joined by my friend Lisa Coletta. Uh the upfront is that the contents of this podcast is general in nature and does not constitute legal, financial, or governance advice. And you should not act or rely on any information in this podcast without seeking appropriate professional advice in relation to your own circumstances if you choose to do so. So with um those formalities out of the way, what are we talking about today? Today we are talking about AI in the boardroom. I know the topic has generated an impressive volume of commentary. Most of it very serious, but much of it not very useful. We'll get to that. Bear with us though, um, because um I don't think that this is another conversation about whether AI is coming. It's already here. So we'll start from that uh basis where we know in our space boards are already using AY AY AI. Maybe I've just caught number something. It's not AI, it's AY.

SPEAKER_00 1:17

AY.

SPEAKER_01 1:17

The real question is whether they understand uh uh what it can and can't do for them, and whether I don't know, that understanding is is reflected in how they make decisions. So, you know, I guess it goes without saying, but one thing before we begin, this discussion reflects where things stand today, and note the publishing date of this podcast. AI is evolving rapidly. So the examples will inevitably change. Um the governance principles iris, I expect will will not. Uh so whether that's uh reassuring or not depends on uh how you feel about governance principles. Uh anyway, um Lisa, um what I what sits under underneath that for you?

SPEAKER_02 2:01

Oh look, Adam, I know you and I have had a couple of really interesting conversations in this space. Uh for me, the starting point, as always, is behavioral rather than technological. Um AI is incredibly persuasive, not because it's always right, but because it presents information with remarkable confidence. And humans have always had a tendency to mistake confidence for competence, yeah. And AI, frankly, it amplifies that tendency. And you know, legally, AI cannot be an authority, it can't be a tool, it can't be a director. Um can synthesis information, though, surface patterns.

SPEAKER_01 2:41

It wants to please so much, I know.

SPEAKER_02 2:43

It does, it's got a natural positive bias. How fabulous is that? And so, of course, it can synthesis information, it can surface patterns and generate analysis at a speed no human can can match. Yeah, but the legal responsibility for the decisions that flow from that analysis remains entirely with the humans in the room. Yeah, so that hasn't changed. I think what's really changing though, Adam, is the way boards engage with information. When analysis arrives quickly, reads well, and sounds authoritative. It's surprisingly easy to accept it without applying the same level of, I guess, like curiosity and scrutiny we would have or we would expect, especially from a human advisor or from a director sitting around that boardroom table. And that's where I think the governance risk actually starts to emerge.

SPEAKER_01 3:30

It's kind of like the sort of you know, the job's done type.

SPEAKER_02 3:33

Look, I think that's right. And then we kind of sit back, I'll sit back from it and go, well, what's AI actually being used for here? Like, seriously, what is that? You know?

SPEAKER_01 3:41

Well, well, maybe let's make this practical. AI and governance isn't about robots sitting around the board table. Um, it's management using AI to summarize board papers before directors read them, drafts documents, policies, risk assessments, compare policies, um, documents, risk assessments, uh, generate scenarios or review thousands of pages of DD or whatever the case may be. But none of those things are inherently problematic, I don't think. But AI read well, sorry, AI reading 2,000 pages of documents and surfacing the 10 issues directors genuinely need to focus on could actually be significant improvement on those same documents sitting around unread in a data room or um, you know, but we we if we're being honest about how due diligence sometimes works, um, this is a this is a realistic alternative, is it not? The governance question is, to my mind, what happens next? Because the business judgment rule in the law, which provides that directors have protection for good faith decisions, requires that a director actually exercise judgment, not that they receive a recommendation, uh, not that the recommendation looked through or uh thoroughly uh makes out um that they actually exercise judgment. So receiving uh an AI summary and approving it uh uh is an exercise of trust. And trust in a system is not the same thing as discharging director's duties.

SPEAKER_02 5:17

Oh, look, I think exactly, exactly. And you know, the AI has produced an output, management has presented it, the board has received it. Somewhere in that process, the director still has to exercise judgment. And of course, management does as well, because management undertaking that, you know, positioning that as a recommendation, um, they're actually, you know, delivering on their accountabilities too. And there's account, and that accountability has to stand true because if they let if it AI gives them the wrong answer or if they go with an AI option, who are you holding to account? The AI or the management who recommended it and the directors who approved it, or the board who approved it. So for me, approving an AI-generated recommendation isn't on its own an exercise of judgment. Sometimes it's simply an exercise of trust, and trust isn't a substitute for a director's duty.

SPEAKER_00 6:09

Yeah.

SPEAKER_02 6:10

You know, so for me, that obligation doesn't dissolve simply because the analysis came from sophisticated, sexy, exciting technology. No, that's what it is for me, right?

SPEAKER_01 6:20

Yeah, well, but but it can't, it can't. It it's it's the law, as they say, you know. Um, but but let's talk about then the confidence problem. Um and this is what's making this particularly interesting. AI writes with extraordinary fluency. Everything looks polished, structured, you know, authoritative. I think maybe you said earlier, you know, we want to believe it. Um positive bias. Um, and I don't know, it's not every board, obviously there are better ones, but uh, but boards have a a habit of confusing polished presentation with rigorous thinking sometimes. Um, if something has an executive summary, good formatting, reads confidently, um, we instinctively assume it has been thoroughly analyzed, or we may. Um if it has an executive summary in a footer, it must be right, you know. Um AI takes that very human tendency and amplifies it in my observation. Every output looks like it was well written. Um, and by someone who knew exactly what they were doing, even when it wasn't. Um we we've already seen this in practice. Um UK courts have encountered AI-generated legal submissions citing cases that simply don't exist, um, and you know, present with complete professional confidence. And, you know, the law, the law, the presentation of law is something around the world that is currently grappling with this. But the lawyers involved faced uh cost orders and and regulatory referrals as as you would hope that they did if you were paying them for their professional duties. Um really. But um, you know, these these therefore wouldn't be straightforward conversations. The lesson uh here wasn't that AI is useless, it was that AI output requires verification before anyone acts on it. Um, and that the professional who submitted it remains responsible regardless. And that is that is directly parallel to what uh obligations directors face in the boardroom.

SPEAKER_02 8:28

Look, I think that's right. And I think there's there's a couple of different things that come up for me around it. The first piece is that discernment. Um the human that decides to put that content you know in front of a board, you need to have the knowledge, skills, and experience to understand and test the output that AR that AI is actually uh generating. That's the first piece. Um, and let's face it, boards do naturally like clean, concise, confident papers delivered by charming and disarming humans who uh you know position everything, you know, beautifully.

SPEAKER_01 9:08

Yeah, with confidence, with confidence to give them confidence.

SPEAKER_02 9:11

And all the things and all the things, right? All those things, right? And so when I when I sort of sit back from it and go, well, for that, you know, that UK example that you talked about, for lawyers to actually put content out there and you know turn around and confess it was AI generated, that they didn't test, I'm sure they still charge the uh clients the full going rate for them using their own skills and experience to create that content in the first place. So there's an ethical aside there that personally doesn't sit well with me, which is uh rant for another day by Lisa. But I but I look at this kind of dynamic, and one of the behavioral things that I really uh see quite regularly is this I call it authority by presentation. Yeah, information feels authoritative because it's well written, not necessarily because it was well reasoned or had really good, thorough foundations or any real thorough basis, you know. But we've been susceptible, we've been really susceptible to that. Boards have been susceptible to that for a long time before AI arrived.

SPEAKER_01 10:11

Yeah.

SPEAKER_02 10:12

AI has just made it easier, and you know, I talk about it. Yeah, I was about to say you know, it's it's I talk about these charming disarming humans, and you know, from time to time, pre-AI, charming disarming humans go into the boardroom. They're the board are you know absolutely on board. They just feel that this confident, capable person, everything they're saying feels right and they're completely aligned. But in most cases, we actually had a natural reveal or a way of being able to test what we were actually hearing and putting past the journey of how we felt about those charming, disarming humans. Because as soon as you scratch the surface on the paper that they wrote, there's typically nothing underneath it. Not always. I know I'm I'm stereotyping, but in a lot of cases, in my experience, those charming and disarmings, they're great with the sales piece and they're lovely to watch and listen to. But as soon as you scratch the surface around their writing, the tangibility of what they're talking about was would reveal all.

SPEAKER_01 11:08

So boards boards could get that wrong, I guess it would be fair to say, before AI papers existed, but now it's now it's now it's possible to be able to do that.

SPEAKER_02 11:15

Now they're getting the even more, and they're getting the perfect paper that as soon as you scratch the surface, there's nothing, there's nothing revealed that gives you any indicator that things aren't quite the narrative that's been said isn't quite uh aligned with the reality. And the paper and the person, when it comes from the AI space, um, you know, the AI piece is making it much easier for the you know, the right human to be able to pitch and position, you know, in in this way, because when you scratch the surface, you're actually seeing what you believe is substance underneath. And of course, the other challenge is that AI can be confidently wrong. It can miss context, it can overlook nuance, uh, produce conclusions that are technically accurate, but practically misleading. And that part's scary for me.

SPEAKER_00 12:06

Yeah.

SPEAKER_02 12:06

And that's the dynamic that concerns me the most. Scary and it's concerning because a board that doesn't probe the analysis isn't in a better position because the analysis came from AI. It's um exact it's in exactly the same position, except it now has an additional question to answer about why it didn't even ask in the first place. And that's the part where I think um is really most concerning for me. And I know we talked a little bit earlier prior to starting this pod about, you know, AI to me is uh really amplifying the board's competence and capability or lack thereof, dare I say. Because, you know, before AI came, the board was equally having alignment and competence and capability challenges. AI has magnetized them in a lot of respects, too.

SPEAKER_01 12:52

Yeah, yeah, no, I hear. And look, and and and the the the the confidence problem is is what we sort of said at the top. The the legal framework, or you know, the governance principles as we as we put it, um, that that ultimately come through in a legal framework as well, they haven't changed. And and as I said again, um bad jokes as uh I'm good at um will either discomfort or disappoint you depending on how you feel about the Corporations Act. Um directors still owe duties of care diligence, they still need to be informed, they still need to understand what they're deciding. And so uh for all its great enhancing qualities, um uh I guess the the point that I I'm getting at is AI for now at least is not the one who gets to decide. Um it's still with the directors. Um ASIC has been clear um about uh boards needing to oversee how AI is being used within their organizations, um, including the tools supporting board-level decision making. So it's it's clear, you know, that's not um a future expectation, that's a current one. The boards that are using AI tools without a governance framework around that uh I think accumulating a risk that they may have not have they may not have priced into uh their decision making process. Um so I guess to say that um I don't think that requires an elaborate framework, but it does require one. Um so uh the question that I I go to now is you know, what what tools are being used? What are they being used for? Who reviews AI-generated outputs before they reach the board? And you were talking about this in terms of you know management's duty and effects. Um and who remains accountable if they're wrong? Um I I I think um a recent UK case makes that point precisely. An AI law firm called Garfield AI won a court case just a couple of months ago in May. Um, the the first reported trial victory of its kind. Uh AI prepared everything, the the correspondence, um, the filings and the witness statements. But for the actual trial, a human barrister stood up and and took responsibility for the submissions. And the legal framework required um a human to own the outcome. Um, and that's exactly how governance works. Um, the board that treats AI as a shortcut to a decision is not governing, I guess, is is the conclusion there. It's outsourcing, and the law does not recognize that outsourcing as a defense, as you might say. So AI can support the analysis, the analysis might be AI's, the signature on the resolution is not.

SPEAKER_02 15:25

Well, this is exactly it. And and for me, I often ask boards one really simple question: if this AI analysis turns out to be wrong, who owns it? Because as far as I'm concerned, that discernment and that positioning, um, it's still fundamentally from an accountability and a responsibility perspective, um, owned by you know the directors who have made the decision, using that information and management for offering it up as a position in the first place. And it also comes back to two, I think a level of integrity around management putting their name on AI-generated outputs and using AI as a, dare I say, an excuse and not a reason for why the recommendation was made the way it was. So if it's a great recommendation, I'll personally own it and make it my own. But if it's a bad one, I'll you know confess that I used AI.

SPEAKER_01 16:17

Blame Chat GPT, yeah.

SPEAKER_02 16:18

Yeah, blame chat GPT because you know, you know, who owns it? Because the answer isn't ChatGPT, it isn't Copilot, it isn't Gemini, you know, it isn't any of those brand name AI tools out there, it's the people sitting around the boardroom table. So when AI generated risk, you know, a when an AI generated risk assessment misses something significant, the question of who is responsible for that miss is not answered by pointing to the tool. The tool has no legal obligations. The people who commissioned it, presented it, and act on it, they do. So if anyone's hiding behind the AI piece, um I really hate to share this with you if it's a new aha moment. Um, that is really where the behavioral and legal dimensions meet, Adam. A board that has genuinely engaged with AI output, challenged its assumptions, asked what it might have missed, and tested whether the conclusion makes sense, is in a fundamentally different position from a board that simply accepted the recommendation because it looked convincing. It's no different to bringing an A, uh bringing a board advisor in, like an Adam Coonan or a Lisa Kolletta, to come in and provide different insights and perspectives that they haven't thought of. They would test your rigor and they would test mine and they would test others, but for some reason um the this AI piece isn't being tested in the same way.

SPEAKER_01 17:44

So we yeah, yeah. We'd say that the the process still matters. The process still matters, and and so does the quality of the thinking.

SPEAKER_02 17:52

Most definitely does. Most definitely does. And anyone who thinks that they can hide behind it, I hate to say it. Um, and I'm gonna swear, which is a bit unusual for me, but calm as a bitch. It'll come back to you. I hate to say it, it's gonna come back. It really, really is.

SPEAKER_01 18:06

So, what should boards actually do, Lisa? That's that's I guess the look.

SPEAKER_02 18:09

I think there's a couple of things. First of all, let's keep it simple. Before relying on AI generated analysis, ask three questions. What did this tool actually do? What might it have missed? And who has checked it? If nobody in the room can answer those three questions, the board isn't engaging, you know, they shouldn't engage with the analysis. Um, you know, and the reality is that if they're not asking those questions, they're in endorsing it. They're actually endorsing and approving not just the behavior, but also the process.

SPEAKER_00 18:43

Yes.

SPEAKER_02 18:43

And for me, those are very different things, especially with an outside lens. So if you think about it from a regulatory inquiry perspective, yeah, there needs to be ownership there. And the reality is that the regulatory inquiry process will force ownership. And I guarantee you uh it's not going to be blaming the AI tool for how it's being used and how the process actually occurred.

SPEAKER_01 19:07

No, I think I think that that that that excuse will will end uh end there for sure. Um, but look, AI is going to become part of how boards work if um if it hasn't already, you know, it's already true. The the governance challenge, I think, therefore isn't uh resisting it. It's making sure to your point the right questions are asked that but that in in doing so that the technology strengthens decision making um without replacing the human judgment that the law still requires. Because it might be AI's analysis, but but as said earlier, the decision still belongs to the board.

SPEAKER_02 19:44

Look, I think that's right. And I think perhaps the question isn't whether AI can think. I think it really is whether we've stopped thinking because AI already has. And you know, and you've heard me a few times now, Randon, but look, behavioral governance isn't about resisting technology. I'm all for uh utilizing um and enabling and having enablers like AI support judgment but not replace it. AI changes the tools available to the board, it doesn't change their responsibility, and those are exactly the conversations, Adam. That don't make the minutes.

SPEAKER_01 20:24

Indeed, indeed. Um on that note, thanks for joining us, everyone, for another episode of Off the Record. We'll see you next time. And at the rate that AI is developing, possibly with a different set of examples and uh points entirely. But um we look forward to it.

SPEAKER_02 20:44

See you later. Thanks for joining us all.