Fix the Car First

All right. Hello, everyone. I'm here with Brad Kaufman, longtime colleague and a friend of mine to talk all things AI responsibility, what goes right, what goes wrong, and very glad to see what comes in this discussion. Hey, Brad, how are you doing today?

Doing really well, Tom. How are you doing?

I'm doing great. I'm doing great. So what's on your mind?

What's not on my mind, especially as we talk about AI, but. You know, you've talked in the past and you've written extensively in the past about empire building and now your your thesis and what we talked about too is it's important to what we're seeing today in organizations with AI with fear with other aspects. Just give me the background on this and how we got to what you're thinking in terms of today in empire building. I'd love to have this start off that way.

All right. No, that sounds great. And it's a disturbing phenomenon that we're seeing. So let's go into the Wayback Machine back to 2011. So my first book, Breaking the Fear Barrier, talked about a progression that organizations go through when there's fear of loss, not meeting a goal, not performing, not getting that bonus, losing recognition or status, whatever it is, people will build walls to avoid that loss. And it starts with parochialism, which is a narrowing of focus. My way or the highway, period, no exceptions. So that may make it easier for those inside the side of the boat to get done what they need to, but not those outside. It's when they define the world by the piece and not the puzzle. So in the face of that, other people who are trying to get their piece of the puzzle done start to feel like, OK, well, I have to really hang on tightly to what I can control. And that leads to the second layer, which is territorialism. And in territorialism, you're starting controlling over information, which, of course, has a big impact on AI. That's maybe a topic for another day. But empowerment becomes imbalanced with accountability, those sorts of things. And the worst part of the pyramid, where you really have costly, costly problems, and the most difficult to undo is the top of it, which is empire building. When someone feels their self-sufficiency is threatened, then they're going to be very tempted to say, you know what, we can't wait for HR to do this training for us. We're going to do our own training. Or we can't wait for legal to give their opinion on this. I need an answer now. we're going to hire our own lawyer. I had one call center client where this got so bad, there were literally seven different groups making changes to their IVR system at the same time and repeatedly crashing it. So when you have empire building, you have organizations that are trying to do the function of another department and what really becomes a problem is the responsibility for that function stays where it's supposed to be, even if other people are doing their work. So now let's fast forward to 2026. AI is everywhere, it's coming out. So I am sitting in my office thinking, I have got to get this contract out, like now. And I've got legal telling me, well, I'll get to it next week. That I'm going to lose the customer. So that's threatening my self-sufficiency, right? But now I have my friend Claude and I could fire up my friend Claude and say, hey, Claude, here's an old contract, take it, modify it, here are the new specs. Claude says, yes, sir, happy to, here you go. And it gets sent out. The problem is the person on the hook or the legal implications of that contract isn't me. It's legal. So they're gonna have to live with whatever decisions, Claude made it. I'm not a lawyer. I'm just going to assume, yeah, okay, that's right. And yeah, there's a little fine print at the bottom about make sure a lawyer deletes that. I don't have to read this. I'm just going to delete that and send it. So it's not just legal stuff. It's research. It's data. It's all sorts of things that other people in the department may be doing in parallel, maybe on the hook for. There may be other parts of that that I just don't have awareness of. But when you start having all these duplicate functions, you start having duplicate answers to the same question, you start having a decision that ends up going as a fork in the road, depending on where you are in the organization, and that creates chaos.

And that, it's what I've seen too, and I've done lots of AI strategy, I've done enterprise tech strategy, and data strategy for organizations, but a lot of what you say resonates, and we've talked about this, and it's absolutely true. But we're also moving at a different pace right now. We've got the FOMO in the organization. You will use AI. You've got organizations coming down from the board. You will use AI and derive value from AI, and come back to revenue and make things happen, and you'll do this now too. So when you start mixing things together, and we've got to cook things faster, we've got to deliver the results faster, but you've also got more and more data. So one of the things, and I've talked to companies about this, but the old tools are not really as workable as they were in the past. You've got, you know, what amounts to 20th century tools, which sometimes feel like 19th century tools, working with 21st century AI. I've got more and more data. You can see this, one of the leading indicators I see in organizations, how much training are you doing? If you're doing a lot of training, You're playing these virtual preseason games and then you're putting the team on the field and make this work. Now they've got more and more data, but where are they supposed to put it? Do they have the right harness? Do they have the right systems in place? Do they have SharePoint? Where they're supposed to store things? Are they sure seeing things stored locally? What happens? How do those dynamics play out? So the end to end process. it sometimes feels that nobody's looking at this, and this resonates with exactly what you're talking about, where now I'm breaking down some of the walls. I'm trying to look at legal, and it's the analog of somebody in the organization vibe coding an app and saying, this is ready for production, and your tech people say, well, did you run this? It's a supply chain attack on this. What libraries did you use? So it gets interesting really fast, but I think we're talking about similar things here.

Oh, absolutely. Absolutely. Let's talk about data for a minute. I've always said that most organizations are data rich and information poor. Right now, there's an interesting twist on that because now you have pockets of data and it's not always shared, it's not always perfect, and it's being thrown into AI, which is going to base its decision on whatever data partial or correct or incorrect that's given. Often, the data that the AI is bad, I've seen doesn't always take into account the second, third, fourth order of facts. It's very narrowly tasked. What that means then is the information that's created from the data is skewed. I would say now it's not so much that you're data rich and information poor. I say it's data rich, but then you have shadow realities that are being created of information that may be partially true, but incomplete, but it's assumed to be gospel. So then you walk into this training piece of it and you're training assuming the answer's correct, right? You're training assuming your organization is prepared, that is aligned, that people aren't gonna be so fearful they're going to hoard bits of the information, that this empire building isn't going to happen, that everything needs to be coordinated and people are going to be doing what they're supposed to do. That's not reality. If you have the best AI system in the world, and the best training in the world, and you plunk that down into a dysfunctional organization without fixing the dysfunction, guess what? It's still going to be dysfunctional. It's just going to be faster.

There's something I've talked about, advised on actually built mechanisms to be able to do this. but the whole idea of source of truth and data. There was a popular way of doing things, approach to doing things in organizations where we're going to centralize data. We've got lots of different systems. We've got Salesforce over here. We've got our ERP system over here, distributed applications, data all over the place. Organizations found that there's disagreement. And you can test for a lot of this, right? But now we've got a different way of working. We're throwing AI into the mix and the whole idea of determinism, code, versus non-determinism, and treating AI like it's deterministic and expecting the same test cycle and the same things to work. And yes, you will deliver by the end of Q3. That mixing this all together doesn't always feel like a recipe for success. But I think what you see is where organizations get nervous and look at dependencies. Okay, regulatory compliance. I need you to answer this question now. Legal, I need you to advise us on what we do here. It's easy to go through over to your friend Claude and try to figure out what's going on. But now I've got essentially data which we all semi-structured data, unstructured data, whatever, I've got that stored over here, that gets fed into the mix. And when does it go from data and systems to noise and indecision? And by the way, don't forget, you told me you're still gonna deliver this by the end of Q3. Yeah, and you know,

this opens the whole can of worms about decision rights. Where does the box stop? Who's ultimately responsible for what? So that's that's number one. And if you define your number two is if you define your decision rights too narrowly, then you're going to lose, like I said, who's looking at the overall big picture, who's looking at the puzzle, and who's accountable for that. One of the problems that I see is these tools are so powerful and so quick, and whether it's illusory or real with answer you get. It's always assumed to be real, it isn't always, but they're so powerful and they're so quick that it creates this illusion of completeness, right? That this is the complete answer that you need. When it may or may not be, but then who's the watchdog? Who's the person saying are these assumptions valid? You know, who's ultimately responsible for that broader decision. I think these are questions that are not always taken care of in training or even considered in these use

case rollouts. Absolutely, and it goes from the idea of we're using frontier models, but are we adequately. Reinforcing it using retrieval augmented generation, RAG. Do we have our content in there? Is it the right content? Is it the source of truth or is it some obsolete document that throw into the mix. And by the way, having done this before, a handful of obsolete documents can really skew the mix and there's a lot of find out late or what I like to call it. You find out downstream and tracing that back to what actually happened. That's a lot of effort mixing that into the puzzle. So absolutely agree with what you're saying, but But the ramifications for the organization, especially when you've got a new paradigm or new sets of paradigm, where now we've got new tools, you've just been trained on all this stuff, make it happen, there's still a lot of overwhelming effects I've seen in organizations where certain people adapt to it, maybe a little bit better, but others are still trying to figure out what to do. and when you put this all into a project and you're trying to deliver, it's hard to herd those cats. I'll give you another metaphor that I used, and where I've worked with organizations who have tried to go very broad and let's roll out a enterprise-wide platform, give everybody the tools, and there felt like a lot of see what happens with it, But also not really realizing the downstream effects of well if this happens, the second, third, fourth order effects that you were talking about before, nobody really planned for that, and while they might have envisioned some of the second order effects, they didn't see the third and the fourth orders coming. But what I've talked about and. I don't know how this works with you too, but the the guide you need on top of this is. It it's a hybrid. It's someone who knows AI really well, knows the power of it, knows how to respect it, knows what can go wrong, but also understands the domain, the business at every level. I've called this idea or this role an AI show runner instead of an AI expert. And I think that mentality in speed, urgency, knowing what can go wrong, on the fly decision making of how we make that better, that goes in a lot to the success or failure sometimes of some of these programs.

You know, that's a really intriguing idea. And I love the concept of an AI show runner or air traffic controller or whatever you call it. I could foresee a challenge that that person is gonna face.

There's a lot of stuff flying around. Yeah, you need your air traffic control systems to manage this.

Yeah, absolutely. But is that person gonna be empowered? Are they gonna be able to say, no, don't do this? And are people gonna listen? I mean, that's the tough part of this, right? Is what authority will they have to try to guide some of these things that are happening? Love the idea. But I think making it making it run is going to require a lot of organizational courage and a lot of people willing to say, okay Yeah, I'll wait. Let's let's talk. Let's make sure this is right and

That may be where the fun comes in right where you've always got someone saying that Who owns what and do they have the authority to make decisions where we need to pivot? When the way we used to draw out battle plans and project plans and put things into, you know, program increment planning, there was no room for that.

But this is the whole key, right? If you get the decision rights cleaned up first, you get the operating rules cleaned up first, you get rid of all what I call the ghosts. Well, this isn't the way we've always done things here. First, then you do what you're describing, then it actually has a chance to work. I mean, we could sit back here and all day long about all the ways this goes wrong. But the reality is this can go very right, right? And that this can greatly help an organization. And the fact of the matter is, speed of decision-making has increased exponentially. You know, when you and I first got out of school, out of grad school, and my first job was working in a survey company doing new product forecasting. So someone would ask, okay, here's the product concept. How many units do you think we could sell in year one? We would design a study, we would recruit respondents, we would set up a simulated store, we would do all this research. Six to eight weeks later, we'd have an answer. Then along came the whole field of data science and dashboards and some automated processes. So then it was, no, I want an answer by the end of the day. Now it's, I want an answer in 30 seconds. And that's the reality of the speed of business. And maybe there's some things you can slow down, your competitors I'm slowing down. So how do we balance that? How do we balance keeping the responsibility while being cognizant of the speed of business? Right and it's absolutely and it

goes back to and I want you to talk more about ghosts and that idea and how that manifests itself. But there's also the idea of what we're getting at with tacit acceptance that what we're getting out of whatever model we've got on the side here, maybe we're doing internally, maybe we do but externally too and you know use Claude on the side not with the company data but a scenario that we wanted to work and we think it's right how much do we trust it is it right because it's trained on vast amounts of data that's not the company's data your proprietary data but the attenuation of and the smoothing of what it's producing. Is it right? I played with these systems for a long time. I've built some of these systems and I've been misled. I've also known when it's wrong. But there's a lot of acceptance to, Claude said it was right. It's got to be right.

Yeah. Well, let's talk about the ghosts come into play with all of this? I think that's a very good question to ask. So let me first just talk about what I mean by a ghost. So there are four types of rules. There's gospels, guidelines, ground rules, and then ghosts. So gospels are formal rules. No exceptions. Always do this. Those tend to be too restrictive most of the time. Guidelines are more conditional. Do this, if that. Ground rules are use your judgment, but don't cross But the fourth type is, and that's where you really want to be, but that's very hard to do in an AI. The damaging ones, though, are the rules that either aren't really rules. They're just ingrained practices or things that have become outdated. Like, let's say I sneezed. What would you say?

I would say, God bless you. Why? Because that's what I say. That's the gospel.

It's a ghost because there's a lot of theories, but the one that I personally give the most credence to is back in the peak of the Black Plague, the Pope mandated that if someone starts showing signs of the bubonic plague, you're supposed to pray for their soul. The first symptom of bubonic plague is sneezing. So if I'm in a room and I sneeze and you're worried that I have the bubonic plague You don't have to say bless you. You're free to leave as you can and put on a mask and get the heck out But it becomes such an ingrained thing we do without even thinking and there there are things like that that happen in organizations Every single minute of every single day There's laws on the books of every state that make no sense because something happened in the past Where someone felt the need to create that law? I don't know if this is still the case, but I live in California, it was illegal to put an orange in a bathtub. And I'm sure there's a great story why that's the case. I have no idea why it could be, why someone would even want to do that.

I have to admit I'm not well versed in the orange in the bathtub.

Neither am I, but it's just one of those crazy things you read about.

Do I run over to Nevada and it's like, well, I really want to put this orange in the bathtub. I let's go. Where do I go like do it?

Yeah, but but people are putting you know figurative oranges and bathtubs in organizations all the time because it's the way they've always done things and they don't question things or their rules. It just don't make any sense anymore. But again, it's the way we have always done things. The reason why it's a steering wheel and cars because that's what boats had. And it's just stuck. So there are a lot of things that just stick. And and we don't question them. And so where that gets to be a problem is then those become part of the decision making process. And it hurts in one of two ways. Either it's something you shouldn't be doing anymore, because it is something you do anymore, then it becomes automated and becomes part of the algorithm. Or, more commonly, it's something that actually defines how we do get things done. And it's unwritten. And it's just something people know. And that never gets put into decision calculus. And so all sudden, you know, Fred, who's the distribution manager in Topeka, had this whole system in his head about, well, I know this person, I know that person, they're going to be upset if this happens, I need to call this other customer and let them know because they like when I do that. There's all this tribal stuff that just gets lost. And the fact that so many people are remote, passing that down becomes even more difficult. So you have these two conflicting things with but both of which can lead to this narrowing of focus of the AI, and when you have too narrow of a focus, and you layer on the empire building, that's when the trouble starts.

But in the way to what you said about tribal knowledge is really important because it's the this is not documented. These are unwritten rules. It's tribal, but it made the organization work, or it made the processes work in the past. I've been on a lot of efforts in trying to correct large distributed applications, or we've got a business process that needs to be improved. And I've gone into various industries and done this, but found out a lot. Well, like your process is like this because it's grown up organically. There's a lot of tribal knowledge. There's a lot of workarounds or variations that you do. you've got all these systems you're connected together. There's a lot of brittleness into making things work. If one dependency falls off the cliff or if we change things, then things can go awry pretty quickly. Troubleshooting this and fixing this, you need the experts involved in here. By the way, this is not in your AI system, which was trained on what was ever in your Microsoft SharePoint systems, your other document repositories, something on one person's laptop that they throw into the mix, it's going to make mistakes, but you're still going to be tempted to think it's right. And therein, part of the problem lies. So what, how do we get out of this? What do leaders do? Especially when a lot of executives are thinking, well, you know, this should be pretty simple, but they don't understand what's really under cover the vast amounts of tribal knowledge, data all over the place, data that doesn't agree, that's where it gets interesting. What happens? What do we do?

Great question, and I think this is why we've developed this whole AI readiness assessment, which is focused more on the organization and the people and the tech. There's certainly a tech component of this, a strong tech component, but you have to make sure you understand how your organization actually operates, how prepared you are from a systems, from an interdependencies for decision rights, from the mentality of your workforce, what it's gonna mean for them. What's the true human in the loop and what you're trying to do and how does that imply governance? It drives me crazy every time I log into LinkedIn and we see all these magic bullets that people are pushing. It's like, oh, you just need this. No, you just need that. No, but yes, you need all those things, but you have to take them and look at them in a holistic manner. I'd say before you do anything, assess your organization, identify where the gaps are. They're going to derail everything you're trying to do, line everything up, and then implement. If you've already implemented, it's not too late. You could certainly then say, okay, where are the things going wrong? Where are the things going right? And there are a few people that if everything is going right, great, then, you know, we actually will certify you and you can you can screen that to the mountaintops. And that's a story that should be screened to the mountaintops because there need to be more good stories about where this has succeeded. It's very easy to complain about where it's wrong. This isn't going away. So if we can find the ways that it's working, help organizations prepare for it by making sure to get all this stuff that can be relevant out of the way, then I think we're going to see something spectacular. What are your thoughts?

I completely agree. If I see AI transformation and empty claims and magic happens, magical thinking, all those ideas on LinkedIn and other sites, especially where you've got a lot of people jumping into from the tech side. Oh, now we do AI. And everybody seems to be talking the AI game, and by the way, every tool now has its own AI, which is creating more confusion. And do we use it? How good is it? How do we test it? Nobody really has clear answers to that. But I think the certification we're talking about, the AI readiness, some very open and honest conversations about here's what needs to be addressed first lot of foundational layers before you can do this And then how quickly can you get things ready? I think that goes a long way and that's a great way to start approaching the subject But real honest conversations about here's what you're not seeing Here's what we do and here's how we make it better

No, I totally agree totally agree with that. So so given everything we've talked about today Brad what? What final word would you have for the people listening in?

I think you have to look at the power of AI and where things are going, and how the models are getting better, and how the harnesses are getting better, and how we can use it. But you have to treat it with respect and the fact that there's still a lot of newness and the speed at which, and this is something I actively follow and work on. Everything is changing at a speed at which we haven't seen before. So to factor that into the mix, to know how to use that showrunner mentality, to thread the needle, to actually deliver things that are workable, not introduce risk, make things better without trying to get too far ahead of our skis. That's part of the, I think, recipe that we're trying to get to. Yeah, amen.

I agree with that 100%. And the only thing I would add to that is, if you're getting a Ferrari engine and you're putting it into an old beat up car with a broken axle and a boot on one tire and a banana stuck in the tailpipe, guess what? It's not gonna matter. You have to fix the environment that the AI is being placed into for it to do what you want it to do. And then you can be racing down the road as fast as you want.

Racing sounds like fun, but not in that old broken car.

Right. Exactly. Brad, thanks so much. I really enjoyed the conversation.

It's always a pleasure talking.

Thanks. Take care.