Katherine Forrest: Hello, everyone, and welcome back to “Paul, Weiss Waking Up With AI.” I’m Katherine Forrest and we don’t have Anna Gressel with us today, so you’re going to have to again just suffer with me. But she’ll be back. She’s coming back. She’s just got some things that she’s taking care of and we’ll have her back very soon and that’ll make things even more exciting. But in the meantime, what you’ve got—and you folks, you know, you’re sort of just listening to me, you can’t actually see it—but my background is now no longer Maine. I’m actually in Woodstock, New York, which is my sort of non-Maine, non-New York City place that I go. So I’m here often when I’m recording. And so that’s where I am. So you’ve got me in Woodstock, New York, and so I am still enjoying coffee, although not the incredible coffee that they have in Maine.
But speaking of places, this brings me to the topic I have for today. And the topic that I wanted to talk about, and I really wanted to talk about this for some time, is where models come from. And I want to talk with you today about why that is really important. I got this idea for the podcast when I was talking to actually a number of clients about risks with certain AI models. And a few things kept coming up again and again, particularly now that we’re in this brand new world of agentic AI. And there are questions about what a company needs to understand and to know for its own basic, really safety and liability about its models, about where they came from. And then also there’s the second requirement of some procurement issues that are part of now the White House’s AI Action Plan. And so I wanted to go through those because they really relate to one another.
So the first thing I thought I’d do—and by the way, let me just tell you, I saw this person on the subway, the New York City subway the other day, and she actually had the word “nerd,” N-E-R-D, tattooed across her fingers. There was like an N, an E, an R, and a D on each of the digits. And I thought, “well, I’m a nerd,” but I’m not sure I’d go as far as tattooing it across my fingers because what if I one day became a non-nerd? But anyway, this episode is going to demonstrate for you all what kind of a nerd I am. So let’s start with the basics of what a company needs to know about the provenance of its tools. And that’s really true regardless of compliance obligations because we all know right now that there are a variety of tools marketed by a variety of vendors, some of which are well-known and almost becoming kind of brand names. Then also every single day there’s a new tool. And some of these tools are going to become well-known even though they’re brand new today. Some of these tools that are just right now, new unknowns are going to become the best known tools in a few years, because we don’t know who the winner of these AI races for any particular domain is going to be just yet. So we’ve got these tools marketed by these vendors, and there are certain questions that everybody needs to ask.
So first of all, the tools can be made in a variety of ways. And one major way is what I have referred to in prior episodes as my Lego block example. And, you know, you’ve got this sort of base model LLM and then you have sort of a fine-tuned model on top of that, sort of like a Lego block, one sitting on top of the other. And so you can also have as that base model either a closed source model, a proprietary model or an open source model. And an open source model can actually originate with one company that released the open source code, but the base model, the base open source model, can go through a variety of hands before it actually gets to the vendor who may then build on top of it.
So you’ve really got to understand a fair amount, not just about the capabilities of the model and the models that you’ve got, but also the risk profile of that base model and then of the fine-tuned model to understand what you’re looking at as a totality. And so you want to understand, for instance, when you get a model or license a model from a vendor, what is the base model? And where has it been and where is it going to, so to speak? And what kind of testing has it been through? And what can you expect of the model in terms of updates? Are they going to be pushed through to you? Are you going to have an option to determine whether or not yes, no, you do or do not want to have the update because it could change the environment in fundamental ways?
So you can think of model provenance as a kind of chain of custody for the AI model. And we’re used to the chain of custody concept as lawyers, as sort of the full history, sort of soup to nuts, from one step to the other to the other to the other to get the full story behind the AI system or the tool. And you need to do this particularly when you’ve got use cases that could result in liability if something goes wrong or in not even liability from an external point of view, but even internal problems if something goes wrong. So in what I think of as traditional software, you might track code changes with version control and you might have little initials where, when there are code changes, you’ll be able to track who and how and what happened with the code. But for an AI model or a system, it’s different because it’s shaped not just by the code, but by the data, the training processes and sometimes even the quirks of the hardware environment that they were run on and also the place where it’s actually being stored. And if it’s open source, as I’ve said, it could go through that a number of times on its way to you. So you really want to ask some questions and have a standard set of vendor questions that you have at the ready that you fill out and that you get answers to when a model first comes in the door so that you can understand the entire life cycle of what the model has experienced on its way to you, what its datasets were and what they’re going to be, the tuning and every handoff along the way.
Now let’s make this a little bit more complicated, and this is where things were coming up with clients, which is in an agentic environment. And one of my favorite things is the multi-agentic environment, or you’ve got multi-agentic tools, AI tools. And let’s talk about what I mean by that and how that can impact this model provenance. Well, first of all, with any kind of single agent AI tool, we know that they have the capability to operate autonomously. That’s the whole point. And they can be given a task and they can plan out the various steps that they need to take to accomplish that task, and they can do it, hopefully, flexibly to be a good AI agent. And many of these agentic tools, these AI tools, will use other tools along the way, or they’ll take other steps and employ other capabilities along the way. But sometimes they can actually access a toolbox, an internal toolbox, along the way. That’s becoming an increasingly popular way of having agentic tools be able to be even more flexible. So you’ve got an agentic AI tool. That AI tool can actually access a little toolbox within the model itself. And you want to understand what’s in that toolbox, not necessarily in some sort of deep technical way, but one of the tools that can be in the toolbox is a browser or a variety of tools that can give you access to external tools. And so if you’ve got an AI agent that’s accessing external tools or is able to utilize a browser, even an embedded browser, that is then leaving your corporate environment, you could then have a whole other series of issues that could come up, which is, are you following the cybersecurity guidelines? Is it locked down sufficiently so it can’t bring anything back? You know, you don’t want your AI agentic tool to bring any friends back that you didn’t want it to.
And then you’ve got these multi-agentic tools. So a multi-agentic tool, it can work in much the same way as an agentic tool except that it will be—it can actually work in a couple of different ways. One agentic tool can actually, I think of it like a baton handoff in a relay race, can do part of a task, hand the baton to another AI agent on the, sort of, the track. And that other, the second AI agent then starts running and starts accomplishing another portion of the task and then hands the baton off to yet another AI agent who does another portion of the task. And why do they hand off the baton from one to the other? Well, one AI agent may be better at a particular task than another AI agent. So you’ve got these multiple agents that are actually working together. And then, just to complicate matters, you can have simultaneous AI agents that are all working on particular portions of the task at the same time. So what that does is it proliferates the issue that I was just mentioning about the AI agent even more which is, what are the tools that that multi-agentic AI tool is going to be using? Are they going to be accessing external environments? Do you know that? Are you aware of that? Have you gotten the appropriate safety steps sort of lined up? So you really want to understand, again, sort of what is this tool that I’m taking in or that somebody has asked the company to take in? And do we really understand all the different pieces of that tool so we can understand the safety and the risk profile of that tool. And of course, many of these tools have got fine-tuned pieces within them and the fine-tuned pieces can have their own issues.
So to be protective of your company, you do want to know where your model came from and you do want to understand what the handoffs are and you do want to understand what the tools are that can be accessed in terms of, again, not the technical aspects necessarily, but whether or not they leave your home corporate environment. And all of this is really part of what we call model provenance or provenance and the kinds of questions that you are going to want to be asking. And it’s about trust, but it’s also now increasingly, there’s an aspect layered on top of regulatory necessity. And so let’s talk about what that is. So as part of the U.S. government, so the White House’s AI Action Plan, they—as we all know, we talked about this during a prior episode—they had a push for American AI sovereignty. And they really are pushing a preference and, in some respects, even a requirement for American-made AI in critical sectors. And that’s not just about wanting to, so to speak, buy local just for the purpose of buying local, but it’s the federal government that’s starting to require that agencies and, by extension, certain contractors and partners prove the origin and the security and the integrity of the AI systems that they’re using. And this means that there may need to be detailed provenance records that aren’t just about having good practice, but are then becoming also this compliance requirement.
So we’re seeing this play out today in terms of procurement. And so if you want to sell AI solutions to the federal government or even to certain state agencies, you’re soon going to need to provide this documentation to show sort of lineage. And we used to talk about AI inventory as sort of one of the first steps of getting control of AI within the company. And now this is going to be a deeper dive. And it’s not, for this particular kind of procurement, it’s not just about the model itself, but it’s also about the workflows that the model powers in terms of what the requirements are to show that you’ve got the right kind of safety and integrity and the step-by-step processes and the decision points that are being taken by an agentic system. So the U.S. government wants to know about different kinds of components about the data sets, the libraries, the plugins and even the hardware that are part of the deployment environment. So. trust. Trust and procurement are a couple of things. And you can see some of the reasons for this, not in terms of the regulatory requirements, but in terms of where a particular company could run into issues. When you think about those baton handoffs and workflows, and you think about where liability might come into the process if something goes wrong. Let’s just say you’ve got a baton handoff from point A with your agentic model to the second agentic model to point B in this workflow, say triaging at a healthcare facility. And so you’re triaging patients, and there’s a baton handoff from one part to the other part of this triage process. Well, what if something goes wrong? And what if the triage process, there’s an accusation that it wasn’t done properly? And you’re trying then to locate where things didn’t go allegedly quite right. Well, you’re going to want to understand what that workflow is, what that process is, how the agentic model handed off between different portions of that process and who bears responsibility for the different parts of the process. Is it the tool developer? Is it the vendor? Which vendor is it? Is it vendor A for the first part of the tool or vendor B? Or do you have vendor C who’s taken control over the entire part of the process? Do you have insurance for one vendor but not another vendor? All of this is part of, obviously, accountability. So we’ve got trust, we have accountability, we have risk scenarios, and without provenance, understanding the provenance of your tool, you’re really sort of flying blind. You can’t answer basic questions like how was the underlying model trained? Where in the workflow does the baton handoff actually occur? And so you’ve got this embedded risk that you’re just sort of flying blind with and hoping that it all goes well.
But then there’s another aspect apart from trust, accountability, risk, and that’s quality control. Because if the model starts behaving strangely or just doesn’t do what it’s supposed to do, it’s not—let’s just say it’s not creating additional risk. Let’s say it’s not in a high risk environment like a healthcare facility, but it’s in an environment where quality control is important and something starts to go wrong. You also want to understand, is there a portion of the model which is misbehaving? And if so, which portion of the model? And was there an update? Was the update pushed through? Was an update pushed through that wasn’t supposed to be pushed through that’s now caused, say, a variety of inventory that you thought was all going to be wonderful? It’s going to have to be redone and that’s going to actually eat into your profit margins. So quality control is another aspect of this whole process.
So again, we used to talk about keeping an inventory of your AI models and you’ve still got to do that and you still should be doing that, particularly because there’s so many now state regulations. I think we’re up to 40 states now that have state regulations for a variety of things, and you want to have your AI records as to what models you’ve got. But you’re going to want to have what I’m going to call, and I’m not the only one who’s calling it this, but a provenance record. And a provenance record will be the particular sort of digging down into the model. Think of it as a tab on your Excel spreadsheet that will have all the particularities of your model and it captures the model’s origin, the data sets that are used, the training parameters, the tools, the libraries that are involved, the APIs that it can call on, whether or not it has a particular tool set that it has access to, whether or not everything is SOC compliant or compliant with your particular cybersecurity guidelines. And that provenance record isn’t just a static document, it’s going to have to be updated continuously as your model changes and goes through updates—which happens all the time—as your workflows evolve, which means the process by which you’re actually undertaking certain kinds of work with the model. And so for agentic AI, this is just all that much more complicated because you’re going to have to do that when the baton is being passed or when the tool is actually utilizing different internal tools within its own toolbox.
So we also, lastly, don’t want to forget about the human element, which is you have to train your teams about the importance of provenance and about making records of model provenance, and you’ve got to push vendors who are brand new at this too, right? Vendors aren’t necessarily used to providing all of the records of model provenance because this whole area of AI tools is so brand new that there isn’t sort of a market ask for what kinds of documentation necessarily comes with a set of tools. So asking for provenance records and the data that would populate your record is a completely legitimate ask, even if a vendor says, “well, I haven’t done this before.” Well, they probably haven’t been in business for all that long, I mean, apart from some of the major companies which now have AI lines of business.
So that’s my model provenance sort of talk for today. And I hope that you have found it useful. I’m Katherine Forrest, for those of you who don’t recognize my voice. And we’ll have Anna back with us very shortly, and you’ll have then her voice as well, which will break it up. But I want to thank you all for joining us on this episode of Waking Up With AI. Don’t forget to like and subscribe, and we will hopefully have you all with us next week. Thanks very much.
