Scott Caravello: All right. And so when we talk about these signals, right, they're also often called watermarks. And not just the watermarks that are visible, like an icon or a legend that specifically tells you that a photo is AI generated. But there are also these signals there, to use that word again, that are called watermarks, but they're machine readable, which means that a computer can actually detect the watermark is there, which indicates that the content is AI generated.
Katherine Forrest: Right. And we're going to give you an illustration—our audio—an illustration of this. And there's no better place to start than a story from just last week. We're recording this right now on August 20th. And so let's go to last week and talk about another development from Anthropic.
Scott Caravello: Yeah, so they announced that Claude, its AI assistant, is going to start putting an invisible watermark on basically everything that it writes. So we're not just talking about images and visual content, we're talking about the written words, text output. And the reaction online from folks was, I think to put it mildly, not warm.
Katherine Forrest: Right. And so what they did is, to sort of give you a little bit of clarity about the episode, they announced it for Claude models that are released on or after August 2nd. So relatively recently, the text that Claude generates now carries an invisible watermark. And to be clear, originally I think that people thought of this as sort of a surveillance issue, but it's directly, at least, responsive to EU AI Act requirements. And it turns out that the feature that Anthropic released isn't just a European feature, it's a worldwide feature. Because of course, it's very difficult to know where the content that you might be creating in one place is going to end up. And there's a lot of different ways of not even being able to precisely tell where content is being generated. So there is this now new requirement that we'll talk about from the EU AI Act. And so what Anthropic was doing was being responsive to that. And in their words, they didn't yet have a way to cleanly scope a region to sort of determine whether or not something was going to be limited to one region and not another. So to comply with the Act, they put this watermark on on everything. So there you go.
Scott Caravello: Yeah, exactly. But so getting into a little bit of the how they're doing this watermark is also really interesting. So as a language model writes, a lot of the words that it picks are basically coin flips, right? Because different words could be interchangeable and you still get the exact same meaning. So to use Anthropic's own example, take the sentence "the weather was cold," and what could the next word be? It could be either gray or it could be overcast. And it's reasonable to expect that a reader wouldn't notice the difference, right? The quality of the output isn't affected regardless of whether that next word is gray or overcast. And so normally in those situations, the model breaks that tie with a random number. But this watermarking method just swaps the source of that randomness. So instead of a plain random number deciding which word you get, there's a secret key that's used in the context of the words that come earlier in the sentence to make the call about which word to use.
Katherine Forrest: And so to note some of the limitations of this, the watermark, it actually can fail on really short snippets of text because there just isn't enough text for the hidden pattern to do its thing. And you know, what Claude, when it's proofreading or tweaking a few words in a human-written document, if it's just too little, it's not going to be able to work. It also has an issue with this watermarking and placing the watermark on certain code and certain math where there's one right answer because there's nowhere to actually hide a signal, and you know, it gets sort of scrubbed out, and you know, a determined rewrite sort of scrubs it out. But one note for the privacy minded, for those folks who are thinking, gosh, has this actually got a surveillance aspect, is that the mark does not carry information about the generator, the user. It can't be traced to a person, to a company, or a particular chat. But of course, Anthropic is just one company, and they're not alone. There are a number of companies that are now doing some form of signal placement within their generated content, yeah.
Scott Caravello: And, you know, I really think that the reaction to this development is kind of driven by just how prominent Claude has become in so many people's lives and what they do, that when they see this, they've, you know, given so much of their attention and interest to it. But Google DeepMind actually pioneered the technique that Anthropic is using, which is called SynthID, and it's been using it to watermark text for some time, including in its Gemini models.
Katherine Forrest: Right. And so SynthID is S-Y-N-T-H and then big capital I-D. So like synthetic, like SynthID. And when it comes to images, what Google DeepMind's technique—it's embedding this signal, as we call it, into the actual pixels of an image, and with audio, into the sound wave of the clip. And it's imperceptible to humans. And for every region of an image or every second of an audio clip, it actually carries that signal so that if you crop the image and take away the edges of an image or you compress the sound file, it doesn't actually just sort of wipe the signal out. So you've actually got that signal that Google DeepMind's technique from SynthID has placed there in every portion of the generated content.
Scott Caravello: And that's actually the technique that OpenAI and others also use for their image generation. But so I think that's a great overview of watermarking, but there's also this concept of provenance. So what is that? How does that differ from a watermark?
Katherine Forrest: Right. So a watermark is a signal that's, as we've talked about, hidden inside the content. But provenance metadata, which is what it's called, is more like a signal label attached to a file or a record of where it came from and actually also what happened to it along the way. So it's got a lot more information in it. The dominant standard is a real mouthful of a name, is called—here we go—the Coalition for Content Provenance and Authenticity. Get that again, the Coalition for Content Provenance and Authenticity. But, you know, mercifully, everybody just calls it the C2PA. And honestly, I think the way to think about it is like as a nutrition label for a file, for a digital file. It's like a little packet of secure information that travels along with generated content. Think of a photo, video, audio clip, and it will allow you to, quote, see—although you can't quite see it—you can actually find out information using certain techniques about where that file came from, who made it, has it been edited or changed or altered along the way. So it's different from a watermark, and it's more complete than a watermark, though it's got some overlapping functionality with a watermark. So with all of this information, if you were trying to figure out whether or not something was real from something fake, it might be that you'd be able to tell, OK, it went through this place, that place, and the other place. It was real, real. And then it started to become fake. It started to have additions added to it. And so you might then figure out that there was something altered, not in its original sort of created form, but along the way. So if you've ever spotted the words "content credentials" on an image, that's just the consumer-facing name for this C2PA standard.
Scott Caravello: And the way that the C2PA works is cryptographic. And I will explain what that means, right. So when a photo is taken or an image is generated, the system, quote, hashes the pixels and the metadata—and the metadata is like the data about the data—into a unique fingerprint of sorts. And then it seals that fingerprint with a signature that only the specific device or software could create, but that anyone can use to check and confirm the origin of the content. And so then, as you had previewed, Katherine, if anyone later alters the file, you could reveal that something changed. So every edit, a crop in Photoshop for example, adds a new signed record that points back to the previous one. So you basically get something that looks like a chain of custody for a file.
Katherine Forrest: Right. But it's not as if you and I could just sort of like, as we see content on, I don't know, Instagram or something, that we're able to immediately understand whether or not that particular image carries and read the data that is from the C2PA standard. It's actually more complicated than that. And to be clear, you mentioned photos being taken and embedding this information, but not every camera will actually embed this information, the C2PA information. That will get into new legal requirements that may require them to do so later. Right now, Google's Pixel phones actually do embed this information, and Apple actually has its own system. And so there are big labs like Anthropic, Google, OpenAI that all embed the C2PA metadata in the images that they generate as well. So what we have are some cameras that are actually embedding the C2PA information as the photograph is originally taken and then others which are not. But then some AI developers are actually adding it to certain generated output.
Scott Caravello: Exactly. And so then shifting gears a little bit towards the legal developments that are related to this news, maybe we could talk a bit about that EU AI Act hook and what exactly it requires that's led Anthropic to make this announcement.
Katherine Forrest: Right. And it's not alone. You know, the EU AI Act is only sort of the first one that people were learning about, but we'll talk about another one as we go, and there's more on the way. So the type of watermarking that we just described is now mandatory under certain portions of the EU AI Act in terms of provisions that are starting to kick in the end of this month, which is August of 2026. So under Article 50 of the EU AI Act, providers of generative AI systems, which is of course a defined term, as we all know from episodes long ago and far in the past now, and then those would include certain big models. You know, there's a whole definition of AI systems and it could capture a lot of what large developers do, but also some of what some small developers do. And they are actually required by this Article 50 to mark synthetic output, whether it be audio, image, video, text, in a way that's machine readable. And there are other transparency requirements, but that's one that's related to this topic here, other transparency requirements under the EU AI Act. So machine readable, again, of course, it just means that it's readable by some sort of computer process. It doesn't mean that as a human, it's necessarily readable by, you know, you, Scott, or me.
Scott Caravello: Exactly. And so then sort of going a level deeper on these requirements under the EU AI Act, there are these codes of practice, which we've talked about in other contexts, but basically they're a voluntary set of guidelines that are designed to help companies comply with EU regulation. And so there's one that was finalized last month that's a code of practice on the transparency of AI-generated content, and that directly addresses the labeling and watermarking issues in more detail. And again, they're voluntary, but Anthropic, OpenAI, Google, Meta, Microsoft, they've all signed on to follow this one.
Katherine Forrest: Right. And so Google, for example, which has tied its C2PA and SynthID work directly to their compliance with the EU AI Act and the code of practice, they've already got it sort of baked in. And Anthropic's announcement about watermarking that we started this episode with actually did the same.
Scott Caravello: Right. And so the guidelines, which cover a lot of ground and not just this specific watermarking issue that we've been talking about a lot today, they don't endorse a single technical solution, right. But what's interesting, and I guess I wanted to flag, is that the carve-outs where it says that the compliance is not necessary line up with some of the limitations of this text watermarking solution that we discussed.
Katherine Forrest: Right. So very short text, text that is so short that it can't be watermarked with reliability, is actually one of the things that's carved out. And the threshold that is set is about 200 tokens, which is about 150 words.
Scott Caravello: Though the code also explains that the threshold is tied to what the state-of-the-art techniques can do at the time it was published, and they expect it to decrease as the technology improves. But I do think it's sort of setting this really interesting standard because there's all of this criticism about how burdensome the EU AI Act and the regulations are. And here we're getting this really interesting look into sort of tailoring what's expected to the state-of-the-art of the technology and allowing it to shift as that changes. So I just think that that is worth a note of interest.
Katherine Forrest: It's sort of interesting because I don't think of 150 words as that short, and you know, as an engineering matter, I don't understand. I'm sure there are really good and sufficient reasons why it's considered so short, but it doesn't—to just sort of the average human, 150 words seems like it ought to be plenty long enough, but it's obviously not. So maybe that'll be for a different episode. But let's go on to the—we said there was another one—California, and California's got something called the AI Transparency Act. And it also requires that these embedded, hard-to-remove markers in AI content actually start to be included. And they've got like a free tool that can be used to check the provenance or the provenance of AI-generated content. In other words, you can upload the output to a developer's detection tool, and it'll tell you if the content came from that developer's model. That's sort of what the California AI Transparency Act is doing. But let's talk about it because it's a little bit different.
Scott Caravello: Yeah, totally. So I would flag two respects in which it's narrower than the EU AI Act. First is that it applies to generative AI providers with more than a million monthly users accessible in California. That is a mouthful, but so it's really aimed at the major labs and app providers, which, unlike what you mentioned before, Katherine, that the EU AI Act, those requirements are scoping in some of the smaller developers. And then second, it applies to image, video, and audio, not text. But anyway, there is another part of the California law that I know you're interested in, Katherine, and it relates directly to our C2PA discussion.
Katherine Forrest: Right. So this is the part of the California AI Transparency Act that doesn't kick in until 2028, but at that point, so this is quite a long time from now actually, you know, right now we're moving so fast. The velocity of change is—I always say—is so tremendous that 2028 really feels like forever from now. But at that point, California is going to require makers of capture devices, so-called capture devices, like, you know, the cameras on on your cell phone or actual phones that can obtain voice recordings or just standalone voice recorders, to embed provenance records by default, with an option for users to turn that provenance record off.
Scott Caravello: And why is that significant? Because I think it says something about sort of the world, the world in which we're living now.
Katherine Forrest: We have these rules about what makers of physical camera devices must do to mark the origin of their content included in a law about AI transparency. So just think about that—rules about what the makers of camera devices have to do about ensuring that their devices actually comply with AI transparency laws. So, you know, it's really a recognition that strategies to make sure that everyday people can figure out whether or not something is content that's been generated by AI or not is available to them, and to reduce the risk of deceptive deepfakes. And so that's actually sort of—AI, I don't want to call it a collision, but a recognition that our worlds of devices and sort of what we think of as computer generated, these are all coming together. And the standards now for AI are going to be embedded in certain requirements for device makers that we wouldn't necessarily always think of as AI devices, like a camera or just your cell phone.
Scott Caravello: Yeah, it's sort of like the balance has shifted, right? I mean, deepfakes and synthetic content are so convincing that the approach lawmakers want to take isn't just to call out the deepfakes, but make us able to verify the authenticity of everything, because that's the way to actually tell what's real and what's not. It's fascinating.
Katherine Forrest: Yeah, you know, it really is. And as we mentioned earlier, that Google and Apple already do this in their devices, but some makers of dedicated cameras, they have actually already implemented this system. Leica was the first to do that back in 2023, and Sony and Canon followed, and Nikon actually added one as well but then suspended it in 2025 because of maybe a security question that they wanted to look further at. So there are some solutions that are being worked out. None of this is bulletproof yet, but it does show that there is, I would say, an escalating interest in finding ways to not just make the authenticity of content something that is discretionary, but that defaults to where something comes from, what its origin is, what its provenance is. And, you know, to be able to follow that because the house cow—let's go back to the house cow. We want to know whether or not that fluffy house cow is a real house cow or it's not a real house cow.
Scott Caravello: It's real, Katherine.
Katherine Forrest: No, I believe in house cows. I really do believe in house cows. I don't know if I'll ever myself be able to get a house cow, but I believe that there are lots of people out there who have house cows.
Scott Caravello: Oh, I, I mean, it was forever ago now, but I can find an Instagram account for you of a family that has house cows. It's real. It's definitely real. So I will try and dig that up if you're interested.
Katherine Forrest: Wait, hold on. Are you saying that you knew this family or you just followed this family?
Scott Caravello: No, no, no, no. It was like pushed into my Instagram Reels, like, feed years ago.
Katherine Forrest: Oh, you're no better off than I am. You don't know if that house cow is real or that even that family is real.
Scott Caravello: Here's the rub. I deleted Instagram like three years ago, so the fact that I had it then, three years ago, before these video deepfakes were that convincing, tells me it was real. I think it was real.
Katherine Forrest: Oh, all right, so there'll be a house cow. I'm just going to find the right house cow, and then one day I want a house cow. All you listeners out there, mark my words, one day we're going to be talking about me and my house cow. All right, well, that's all we've got time for today. I'm Katherine Forrest.
Scott Caravello: And I'm Scott Caravello. Don't forget to like and subscribe.
