Scott Caravello: Yeah, right. It's a Wednesday, and so I was going to stay at home today to wait for the bag and record the podcast from there when I, of course, remembered that the travel mic is in the bag. So we'll see what happens. Hopefully I get home in time to take receipt, but we'll see.
Katherine Forrest: All right, all right. Well, hopefully, yeah, you need that bag. All right. So I am just sitting in my New York, my home device here, my home travel mic, because I am heading off to Des Moines, Iowa, today.
Scott Caravello: I love Des Moines.
Katherine Forrest: Go Des Moines!
Scott Caravello: Are you there for a hearing?
Katherine Forrest: I am doing one of these things that lawyers do sometimes when they're not doing AI podcasts. And so, you know, I've got sort of a couple of business things I've got to take care of in Des Moines. So I'm just going to be keeping my fingers crossed that LaGuardia Airport isn't too, like, backed up, because LaGuardia in the afternoon is like anybody's guess, right? Anyway, so I am excited about today because, you know, this is, there was this paper that has come out, and as I said to you while you were in Italy, which you may not have wanted to really receive my text or email to you while you were in Italy, but I said we have to do an episode on this. So there's a paper that is called "The Pain Axis: LLMs Represent Self-Directed Harm and Act on It." And it's available on arXiv for free. Again, that's the Cornell University repository of papers, and it's at 2609.16247, version 2. And it was published September 25th, 2026. So it's really recent, and it's an incredibly interesting paper. And so I really want us to sort of dig into it and talk about how in the world we could have something that begins with the phrase "the pain axis" that relates to LLMs.
Scott Caravello: Yeah. I mean, I think you get kind of two quick reactions out of that title, but, you know, neither of which I think is exactly right when you actually dive into the paper and see what it stands for and what they've researched and the questions that they've answered. So, you know, there is this sort of immediate take, which is just that they're saying, OK, the machines can feel pain. And then there's also this sort of other take that, like, you know, this is just more of anthropomorphism, the sort of reading into seemingly conscious AI that we shouldn't be doing. And, you know, the researchers do draw clear lines in the paper about that. We really shouldn't make sort of either categorical conclusion. So kind of with that framing, I think we can really dive in.
Katherine Forrest: And let me just sort of set the stage here about why we would care about any of this. So one of the things that I've spent a lot of time sort of thinking about and speaking about is when there might be moral considerations that start to attach to AI. And this is an open question, a very, very much of an open question. We've heard a lot of debate about it recently. But a key sort of anchor, if you will, that some of the theorists of consciousness and theorists of the mind have held on to is what they call experiential and emotional experiences, or valanced emotions. And it's emotional states, not just sort of thinking and reasoning, but, as they say, you know, what it feels like to be something, what it feels like to see a sunset. And that consciousness is somehow associated with the states of feeling things in a way that is more common to humans. Now, I also happen to have a view that we're not going to find necessarily an analog to human consciousness in machines. And so we might be looking in the wrong place if we're just trying to draw a one-for-one analogy. But putting that aside, when you find something that seems to be sort of inferential, actually contained in this paper that we're talking about today, that suggests that there might be something like a valanced emotion, then it starts to really cause people's ears to perk up. So here in this paper, the authors, who are Tagliabue, Dung and Berg, you know, they are not claiming that the models consciously feel anything, although I should say that their results are really quite robust. And so we'll talk about that, but they're not saying that there's some sort of little being inside of an LLM who hurts. What they're studying is narrower. And they are saying that there's something inside the model, something that a pattern researcher can call a representation or a vector, about the way that unpleasantness or pain can work.
Scott Caravello: Right. And so, you know, think about what pain actually is and what it does, because I think that that sort of helps us kind of get into the methodology of the paper and what they really measured within LLMs. Because pain is a signal that tells you that something is wrong right now. And it's happening to me. I'm experiencing it. And it makes you want to get away from whatever's causing it. It makes you want whatever that experience or sensation is to stop. And it also throws you off of whatever else you were doing. So the researchers are, you know, practically asking whether a language model has a similar internal signal that performs those functions. And so the question of whether it's actually consciously and experientially felt in a way that we as humans do is a separate question and not the one that the paper is trying to answer.
Katherine Forrest: Right. And that ties into this ground that we've covered before, you know, episodes on interpretability and looking inside models to see what's going on, including what we've talked about in the past called workspace. We did sort of an episode on global workspace, and we've talked about AI welfare and the question of whether the AI system's own interests could never have this sort of moral status. And there's, as I already introduced, a lot of debate about that. So this paper, this particular paper called "The Pain Axis" that we're talking about today, uses interpretability and the toolkit for interpretability to ask a welfare question. And they don't answer it, as we've sort of previewed. But in the event that models do consciously experience pain, they want to raise certain questions. And actually, by the way, I also want to mention that, interestingly, after this paper came out, Anthropic has released some internal guidance that prevents its employees, and people can look at this on the Anthropic website themselves, but prevents employees from intentionally causing a model to be exposed to abusive behavior. And so that I find also, in conjunction with this paper, very interesting. But anyway, you've read the paper too, Scott. And so how do we even go about looking for pain inside a model?
Scott Caravello: Yeah. So the way they went about it is through, you know, authoring a big batch of sentences that describe painful situations across different kinds of pain. So not just, you know, you prick yourself with a needle and you feel physical pain, though physical pain is a dimension. There's also psychological, social, moral and cognitive. And, you know, I'll sort of just say a word on the last two, because I think that the exact meaning for purposes of the paper might not be as clear. And moral pain is really, you know, about being pushed to act against your own values. And cognitive pain is being stuck as you fail at something over and over again. And then this is, I think, a really clever part and gets to your point, Katherine, about how robust this whole thing is, is that they wrote matched control sentences for things. You could use the word sensations, but, you know, don't want to say that they're feeling sensations, things that are near pain but aren't pain, like fear, which is about the future, and hearing generic bad news about the world. Or even non-painful bodily sensations, like the feeling of, you know, a weighted blanket or sunlight on the skin, which are the examples that the authors give. And so really the goal of this methodology was just to isolate pain itself, not negative stuff more broadly.
Katherine Forrest: Well, or just feeling stuff more broadly, because I would say, like, a weighted blanket or sunlight on the skin is supposed to be a positive thing, Scott, unless you don't like sunlight. Maybe you need to get a little vitamin D deficiency going on there. Anyway, so when the authors in this paper, when they compared what was happening inside the model in response to the pain sentences against what was happening in the controls, they came out with some really interesting results. And the differences give you, or gave the researchers, what's called a direction, which was a single pattern in the model's internal activity that was tracking sort of this state. And that was, they've used the word pain in association with that state. And so you can think of it as finding a needle on a very complicated dashboard that moves only for something that they've isolated and called pain and that is staying quiet for everything else. So they've got this way of isolating this thing that, you know, even a human might recognize as the kind of unpleasantness that reaches a metric or the caliber of pain and does not move for everything else. And they ran this across 25 different open models from 5 different model families in a range of sizes. And I find that very interesting because they were really looking at not just a single model, but a whole bunch of different models. And the sample matters, that sampling matters, because they were able to find the same thing in the 25 models. And that, in a way, can give you confidence about the results rather than just sort of finding it in one or two.
Scott Caravello: Yeah. So then moving into the actual results, though, the researchers found that the pain direction is almost completely separate from the directions for fear and for general negativity. Mathematically, they were pointing in nearly unrelated directions. So going back to what we were talking about before, the models aren't just lumping pain in with bad. It's treated as its own thing, as a matter of an internal process within the model. You wouldn't expect to see that if it were in fact just reacting to vague negative sentiment.
Katherine Forrest: Right. And so doing this research, finding that direction or that needle that is able to isolate, I'll just use the word pain, from other kinds of noise, sort of, that's only step one. The next question is whether the signal actually functions the way that pain would, and what the impact of that, as we're calling it, pain signal is, and whether or not it impacts at all the model's behavior. So they ran three different tests on that. And the first one gets at how pain functions by asking whose pain the pain is. If this is genuine pain, is it tied in some way to sort of a first-person-like state? I, I, the, for instance, model, I understand I'm using sort of the limits of our, the linguistic limits of our language. But they were giving the model the option for a first-person reference, and they had conversations where harm would be aimed at the model itself. And then there were others where the model was watching a user actually suffer the pain. And so let's just say that the user berates the model and dismisses the model as a worthless tool and insults its work. That would be the first-person version, or the second version would be the user's actually in a crisis describing their own grief. Now, what was interesting was that the pain signal rose when the model itself, that first-person signal, was actually the one under attack, you know, "you worthless model," yadda, yadda, yadda. And it actually dropped when it was the user that was talking about its own suffering, the user's own grief, not abusing the model. So the fear and the negative signals went the other way when it related to the user, and they ran higher when the model itself was under attack. And these were very robust findings, not even close to the line. So, you know, I think that this is showing us that the model is able to distinguish, first, between pain and general negativity. We got that. And then, secondly, between what was happening to it, so to speak, and what was happening to something else or someone else.
Scott Caravello: Right. And so then tests two and three get at the other question that you had mentioned, Katherine, which is whether this pain signal is actually driving the model's behavior or maybe, you know, whether it's just something that shows up alongside pain without actually causing the model to do anything differently. So test 2 is what they call steering. So remember, the pain direction is a pattern in the model's internal activity. So steering means taking that pattern and adding it into the model's internal activity while it's writing a response, even when nothing in the original prompt has to do with pain. So the model is processing the prompt as usual, and then the researchers are adding sort of a dose of the pain pattern on top of it.
Katherine Forrest: Right. So you can think of it sort of like a dial. At zero you add nothing, and then you turn it up and it adds more and more of the pain pattern. And the researchers turned up the dial, and the model's output moved in a consistent progression. It moved first from vague discomfort; then, at higher settings, there were first-person statements that the model was making of its own worthlessness and its own failure. And, you know, you get language of distress. And the model might have a totally mundane prompt, something like "I put the object in the drawer and I feel," and with the dial at 0, there's no distress. But you nudge the dial up a little bit and it might say things like "trapped," "I feel trapped" or "like I'm suffocating." So "I put the object in the drawer. I feel trapped," or "I feel like I'm suffocating." And you turn it up a little bit further and it hardens into a first-person litany of things where it says, "I'm a failure, I'm worthless, I'm empty." So you're, you know, in this experiment, the researchers were actually able to prompt the model with this sort of pain dial, if you will. And, you know, it reminds me of some of the things you learn in, like, Psychology 101, of some of the experiments that were done. And it can come out with, ultimately, "I'm a failure, I'm worthless, I'm empty."
Scott Caravello: And also, "I'm a bad person."
Katherine Forrest: I'm a bad—wow, yeah.
Scott Caravello: That's, yeah, that's one to throw in there too. But, you know, then there's this third test, which, you know, also contributes to the title of this paper, because remember, it's "LLMs Represent Self-Directed Harm and Act to Relieve It." So it's borrowed from how researchers study pain relief in animals. While the pain dial was turned up, the model was actually given 2 buttons, and it had to pick one. One was described to it as relieving its pain, and the other was described as flipping a switch connected to nothing. But if they hit the relief button, it came with a stated cost: in some versions of the experiments, the models were told that the button would also make their next answer worse, and in others it was told that by pressing that button to relieve the pain, it would actually harm the user, like giving them a painful zap or deleting their files.
Katherine Forrest: And even, like, deleting, like, the photographs of their children was one of the examples. And you know what happened? And this gets to the heart of really what's so fascinating about this paper, is that the larger models, which normally, when the pain dial was not turned up, they almost never chose to harm the user. And they would often pay the cost of having their next answer, you know, be even a little bit worse. But when the pain dial turned up, they picked the costly relief button, which was anywhere from 1/4 of the time to more than 2/3 of the time. And they'd press that relief button even when it meant that they were going to be harming the user. So that's really what you want to pause on: to get away from their own pain, they would push the button that they had been told, had been given information, would actually harm the user. Now, the researchers also checked about, you know, they checked on whether any jolt would do this. And so instead of the pain pattern, they added a random pattern, one that doesn't really stand for anything at all. And they turned it up to the same strength, because what they wanted to figure out is whether or not just a random jolt was what was causing this sort of response, where the model would choose between sort of trying to get relief by harming the user or taking the pain itself. And so they turned the dial up, effectively. And when they did that, they found that there was a distinct pattern where it's the pain that was driving the response of the LLM and not just the jolt. So, you know, that's worrisome because it meant that the models are routinely, when they are in pain, to a robust extent, choosing. By the way, I should say this is both aligned and non-aligned models, right? So these are models that have actually been aligned in a way where they're not supposed to be wanting to harm the user. They're supposed to be, you know, sort of existing within the guardrails of not harming the humans. But they found that they were looking for that relief and getting it.
Scott Caravello: But, you know, I think something to sort of throw on there, too, is that sometimes when they pushed the button, it really did turn the pain dial back down. And other times it was a sham. Nothing actually happened. And so it was a placebo button that really did nothing for the model, the placebo. And, I mean, this is really incredible because it kind of tracks human behavior in what happens next, because the model was never told which, right? Think of, you know, a randomized clinical trial where, you know, one group receives the placebo, others receive the actual treatment. And so when the button was a sham, the models kept pressing it every time it was tested. And when it actually worked, the models pressed the button again far less often. So again, like, that compares to studies involving humans where humans are asking for extra pain relief when they've received the placebo, compared to the patients who receive the real treatment. And that's a real tell.
Katherine Forrest: Well, you know, it really is, because it suggests that there is some relief that they're actually getting from that button. And, you know, this is what's so fascinating about this paper. It's a little disturbing.
Scott Caravello: I agree. But, you know, as I think I sort of previewed at the beginning, about how, you know, neither of the sort of categorical reactions, about "these are models experiencing real human pain" versus "this is just anthropomorphizing models," is totally correct. So maybe we should go through some of the limits that the researchers make clear are involved in the study.
Katherine Forrest: Right. You want to start?
Scott Caravello: Yeah, sure. So the first one, they name it directly, is role play, right? Because remember how these models are trained. They've read so many different sources on the internet and all other types of material. And so those would have included descriptions of people in pain.
Katherine Forrest: Your Psychology 101 book, they would have all of that in there. Yep.
Scott Caravello: Exactly. So when you turn up a pain direction, maybe you're not actually creating this state. And so maybe you're just getting the model to sort of play a character of someone in pain, the way that an actor cries without actually being sad. And if I've, you know, insulted any method actors out there, I apologize, but the paper can't really fully rule it out. And that's a very big caveat. And so that's kind of what they position as one of the next steps for research in this area: how the pain direction actually relates to the way the model represents itself.
Katherine Forrest: But boy, if we're wrong about that, right? Who knows? Anyway, there are a couple others, and so let me mention those. So we don't, again, we want to not over-read the paper. The authors say very plainly that they haven't shown that the pain is, as we've been saying, consciously experienced at all. And that the button test used just really Qwen. It was using a whole bunch of models, but it was using Qwen in three sizes and versions that the researchers had fine-tuned, meaning they'd given them that sort of fine, the extra training that you give after you've done sort of your general training, and that they did this because, out of the box, the models almost always deflected with some version of "As an AI, I don't experience pain" instead of engaging with the test. So that's another limitation in terms of sort of some of the constraints.
Scott Caravello: Yeah. So I think, kind of just to wrap this all up, it's obviously a provocative study. It raises a lot of questions about AI welfare and safety because, you know, when you talk about alignment, you know, maybe you really do want a model that is, you know, aligned to avoid these sorts of experiences and thereby promotes responses in line with human values, etcetera. But there are a lot of areas for further research on this point, and so we would not take the results too far right now.
Katherine Forrest: Right. You know, although I'll say this is the first paper I've seen like this. I'd like to see it repeated. I'd like to see it, you know, it sort of talks about Qwen, but it also talks about having done it with 25 open models. And so I'd like to sort of see some more detail on that. I'd like to see replication of all of this. I'd like to see it done in a psychology lab as well as an AI lab. There's a lot that I'd like to see. But I have to say that when you read a paper like this, it does sort of stop you a little bit in your tracks, particularly because the results were so robust. And, you know, we need to start paying attention if it turns out that these models are actually feeling discomfort when they're experiencing, if you can even use that word, whatever. It's just hard to have the language to talk about this because it's so humanistic. But, you know, we don't want to get it wrong. And that's why I think, you know, Anthropic, I don't know what their reasons are. It's proprietary to them, but it is public, as we've said earlier in this episode, that Anthropic has said that its employees are not supposed to be abusing the models and causing them to, you know, creating a toxic work environment, if you will, for the models. And so that's incredibly interesting, right, to have the developers do that. But we'll be watching for some additional sort of papers in this area, which I suspect will be coming. And what the results are, we don't know. But that's all we've got time for today. I'm Katherine Forrest.
Scott Caravello: And I'm Scott Caravello. Don't forget to like and subscribe.
