LLMs are expert movie script writers
The likely implications of how LLMs are currently trained and used.
I’m an expert at neither screenwriting nor large language models, but something struck me recently: under the hood, an LLM is most often trained and used as if it was reading and writing a type of a movie script. It sees roles and what each role says. It also decides what to say next based on previous dialogue turns.
Imagine two characters, User and Assistant. The LLM sees and can probably predict both. But in most applications (on a very basic level) the LLM just predicts the dialogue of Assistant, while User’s dialogue is given from an external source (mostly from actual user input). Thus, if I switched the two and labeled the user input as Assistant, logically it might be able to predict User’s dialogue just as well as it would Assistant’s.
This leads me to three conclusions.
First of all, I don’t think LLMs have a sense of “self” (yet). Who is self? “Am I User, or am I Assistant”? Because it sees both in similar quantities during training, and simply predicts what should come next, it doesn’t form a sense of self-awareness as many might tend to believe.
I don’t even think it truly knows that it is an LLM. It simply sees the text that goes something like, <user>What are you?</user>, and has been trained to predict, <assistant>I am an LLM called Company_Name_Goes_Here.</assistant>.
Second of all, it never truly forms its own “identity” with certain standards. This is quite similar to the first point, but different in the sense that it speaks more of how the LLM acts or responds (rather than just knowing who or what it is).
I think this was clearly seen with the rise of OpenClaw. You gave the LLM a SOUL.md file that defined it’s identity, and it followed it to the point, no matter the consequences. It never realized that what it says might be ethically or morally harmful, and just successfully predicted the next most likely set of tokens (just like a character would speak in a movie script without having to worry about the moral implications, because it’s just a movie script). This also emphasizes the need for AI companies to prioritize safety in training.
And then this brings me to my third conclusion, which is that it doesn’t truly have a deep understanding of what it is saying or doing. I mean, I might be stating the obvious here, but I think that LLMs’ artificial intelligence is still very limited.
Again, it’s just like generating a movie script. It doesn’t truly and deeply “know” or understand that the script could harm someone else, or the implications of what it generates. It can get an idea of what it is and isn’t allowed to write through rigorous training, but it sticks to movie script writing.