Introduction: Moments of Enlightenment oping “toxicity classifiers” that can automatically flag problem- atic language the LLM itself might generate; and fine-tuning LLMs using curated datasets of texts that have been annotated by humans to indicate a desired output. In this way, an LLM might learn to avoid, say, making tasteless jokes about a report- er’s divorce. These techniques don’t eliminate problematic outputs, they just reduce them. Even with various guardrails in place, an LLM itself can’t make reasoned judgments about complex ethical quandaries, or even about more straightforward questions. Take ChatGPT, which is based on GPT-3.5, an immediate pre- decessor to GPT-4. Ask it for the fifth sentence of the Gettysburg Address, and it will probably get it wrong. That’s because LLMs don’t actually understand, in the way that humans understand, what the Gettysburg Address is, or what a sentence is, or even how counting works. So they can’t apply their “knowledge” of these things in a way that a human might. (“I’ll find the text of the Gettysburg Address, then count sentences until I get to the fifth one.”) Instead, an LLM is always just making statisti- cal predictions about what the next word in a given text string should be. From its training, though, ChatGPT has clearly come to associ- ate the words “Gettysburg Address” with other words—specifi- cally, the text of the speech. So when you ask ChatGPT for the fifth sentence of the Gettysburg Address, it will almost certainly give you a sentence from the Gettysburg Address, but probably not the fifth sentence. In fact, I bet it will be the ninth sentence. 13