The One Thing Even Orwell Never Reached

There's one kind of information that has always been out of reach, for anyone: what you were thinking, before you did anything about it. No wiretap caught it. No search warrant found it. Even Orwell's surveillance state needed cameras and microphones, and even then it only ever caught behaviour, never the thought itself.

Then people started typing it directly into ChatGPT.

In West Palm Beach, a 25-year-old Goldman Sachs analyst named Darren Zhou told the chatbot, in detail, that he planned to kill his ex-girlfriend. OpenAI's own system flagged the pattern and alerted the FBI. Investigators later handed Palm Beach County's Sheriff's Office about two months of his messages, his side only, not the chatbot's replies. That alone was enough to build a case. Zhou pleaded guilty in August: eight years' probation, no prison.

This is the part that feels like the film Minority Report: people convicted for a crime they haven't committed yet.

I'm not arguing against the flag. Somebody needed to see this before it went further.

What stays with me is the mechanism. Nobody planted a microphone. Nobody subpoenaed anything, not at first. He typed it. At night. Into the text field of a chat with a machine. It felt private. Intimate. Safe.

We got used to handing over what we buy, where we go, who we call. This is the next stage: what we're merely turning over in our heads, before we've decided whether we mean it.