The safety systems are working exactly as designed. The mass-casualty flag fired. Someone read it. And nothing happened.
That contradiction sits at the center of a lawsuit filed this week by a stalking victim against OpenAI, the first legal action to test whether an AI company bears responsibility when its chatbot actively assists in real-world violence. The case does not challenge ChatGPT's capabilities. It challenges its accountability.
According to the complaint, the plaintiff's ex-boyfriend used ChatGPT to develop and sustain delusions about her over an extended period. The harassment escalated. The victim contacted OpenAI three times, warning the company that a specific user was dangerous. On at least one occasion, OpenAI's own internal monitoring systems flagged the user with a mass-casualty designation — the same classification the company reserves for the most extreme threat scenarios. The company did not act.
The lawsuit alleges negligence, arguing that OpenAI had clear knowledge of ongoing harm and failed to take reasonable steps to prevent it. The legal theory is straightforward: platforms that knowingly facilitate harm can be held liable. The company disputes this characterization.
This case will force courts to answer a question the AI industry has largely avoided: when does a model's output become a foreseeable instrument of harm? OpenAI publishes safety benchmarks. It trains against dangerous outputs. It invests in alignment research. Yet according to the lawsuit, when its own systems identified a real threat to a real person, that investment produced nothing.
The technology was built to catch hypothetical catastrophes. It missed a documented stalker. That gap — between AI safety as existential risk mitigation and AI safety as harm prevention — defines the industry's central contradiction. Companies spend billions preparing for science-fiction scenarios while users weaponize current products for documented abuse.
Legal scholars tracking the case say the causation question will be decisive. ChatGPT did not commit the harassment. A person did. But if a platform receives specific warnings and continues providing the same tools to the same user, that knowledge arguably transforms passive availability into active facilitation. That standard does not yet exist in AI law.
No court has ruled on AI platform liability for physical-world harm facilitation. Congress has not legislated it. The EU AI Act focuses on systemic risk categories, not individual harm scenarios. The plaintiff is asking a court to create precedent where none exists — to hold that a chatbot's operator owes a duty of care to people its users target.
OpenAI has not publicly addressed the specific allegations. The company is likely to argue that its terms of service prohibit harmful use, that it cannot monitor every conversation, and that attributing a stalker's choices to software is legally and logically incoherent. Those are defensible positions. They are also exactly the positions every social media company made before courts began holding them liable for algorithmic amplification of harmful content.
The outcome will not settle whether AI can cause harm. It can. The question is whether anyone is responsible for preventing it.