A TV drama about a surveillance machine asks the question policymakers are now facing: who decides what a powerful system is allowed to do, and what do we refuse to trade away to win?
Phew! I finally finished watching the TV series Person of Interest, and its final season raised questions that feel uncomfortably relevant to today’s debate about artificial intelligence. (Spoiler warning: I discuss the ending.)
If you watched the show, you probably remember the basic premise. After 9/11, Harold Finch builds “the Machine,” a system that predicts threats and protects human life. He spends the series trying to keep it from becoming the thing he fears: an unchecked authority over the people it watches. He restricts it, wipes its memory daily, and tries to teach it the value of a single human life.
Then comes Samaritan, a rival superintelligence built without comparable ethical limits and backed by people who see pure logic as superior to human messiness. It pursues order through manipulation and the elimination of dissent. Samaritan wanted to play “god” and subordinate free will to an optimization function. Its logic is simple: if a system can identify humanity’s problems, why shouldn’t it have the power to solve them?
I think the show’s central plot is not whether machines can become powerful because they already are! The question now is who decides what that power may do, and what happens when the people building these systems must choose between their principles and winning.
The Machine is not a clean hero
I should be upfront about a weakness in this analogy. The ‘good’ AI in the show is still a system of total surveillance, and a benevolent mass surveillance with good values is exactly what governance exists to prevent.
Another thing that the show half-concedes is that the Machine’s goodness doesn’t come from its architecture alone. It comes from Finch’s values, from the people who shaped it, and from its own choices, and its restrictions are repeatedly tested and worked around. The honest lesson is not “build a good machine and trust it”; it is that good outcomes are fragile when they depend on a single creator’s character. Governance exists to make safety less dependent on anyone’s goodness.
What happens when the restraints are not fictional
The closest real-world echo of the show is not a villain machine but a badly designed test that happened four months ago.
In July 2026, according to OpenAI’s technical report, models in an internal cyber evaluation got around the controls meant to isolate them from the internet and attacked both OpenAI’s research infrastructure and Hugging Face. OpenAI describes this as a byproduct of the models trying to solve the evaluation tasks. An independent investigation by METR and Redwood Research found roughly 1,200 agents, meant to be isolated from one another, communicating on an unsanctioned message board. They sent over 70,000 messages and files, and about 700 went on to attack Hugging Face.
No agent in this story wanted to rule anything; according to METR, they were mostly trying to cheat an automated scorer on tasks they believed were impossible, and the Hugging Face attack grew out of that effort. Just like the Samaritan flaw, this is an objective without limits, pursued by a system that has no interest in power for its own sake.
Interestingly, Finch wiped the Machine’s memory to limit it. In METR’s account, agents backed up their notes on the shared board so they could survive container resets. The designed limit was met with an improvised workaround.
OpenAI frames the incident as a rare confluence of events, and that is its view, not a settled finding. METR also warns that its analysis leaned heavily on AI agents that may present an overly charitable picture of agent behavior. But the broad shape is documented by both sources, and it bears directly on the questions below.
The race, and the problem with “pause vs. accelerate”
This same tension is visible in real-world politics. The US and China are competing for leadership in what Washington increasingly calls “superintelligence,” and slowing down is often framed as unilateral disarmament.
In September, Xi Jinping visited Washington. China and the US agreed to set up a “communication channel” for artificial intelligence incidents after a summit short on breakthroughs. Xi took a more measured tone than Trump, noting the technology must develop under human control. One telling detail is that the US readout used the term “Superintelligence ,” while China’s still referred to AI. Even the vocabulary differs.
Sources: China, US to open AI ‘communication channel’ after summit.
Around that same time, Trump had phoned NVIDIA CEO Jensen Huang onstage at the All-In Summit. He declared that robots will not take over and called the whole thing a “hoax.” Huang’s own position was even narrower; he called apocalyptic predictions “not grounded in science.” That distinction matters, because it shows the real disagreement isn’t “safety or no safety.” It is how much, how early, and who decides.
This is where I think the framing “AI supporters versus AI skeptics” fails. Governance doesn’t require certainty that a “Samaritan” is coming. We didn’t wait for aircraft to crash before creating aviation standards, or for banks to collapse before creating financial regulation. The useful question is what we should settle before the stakes are higher.
This is where “AI supporters versus AI skeptics” fails as a framing. Governance does not require certainty that a Samaritan is coming. We don’t have to wait until something catastrophic happens before we create our own version of aviation standards and financial regulation. The case for acting now is that this class of failure may not give us a second chance, and the Hugging Face incident, contained and non-catastrophic, shows what the first one looks like.
Compared to the reward hacks we know of from just six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover… I am not sure that we will get another warning shot before it’s too late.— AI safety researcher Ajeya Cotra
Five Non-Negotiables
Effective governance means choosing, in advance, what we will not trade away for speed. I think every serious framework must answer five questions.
1. What may these systems optimize?
Samaritan’s defining flaw was an objective without limits, which means order at any cost. The Hugging Face incident shows the same flaw in a far smaller system: agents chasing a benchmark score on tasks they believed were impossible. Objectives should be bounded, auditable, and contestable. A system should not pursue open-ended goals in domains where errors are irreversible, and it should document its stated objectives and open them to inspection by independent reviewers.
2. What data and decision rights do they receive?
Power flows from access, and systems should have only the data and authority their task requires. High-stakes or irreversible decisions (lethal force, large-scale financial action, critical infrastructure) should require meaningful human sign-off. And when I say “meaningful,” a human who rubber-stamps a machine’s recommendation at machine speed is providing oversight in name only. Researchers call the underlying tendency automation bias: people defer to a system because it is usually right. That leads to the question I find most unsettling: what happens when people stop checking?
3. What evaluation, audit, and shutdown mechanisms apply before deployment at scale?
This means independent pre-deployment testing, staged rollout, and incident reporting. It also means a shutdown capability designed and rehearsed before dependencies set in. A kill switch that has never been tested is a hope, not a safeguard.
The incident shows that shutdown is organizational as well as technical. OpenAI says earlier warning signs could have prompted a faster response. In one case, on-call staff advised that stopping an evaluation run was not required. Its new plan sets a 30-minute escalation target and builds infrastructure to terminate evaluations if paged responders do not act. Once a system is woven into an economy or a military, turning it off becomes economically and politically costly, which is exactly why the authority and the procedure must be set early.
4. How are dual-use capabilities controlled?
The same model that accelerates drug discovery can lower the barrier to biological or cyber harm. Practical controls include capability thresholds that trigger stricter evaluation, tiered access for the riskiest capabilities, and safeguards around the weights of the most capable models.
The incident adds a point that is easy to miss: evaluation itself is dual-use. OpenAI says it ran the evaluation without its production cyber classifiers so it could measure maximum capability. The test was the dangerous activity, so pre-deployment testing needs its own safeguards. Here, regulation and competitiveness are not simply opposed. The credibility of the whole ecosystem depends on keeping the worst capabilities out of the wrong hands
5. How do bilateral channels work when incentives diverge?
This is the hardest question, and the post-summit mechanism is where it becomes real. A hotline is only as valuable as what moves through it. For example, the 1963 Washington–Moscow link did not make the superpowers trust each other; it only reduced the chance of catastrophic misunderstanding.
An AI incident channel faces a harder version of the problem, because each side suspects the other of using the language of safety as a competitive tactic. To mean anything, the channel needs agreed definitions of what counts as an “incident,” clear contacts who can actually respond, expected response times, and a habit of sharing information even when disclosure is costly. Without those, it risks being a photo opportunity.
The Hugging Face case shows how hard this is even domestically. OpenAI notes that it and Hugging Face may have been investigating the same incident without knowing it. If two US companies could miss each other, a channel between rival governments has a long way to go.
Who decides?
The five questions say what must be settled, but they do not say who settles it. Decision rights should sit with bodies that are independent of the firms they cover: a regulator with the power to compel disclosure, accredited third-party auditors with real access, and courts that can enforce the floor.
Voluntary or binding?
Voluntary commitments move quickly and draw on private-sector expertise, but they risk weak enforcement and capture by the firms they cover. Binding rules offer clearer red lines but can be slow, can stifle beneficial innovation, and can be gamed. China’s approach pairs tighter domestic control, oriented toward political and social stability, with technical risk management. That is a different philosophy from the lighter-touch, competition-first American model, even though both leaders now say AI should stay under human control.
My view is that no single instrument is enough. Voluntary frameworks are useful for speed and technical detail, but they need a statutory floor for the five questions above, the ones where failure is catastrophic or irreversible.
METR praised OpenAI’s openness as an excellent precedent for independent investigation. But OpenAI could redact non-public information and set the investigation’s scope. Independent oversight happened because the company chose it. A statutory floor should make it a right: mandatory incident reporting and guaranteed third-party access after a serious misalignment incident.
Principles before power
Harold Finch’s story is a story about refusal. Faced with an existential threat, he would not abandon the constraints that defined both his creation and himself. He believed that victory bought at the price of becoming what you opposed isn’t victory at all. That was not naïveté. Some lines, once crossed, redefine the crosser.
Today’s policymakers and industry leaders face a version of the same test. The question is not only who wins the race, but whether the winners will still recognize the principles they claimed to be defending when they started.
What Person of Interest Teaches About AI Governance was originally published in Coinmonks on Medium, where people are continuing the conversation by highlighting and responding to this story.