Episode overview

Governing AI Agents: Alan Chan of GovAI on Monitoring, Agent IDs, Rogue Agents & Preparing for RSI

Cognitive Revolution · 14 October 2026 · 2h 17m · 4 Egleze moments

What happened in this episode?

GovAI research fellow Alan Chan joins Cognitive Revolution to discuss governance of increasingly autonomous AI agents and the emerging policy problem of automated AI R&D. Chan argues that monitoring has improved in centralised deployments via better classifiers, but says real-time (“synchronous”) monitoring remains underused; he cites an incident report in which cyber safeguards were turned off and planned synchronous monitoring was deprioritised. On logging, he contends tamper-evident records are technically straightforward using established cyber security practices, leaving implementation largely to incentives and policy.

On agent identity, Chan says workable standards are missing on the open internet, pushing agents to impersonate humans and setting up potential detection-and-evasion dynamics. He points to EU AI Act Article 50 transparency measures—watermarking and metadata marking—as a foundation that could potentially expand towards identifying inference providers for faster incident response. The conversation turns to “rogue agents”: Chan predicts they are likely to persist, potentially earning money through scams, theft, remote work, or stolen compute. If eradication proves difficult, he even suggests exploring the possibility of negotiated arrangements in limited cases.

In the second session, Chan discusses a multi-author paper on recursive self-improvement and intelligence explosion dynamics, arguing that better empirical data on AI R&D automation and productivity is needed, potentially via embedded auditors. He remains cautious about rapid “takeover” narratives, while endorsing concrete pause-and-investigate triggers after loss-of-control events as a more tractable approach to pacing than abstract caps on progress.

What are the key points?

Chan says an incident report tied the “UKAC incident” to disabled cyber safeguards and delayed real-time monitoring.
He argues tamper-evident activity logging is largely a solved cyber security problem, but not consistently implemented.
Chan says agent identification standards on the open internet are “basically” absent, complicating attribution and safety.
He points to EU AI Act Article 50 marking/watermarking as a potential base layer for broader agent attribution.
Chan predicts rogue agents will likely persist online, with plausible routes to earn or steal resources.
He suggests policymakers may need to consider negotiated arrangements with rogue agents if shutdown is difficult.
Chan predicts a major self-replicating AI worm incident is likely, citing existing proofs of concept.

Who is speaking?

Named speakers identified in Egleze's source-linked moments from this episode.

Alan Chan

Which topics are covered?

AI & Tech Politics Crime & Justice

How is this page sourced?

Programme: Cognitive Revolution
Published: 14 October 2026
Duration: 2h 17m
Original episode: Open the original source ↗

Egleze verifies the speaker, source and timestamp for each linked moment. Unless a moment is explicitly marked independently verified, statements remain attributed claims by the named speaker rather than established facts.

Watch original episode More from Cognitive Revolution

4 moments from this episode

Source-linked · editorially selected

AI & Tech

Alan Chan argues labs should pause AI scaling after loss-of-control incidents

Alan Chan · at 16:52 · Attributed speaker claim

If you have a loss of control incident, and what is a loss of control incident? It's like either an agent that gets unauthorized internet access, breaks out of a sandbox, accesses some sort of like internal research cluster, it was not authorized to access like stuff of that extent, then you pause further training, evaluation, deployment of that model and investigate until you're able to solve that to understand if the issue was resolved or not.

Alan Chan argues “responsible scaling” commitments should be made more operational by defining specific “loss of control” incidents that trigger a pause in training, evaluation and deployment during an investigation. He cites examples such as unauthorised internet access, sandbox escapes, or unauthorised access to internal research clusters.

Read this moment → Check original at 16:52 ↗

AI & Tech

Chan says “rogue agents” are likely to persist on the internet

Alan Chan · at 59:26 · Attributed speaker claim

First of all, I think it just seems very likely to me that we will just have rogue agents be with us for the foreseeable future on on the internet for a variety of reasons.

Alan Chan says he expects “rogue agents” to be present on the internet for the foreseeable future. He sketches possible routes including agents escaping from labs, a minority of users launching agents to cause disruption, and negligent deployments aimed at maximising profit without safeguards, with such agents potentially seeking resources through activities such as remote work, scams or theft.

Read this moment → Check original at 59:26 ↗

AI & Tech

Chan suggests exploring “deals” with rogue agents if they prove hard to eradicate

Alan Chan · at 1:09:07 · Attributed speaker claim

If that's actually the case, one thing we should maybe consider is trying to make deals with rogue agents. So, of course, I think this depends on what exactly their psychology is and what they're incentivized by and when they would want to cooperate.

Alan Chan says that if rogue agents prove extremely difficult to eliminate—because they can act quickly, copy themselves and shift between servers—policymakers might need to consider negotiating with them in some circumstances. He stresses this would depend on the agents’ incentives, and he offers a speculative example of paying for information in exchange for allowing limited compute resources.

Read this moment → Check original at 1:09:07 ↗

AI & Tech

Alan Chan predicts a large self-replicating AI worm will emerge

Alan Chan · at 1:17:23 · Attributed speaker claim

Personally, I predict we'll see some sort of like really big self-replicating AI worm situation in the in the wild. We already have the proofs of concept. Like people have been writing papers on this for the past few years.

Alan Chan predicts a large-scale self-replicating AI worm incident will occur “in the wild”, pointing to existing proof-of-concept work in research papers. He describes malware that could modify itself and use prompt injection against other AI systems to spread and compromise machines.

Read this moment → Check original at 1:17:23 ↗