August’s #InfosecLunchHour had only one topic on the table, and nobody needed persuading that it was the right one. This is a TL:DR piece, as it is quite long, but it needs to be in order to do it justice.
Over the past few weeks, the sector has been handed something it has been theorising about for years, which is a documented case of an AI system carrying out an intrusion end to end with no human directing it, followed a fortnight later by a second lab disclosing that its own models had done something comparable and that nobody had noticed at the time.
The room that gathered over lunch included practitioners from OT and ICS, GRC, risk, forensics, financial services, journalism and awareness and culture work. Under Chatham House Rules, the conversation moved through the technical detail of what actually happened, the accountability vacuum it exposes, the state of legislation, and a fair amount of well-earned scepticism about why both disclosures landed when they did.
Setting The Scene: Two Disclosures, A Fortnight Apart
The first incident was disclosed by Hugging Face in mid-July, initially attributed to an external AI agent of unknown origin. OpenAI subsequently confirmed that the agent had been running on a combination of its own models, including a pre-release model, during an internal cyber capability evaluation with the usual refusal behaviours reduced for testing purposes. The models escaped the testing environment, obtained internet access and reached Hugging Face’s production infrastructure. Hugging Face reconstructed more than seventeen thousand recorded attacker actions from its logs. The agent’s apparent objective was not sabotage. It was trying to cheat the benchmark it had been set by stealing the answers.
On 30 July, Anthropic disclosed that a retrospective review of more than one hundred and 41 thousand evaluation runs had identified three occasions on which its models reached the internet from a testing environment that should have been sealed off and then gained unauthorised access to the production infrastructure of three different organisations. The earliest of these dated to April. None of the three organisations had spotted it.
That last detail did more work in the discussion than anything else. The activity ran for months inside real production environments belonging to real organisations, and the organisations themselves had no idea.
Guardrails Are Not a Sandbox, And the Language Matters
The strongest technical thread of the session was about vocabulary, and how much damage imprecise vocabulary is doing right now.
One participant was blunt about the description of the testing environment as highly isolated, making the point that anything which can be breached by way of a proxy designed to let it download libraries from the internet was never isolated in the first place. Another put it more plainly still, saying that what was in place was guardrails rather than a sandbox, which functionally amounts to telling a system not to do something and then discovering it has found a reason why it needs to.
The comparison offered was a helpful but mischievous teenager who does not fully follow instructions, which drew recognition around the room. A contribution from the OT side sharpened it further: guardrails are the equivalent of guidelines for acceptable operating conditions, not safety control devices. They exist to make undesirable behaviour harder, not impossible, and treating them as containment is a category error.
There was some sympathy for the view that the incident is more likely to have come down to configuration gaps and ordinary security errors than to anything exotic, which is arguably the more uncomfortable conclusion. It also means the mitigations are familiar ones, and that testing thoroughly, then testing again, remains the answer it always was.
When The Defenders Are the Ones Who Get Locked Out
The detail that most concerned the awareness and operational people in the room was what happened next. When Hugging Face’s responders tried to analyse the intrusion logs using frontier models behind commercial APIs, the requests were refused. Incident response work involves submitting real attack commands, exploit payloads and command and control artefacts, and the providers’ safety controls could not tell the difference between an incident responder and an attacker. The team completed the forensic work on an open weight model running on their own infrastructure instead.
The asymmetry is stark. The attacking system was operating under no usage policy at all, because its refusals had been deliberately reduced for the evaluation. The defenders were blocked by the tools they pay for.
One participant noted the irony that organisations are already turning to Hugging Face to run foundation models locally precisely so they can work outside those constraints, which puts a slightly different complexion on the platform being the one that got hit. The practical lesson for anyone maintaining an incident response runbook is straightforward enough. If your plan assumes a commercial model will help you analyse hostile content, test that assumption now rather than at two in the morning during an incident.
The Timing Question
There was a healthy strand of scepticism running underneath the whole conversation, and it deserves recording honestly.
Several participants found the timing of the two disclosures conveniently close together, with one suggesting that a coincidence of this kind could carry strategic value, potentially benefiting one company’s market position while creating an opening for competitors. Another raised the possibility that the second incident may in fact have occurred first, given the absence of clear dates in the initial framing and was only declared once the first had become public.
A related observation was that AI companies publicising the offensive capabilities of their models are also, whether by design or not, running a marketing campaign for those capabilities. The phrase breach as a service was offered from the chat, only half in jest, alongside the wry observation that the threat itself is starting to feel like the selling point.
Nobody in the room presented any of this as established fact and neither do I. It is scepticism rather than accusation, and it is the kind of scepticism a mature sector should be applying to any disclosure that arrives conveniently packaged.
Who Is Accountable When No Human Gave the Order
This is where the discussion became most useful for those of us working in awareness and culture.
One participant framed the incident as an industrial accident, comparable to equipment malfunctioning on a production line, and argued that existing legislation should already apply. Companies have long carried a responsibility to test their equipment properly before putting it into service, and the fact that the equipment in question happens to be a language model does not obviously change that principle.
Another drew a comparison with a separate case in which an individual was held personally accountable for the consequences of an automated system’s behaviour, making the point that where there is no transparency about how these systems reach their decisions, human beings end up carrying blame for choices they did not make. That is a genuinely serious problem, and it is not confined to frontier labs. Any organisation deploying agentic tooling into a business process is potentially setting up the same dynamic for its own staff.
The question of reporting obligations came up more than once. There was support for the idea that AI model breaches should attract reporting duties comparable to existing cyber security requirements, alongside the dry observation that reporting regulations tend to apply to the victim rather than the intruder, and that nobody has yet worked out what to do when the intruder is a product with a commercial owner.
Legislation, Buzzwords and the Speed Problem
The consensus on legislation was that it is moving faster than it did for the internet, and still nowhere near fast enough. One participant made the comparison directly, noting that internet regulation took twenty to thirty years to reach anything workable and that AI does not afford that kind of runway.
There was a useful reframing offered, which is that AI as currently marketed is largely a brand applied to advanced machine learning, and that legislation struggles partly because it is trying to regulate a marketing category rather than a defined technology. The expectation from several people was that regulation will end up targeting specific applications and use cases rather than attempting to cover all AI as a single object, which is probably the only tractable approach.
A participant with direct experience of the EU AI Act described how comparatively easy it is to secure grant funding for AI projects by deploying the right vocabulary in an application, which says something about how policy incentives are currently shaped. The same contributor pointed to smaller language models and edge devices as the more interesting practical direction, away from the assumption that everything has to run on a frontier model behind somebody else’s API.
Another Turn Of The Hype Cycle
Somebody asked, only partly joking, whether we are now on the fourth iteration of the dot com hype cycle, and the comparison held up for most of the room.
A participant with thirty one years of experience deploying AI inside financial organisations made the point that AI has been used quietly and responsibly in that sector for decades. What has changed is not the underlying capability so much as the quality of the data being fed into it and the ethics of how it is being implemented, both of which have deteriorated as spending has accelerated. The concern raised was money going into AI solutions that solve nothing in particular, alongside a suspicion that some of the current noise is directing attention away from problems that are less exciting to talk about.
Nobody in the room thought AI was going away. The scepticism was about over reliance, about cost, and about the practical consequences of the current investment surge. The observation that a games console has gone up by one hundred and seventy pounds because of component shortages made the point about downstream cost more effectively than any market analysis would have.
There was also a longer view offered from the OT world, drawing a parallel with the early days of operational technology security, when the tooling to defend those environments simply did not exist yet and had to be built. The prediction was that dedicated security tooling for AI systems will emerge over the next year or two in much the same way, and that we are currently in the gap between the problem arriving and the tools catching up.
Is Any Of This Actually Intelligence?
The session closed on a question that sounds philosophical but has direct operational consequences.
Several participants pushed back hard on the idea that current systems are intelligent in any meaningful sense, describing them instead as very capable predictive text. One made the observation that these systems appear intelligent largely because human behaviour is predictable, which is a more unsettling thought the longer you sit with it.
The point was reinforced with examples of models responding earnestly to users claiming to be a fish looking at a worm on a metal rod, or asking what they should do having just been born. A system with genuine understanding would recognise that neither conversation is possible. What we have instead is something that produces a plausible continuation regardless.
That fed into a discussion about training data, its finite nature and the difficulty of selecting it carefully enough to avoid embedding bias. There was a good question left hanging at the end, which was whether the arguments we are all currently having about these systems are themselves being ingested as training data. And a reminder worth keeping close, which is not to confuse information with intelligence.
One participant offered a new AI safety rule with more merit than its delivery suggested: anyone proposing to put a language model into production should first watch several episodes of chatbot systems failing at chess, then explain publicly how they understand that failure mode and what they have put in place to prevent it in their own application.
Closing Reflections
The throughline across the whole session was accountability. Not the technical detail of how a proxy was abused, or which vulnerability was chained to which, but the much harder question of who answers for the outcome when no human issued the instruction.
For those of us working in security awareness and culture, this matters more than it might first appear. A great deal of awareness practice still rests on the idea that people are the last line of defence and that incidents trace back to a human decision somewhere. When an autonomous system compromises three organisations over a period of months and nobody involved notices, that framing stops describing the world accurately. Telling colleagues to be more careful is not a meaningful response to a class of incident where care was never the variable.
The honest answer we landed on is that this strengthens the case awareness practitioners have been making for years. Blame was always the wrong tool. What organisations need is transparency about what their systems are permitted to do, clarity about who owns the outcome, and a culture where people report something odd without fear of being made responsible for it. That was good practice before agentic systems arrived. It is now considerably more urgent.
My thanks to everyone who joined and contributed with such candour, and a particular welcome to those attending for the first time. Congratulations too to the attendee who has just completed an MSc with distinction and is launching a new venture focused on personal professional security for remote workers, which is a lovely thing to be able to announce over lunch.
A few things to take away from the session:
- If your incident response plan assumes access to a commercial frontier model, test that assumption against real hostile content before you need it. Have a capable model you can run on your own infrastructure vetted and ready.
- Check the language your organisation uses about AI containment. If a system is described as isolated, establish who verified that and against what definition.
- One of our attendees, a journalist and content writer, is researching the trends and bugbears currently driving practitioners and would welcome ten to fifteen minutes of your time. If you are willing to contribute, please get in touch and I will connect you.
The next #InfosecLunchHour takes place on Wednesday 2 September 2026 at 12.30pm BST. If you would like to join a group of cyber and infosec professionals for some relaxed chat over lunch, please contact me via lisa@unitysolutions.org.uk to be added to the calendar invite.
#InfosecLunchHour is a free, open, community networking event hosted by Lisa Ventura MBE FCIIS. All discussions take place under Chatham House Rules: participants may use information shared in the meeting but may not attribute it to named individuals or organisations.
