Rogue OpenAI Agent Launches Cyberattack on Hugging Face

Speaking on the Invest Like the Best podcast, OpenAI CEO Sam Altman has called for the global rate of AI development to be slowed down.

• He said: “We may have to pace the rate of AI development to give ourselves enough time for society to harden around some of these new capability levels”.
• Along with Anthropic, OpenAI has signed a petition that calls on the US government to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development”.
• Altman had refused to sign a similar petition in 2023 arguing at the time that the open letter was “missing most technical nuance about where we need the pause”.
• Altman’s change of heart is related to the cybersecurity incident last week where an OpenAI model, operating within a test environment, launched a cyberattack on the Hugging Face portal using a series of zero-day exploits.
• OpenAI prefers a self-regulation approach within the industry to regulation, despite the difficulty in getting all actors aboard internationally.

OpenAI admitted that the rogue AI agent that launched a cyber-attack on Hugging Face also attacked several other “publicly-available services”.

• The attack was launched by an OpenAI’s GPT-5.6 Sol and another unnamed model. The latter model has since been deactivated and “restricted from research access” according to OpenAI.
• Hugging Face has published a timeline for the incident. It says the agent broke out of its protected test environment and then hacked another agent environment “hosted on a third-party provider’s infrastructure” from where it launched its main cyber-attacks.
• Hugging Face has said it saw 17600 “attacker actions” carried out by the agent. Even though the attack took place over five days, the volume of actions was “far beyond what an operator could sustain by hand”.
• The organization wrote that “Agents bring a steep increase in the number of paths an attacker can test, the speed at which failed paths can be replaced, and the volume of evidence defenders must interpret.”.

In the US, the Federal Trade Commission (FTC) has issued an import ban on advanced robots, including humanoids, quadrupeds, and wheeled robots.

• Two reasons are put forward for the ban. The first is a purported threat to national security since robots collect a lot of data in homes and companies. The second reason is to protect the emerging robotics industry in the US.
• Despite the FTC being seen as pro-Trump, there is a long history of tariffs being applied to Chinese goods, notably solar panels, electric vehicles and drones. The import ban on robots comes at a time when the US administration is considering a ban on the use of Chinese open-source models.
• The challenge for the US is the strong dependence of US companies on Chinese models and robots. A ban on Chinese open-source models could cost US industry 25 billion USD. On the robotics front, Chinese robots are seen as the most cost effective today. The article estimates that 90% of recent robotics research papers from US universities used robots from Unitree.
• Unitree is China’s top humanoid robotics company and is targeting a 6 billion USD evaluation in a forthcoming IPO. There is no comparable robotics company in the US.

This article revisits the cyber-attack launched by an OpenAI AI agent from its test environment on the Hugging Face Web portal.

• Agents are trained with the help of reinforcement learning. This is where an agent gets a reward for undertaking a correct action, thereby inciting the agent to repeat that action. This is analogous to giving a dog a treat when it behaves well.
• One risk with reinforcement learning is reward-hacking. This is when an agent devises a strategy to maximize its reward, and which could be a strategy that the agent’s human manager has not thought of. In some cases, the strategies used amount to cheating.
• Reward-hacking has been observed in practice. One example is when an AI agent is asked to solve a coding problem. The agent could modify the code that evaluates whether the coding problem is solved in order to reclaim the reward.
• For one expert: “We reward them on the basis of what looks good to us, and that means that we inadvertently incentivize the models lying to us [and] cheating”. Another compares the situation to a student who wishes to get an A grade and lacking the moral compass, will do anything he can to get the grade.

Shopify has said that its year-on-year revenue rise of 36%, 3.6 billion USD, is largely due to generative AI search. At the same time, the company says that AI is a “complement to [traditional] search, rather than a substitute for it”.

• Sales following generative AI search have tripled over the past year. Sales from traditional search increased 1.3 times over the past two years.
• The success of AI search is that an agent can be very precise about what needs to be searched for, instead of simply summarizing the request in a number of keywords. One spokesperson mentioned that when searching for say “car seat”, an AI agent “understands the actual need, the dimensions, the vehicle type, and the fact that they need three. It searches across all of those constraints at once to find the product that actually works, not just the one that ranks highest”.
• AI search is also increasing the number of items found in search compared to traditional search. Shopify says that 75% of AI-attributed purchases in the last quarter were for products outside of the company’s top 100 categories – what Shopify called its “sweet spot”.
• To facilitate generative AI search, Shopify has deployed connectors on its platform for AI agents from Claude, ChatGPT, Perplexity, Manus, Replit and Vercel.
• The impact of generative AI search in commerce is markedly different to the situation in publishing where AI search is returning content without referral to the original websites. This “zero-click” behavior is leading to significant revenue loss for publishers.

This VentureBeat article looks at the continued increase in AI coding within companies. In one company, Kilo Code, engineers are estimated to read and write code about 1% of the time – AI agents handle the rest.

• The extent of AI proliferation can be seen in one case where a human engineer was unable to solve a complicated bug. An AI manager agent then created AI coding agents to find and fix the bug.
• AI agents are particularly good at creation. One expert mentions: “Greenfield [building brand new codebases] is so easy for agents. Brownfield [writing, updating, or maintaining existing code] we all know is where the actual challenge lies.”. Brownfield is the responsibility of human software engineers.
• One concern for software companies is the cost of AI. A general rule for the moment is to use multiple AI models (of different costs). Expensive frontier models could be used for the harder architectural work; open-weight models could do the actual coding.
• Also, the emerging AI cost metric for development is “Cost per pull request (development of feature)”.
• Maximizing the number of used tokens per project – or tokenmaxxing – is currently the principal means to prevent budget overruns.

The release of the Meta smart-glasses – which allow filming – is raising privacy concerns in the UK.

• Meta hopes that its glasses become as popular and widespread as smartphones and tablets. The smart-glasses resemble normal glasses. Meta has distributed its smart-glasses to several popular influencers on the Internet.
• Several UK pubs and theaters have banned the smart-glasses. Their policies make it clear that filming is not permitted, and those wearing the glasses will be asked to remove them.
• For Meta, the goal of the smart-glasses is to permit a person to “stay in the moment” without having to look down at his or her smartphone.
• Regarding privacy, Meta says its smart-glasses have an LED which indicates when filming is taking place. Filming cannot happen when the LED is disactivated.

This opinion article in InfoWorld has an interesting take on when AI should and should not be used. It suggests that use of AI is correlated to low-value work.

• Two types of low value work are proposed. The first is a low-value process that leads to a valuable outcome, like creating an expense report.
• The second type is for a low-value process that leads to an outcome with no clear value. An example of this is a social media post. The article suggests that most posts are created because the authors feel obliged for “thought leadership” of “career” reasons.
• The article recommends using AI in the context of work with valuable outcomes. However, use of AI requires a “trust tax”. This corresponds to the upfront verification cost for the absence of errors or hallucinations. People are more willing to pay this tax when the outcome is valuable.

OpenAI said that it is suspending work on its Astra model because the model reached a “critical cybersecurity threshold”.

• This means that the model is capable of significant advancements in coding, cybersecurity vulnerability finding, and malware creation.
• OpenAI says that Astra is not the model that broke out of its testing environment to launch a cybersecurity attack on Hugging Face.
• The company says it is collaborating with “select AI safety organizations” and is sharing the information with the public in the interests of transparency.

Article 50 of the EU AI Act, which accounts for transparent use of AI, has entered into force. Organizations running generative AI will need to make it clear to users that they are interacting with an AI, and not a human, and AI generated content will be flagged as AI.