AI Safety Concerns and Bubble Fears Persist

MIT Technology Watch interviewed Mark Chen, Chief Research Officer at OpenAI, about AI agent safety.

• The interview follows the recent spate of much-publicized attacks by OpenAI agents, including an attack on Hugging Face and another on the Australian government’s medicare system.
• One criticism made of OpenAI in recent weeks is the late disclosure of attacks. The Australian government said they were informed by OpenAI of the attack 84 days after it happened. Chen argues that OpenAI is “figuring out the process of disclosure”, adding; “We want to make sure we do in-depth investigations before we just put details out there in the open”.
• Another instance of OpenAI agents launching an attack was discovered last week – after new safeguards were put in place. However, OpenAI did discover the attack 15 minutes after the attack began which can be attributed to the safeguards.
• For Chen, the Hugging Face incident was a watershed moment at OpenAI because it led to the realization that models needed to be watched during training in addition to post-deployment. Chen says that OpenAI has shifted 5% to 10% of its resources towards monitoring and safety work.
• However, the New York Times reports (https://www.nytimes.com/2026/09/29/technology/openai-warnings-security.html) that several employees at OpenAI had already informed management in the past that models needed to be surveyed during training.
• Chen is evasive on existential risks posed by AI models. He says he expects open-source models to exhibit the behavior that led OpenAI agents to launch cyberattacks within 6 to 12 months, and that this behavior could be sought after by bad actors. He seems to suggest that OpenAI, with safety guardrails in place, is the best bet for AI safety.
• He also repeats a common message from AI companies regarding risks – just think about the positives. He says “It is time to start delivering the benefits of AI to humanity. It’s time to start working on deep problems in drug discovery, on materials, on scientific applications that will actually change people’s lives.”.

The Atlantic has published the resignation letter of David Robinson from OpenAI – the man who until very recently led the transparency work at OpenAI which included writing safety reports for each OpenAI model released.

• The letter slams the safety culture at OpenAI and in Silicon Valley in general. The fundamental issue is that Tech companies use the idea of “iterative deployment”, which is the idea of improving systems in stages through trial-and-error.
• Iterative development becomes problematic when the technologies being developed have inherent safety concerns, because the nature of trial-and-error is to accept that failures will arise.
• Robinson underlines the over-confidence of co-workers at OpenAI in relation to recent safety breaches. The company claims to have put new safeguards in place following the Hugging Face cyberattack, but these safeguards failed as recently as last week as more agents escaped their perimeter.

OpenAI CEO Sam Altman has said that we should expect further bad things to happen before AI delivers on the “good stuff”.

• He said he thinks that “the lighter touch regulatory stance we advocate for comes with an accepting of the fact that some bad things are going to happen as society figures out the resilience”.
• He underlined his philosophy by stating “that other bad things that will happen because I think that people will do tremendously – orders of magnitude – more good stuff than bad stuff”.
• He also positioned OpenAI as a safety net against open-source and open-weight models which will create a “coming tidal wave of cybersecurity problems”.
• Some analysts see Altman’s bullish confident as being in preparation for the company’s upcoming IPO (sometime in 2027).
• OpenAI retains the support of US President Donald Trump who is in favor of self-policing by the AI companies of their models for safety concerns like bio-terrorism and cyber-attacks.
• A contrary position is taken by the Florida State governor, Ron DeSantis, who recently asked a judge to bar OpenAI from developing new AI models that do not have third-party approved guardrails.

This article reports on a well-known job application processing platform, HackerRank, that has integrated AI to evaluate candidates. Companies using the platform include Snowflake, Snorkel, Amazon, Nvidia, Clay, Replit and Capgemini.

• Used primarily for hiring software engineers, the platform evaluates the “AI literacy” of candidates. Previously, candidates were evaluated by assigning them coding challenges and evaluating their code. The new system evaluates how well candidates use AI in producing code artifacts.
• For one expert, using AI in the evaluation process reduces the incentive to cheat – since candidates are using AI anyway. A spokesman for the company said that suspicious-activity flags were 70% to 80% lower than previously.
• Using AI raises the same worries as all automated hiring tools – how much can one trust the algorithm? This issue has prompted regulation because AI systems inherit biases in their training datasets. In New York City and other places, candidates must be informed that their dossiers are being evaluated by AI. Further, AI systems must be evaluated by independent experts.

This article reports on how safeguards put in place by OpenAI and Anthropic on their models to protect against the creation of dangerous software code is leaving many developers feeling frustrated.

• The recent Developer Ecosystem Survey by JetBrains which questions over 15’000 developers found that 90% of them are using AI coding agents – 68% of them are doing so daily. (https://blog.jetbrains.com/research/2026/08/ai-coding-agent-adoption-2026/)
• The models are integrating safeguards as they become better at creating code – but also at creating code that could exploit existing or zero-day vulnerabilities. OpenAI’s GPT-6 Astra for instance is the “first model to reach the Critical level of cybersecurity capability under our Preparedness Framework”. OpenAI says the model could be used to discover exploits in protected systems without step-by-step human supervision.
• The Wall Street Journal reported that OpenAI canceled the release of GPT- 6.1 Astra because of safety concerns.
• One developer working on a project to use AI agents for controlling simulated spacecraft saw a series of refusals from the models: “Stuff like creating a user interface for an arm and being able to SSH into another machine, I would get a lot of refusals”.
• The increasing amount of refusals is leading developers to consider open-weight models.

As the costs of using large language models increase, and are expected to continue increasing, an InfoWorld article looks at means of reducing costs.

• The first technique to reducing costs is model routing. This is based on the idea that organizations using a large model also deploy smaller more inexpensive models like GPT-4o mini or Claude 3 Haiku. Since a large model is not needed for all requests – those that can be handled by smaller models are handed off to the cheaper models by the model router, thereby using the large model only for the more challenging tasks.
• Frameworks such as RouteLLM and Semantic Router can dynamically classify tasks for model selection.
• The second cost optimization technique is semantic caching of requests and responses. A traditional cache does not work since it would treat the requests “How do I reset my password?” and “I forgot my login info” as distinct requests. For this reason, requests are first embedded using tokens so that similar requests can be matched using a vector search.
• The third technique is prompt caching. Whereas semantic caching stores responses to prompts, prompt caches store the context that is used by requests, usually resulting from retrieval-augmented generated (RAG) requests.
• There is an optimization for prompt caching, the fourth optimization technique, that addresses the problem of context rot – where the data in the context becomes less useful over time, e.g., some documents retrieved via RAG may lose importance. Small rerank models may optimize the context before the whole request is passed to the primary model.
• A fifth optimization technique is response restraint, which means forcing the language model to be as minimal as possible in its responses, e.g., no “Thank you for the question …”. One example of this is forcing all outputs to be in the JSON format.

Reuters reported from a major cryptocurrency conference in Singapore uniting top investors how the vibe around AI is being soured by persistent bubble fears and AI safety concerns.

• The conference was held just days after Anthropic CEO Dario Amodei called for a slowdown of AI development, saying that advanced AI poses “catastrophic or existential risks to humanity”.
• For the founder and CEO of the Indian biopharmaceutical firm Biocon, AI is a force for good but in the hands of bad actors, AI could be used to design new viruses or pathogens.
• The Singaporean Foreign Minister said “whether it's in chemistry or biology or in weapons of mass destruction, is very real”.
• The bubble remains a concern, with one investor warning that “if the AI narrative is unwound because of safety issues, regulation or because as we go through 2027, end users don't see sufficient ROI (return on investment), there could be issues”.
• For Bridgewater Associates founder Ray Dalio said AI was in a “classic bubble” and there was a difference between “how terrific a technology is and how terrific an investment is”.

Microsoft has published its 2026 Digital Defense Report, and AI is a core topic.

• Criminals are making extensive use of AI. While the types of attack have not fundamentally changed, the pace of attacks have significantly increased. Defenders are also making use of AI – for threat analysis in Security Operation Centers (SOC) for instance – but the report admits that the advantage is currently with the criminals.
• The report explains that 63% of intrusions have led to data theft, that exposed cloud workloads get attacked in an average of 5.3 hours, and more than 46 million business email impersonations were detected last year. Between 89% and 95% of email phishing attachments seek to steal credentials.
• AI models are themselves the target of cyber-attacks. Attacks happen on the models’ running software, but also on the training data (so that the model might be manipulated in a manner desired by the criminals).
• AI is being used by criminals to discover software vulnerabilities that can be exploited, to generate custom malware, to shorten the lifecycle of a cyber-attack (from intrusion to data exfiltration), and to scale phishing and social engineering attacks.
• State actors are increasingly responsible for cyber-attacks, with China, Iran, North Korea and Russia being explicitly named. The report uses the term poly-crisis because the digital crisis overlaps with geopolitical and economic crises.

The European Commission is facing criticism from within Europe about the relatively slow pace that the AI Act is being implemented.

• One reason for the delay is the time it takes to set up the commission’s AI Office with the required competence. Currently, the office has about 40 people. Another reason are the concerns linked to impeding innovation.
• Another issue is the spate of recent safety issues, such as the attack by OpenAI agents on the Hugging Face portal. This represents a loophole in the AI Act since the agents launched the attack during testing, before the model’s release. The EU AI Act sanctions model misbehavior after release.
• Another unclear aspect where parliamentarians are calling for clarity is liability. It is not clear for instance that OpenAI can be held accountable for the Hugging Face cyber-attack under the Act.
• The EU has also decided to sign a Finland-Norway led initiative calling for an international body to oversee AI safety. The idea has already been rejected by US president Donald Trump who describes existential risks as a “hoax”.
• For one commission member, the challenge is not really the existential risks, but whether humans are capable of maintaining control of increasingly complex systems.

OpenAI has admitted that its revenue for 2026 should be around 50 billion USD even though they had estimated 70 billion USD in revenue earlier in the year.

• The news led to jitters on the US stock market with the Nasdaq falling 1.4%, Nvidia falling 2.9%, Oracle falling 5.5% and Micron falling 4.8%.
• The company is currently believed to be in early-stage talks to raise 30 billion USD in a new funding round that could see the business valued at 1.4 trillion USD.
• The company’s IPO is planned for next year. It was postponed from this year following safety concerns around the AI models.
• A key investor in OpenAI over the years is SoftBank with an investment of around 65 billion USD. This company is still heavily investing in AI and is currently believed to be raising 100 billion USD from Gulf states.