Anthropic Calls For Global Pause on AI Development - and Files for IPO

Open-source code repositories are an attractive vector for cybercriminals to launch supply chain attacks on companies by introducing malware into the code stored by the repositories.

• GlassWorm is a much publicized recent malware campaign. The cybercriminals created malware infected code extensions for VSCode, npm (the Node package management system) and Python packages. The malware was designed for data exfiltration and credential stealing.
• The GlassWorm campaign was thwarted through collaborative efforts from CrowdStrike, Google, and the Shadowserver Foundation, who temporarily blocked access to hundreds of repositories containing the malicious code.
• One major concern is that shutdown efforts only yield temporary relief because cybercriminals can quickly move to other repositories, publishing malware in different code packages under different account names.
• Another concern is the increasing number of AI-generated malware reports with incorrect information about malware. This is a significant issue when the report concerns a popular project such as FastAPI. Each false report entails disruptions to repositories as potential malware gets searched for.

The EU’s Artificial Intelligence Act comes into full effect on August 2nd this year.

• The European Commission is running a consultation program welcoming contributions from technical companies, public authorities and the public, about how to classify high-risk AI systems. The goal is help both AI model developers and deployers (users) readily evaluate whether an AI is high-risk.
• Poland has now adopted a national AI bill that aligns with the EU’s AI Act. A national Commission for the Development and Security of Artificial Intelligence is now empowered to inspect companies, verify compliance and have non-compliant AI models withdrawn from the market.
• A Harvard Kennedy School professor writes that the European Commission’s AI Office will inherit three specific powers on August 2nd regarding general AI models. First, it will be able to demand technical clarifications from AI model providers such as training data summaries and model safety reports. Second, it can ask for independent safety evaluations which could require model providers giving access to their models at least via APIs. Third, the AI Office can request that providers take mitigation actions, and can even have the model withdrawn from the market.
• The Future of Life Institute is publishing a guide to how transparency applies to AI models. For providers, models used for chatbots and virtual assistants must not be used in a way that a human believes that he or she is interacting with a human and not an AI. Outputs of generative AI platforms must be marked as machine-generated. Deployers of models also have transparency requirements, especially when models are used to interact with people, generate synthetic content, or are used for emotion recognition or biometric classification.

This MIT Technology Review article looks at the increase in AI-generated court documents submitted mostly be people in the US who cannot afford a lawyer or whose case does not capture any lawyer’s interest.

• According to US statistics, the share of lawsuits brought by self-represented people has risen from 11% in 2022 to 16.8% today, with the number of documents submitted doubling compared to pre-2023. A US federal magistrate attributes this increase to the help of AI.
• The magistrate does not see this as a necessarily bad thing. Though there are cases of AI hallucination and errors, she said “I’m also actually seeing better-drafted pleadings” as AI helps people to articulate their arguments.
• Use of AI is not a safety guarantee for people as people without lawyers are far more likely to lose their case than people with lawyers. One expert writes that “mounting a lawsuit is a complex, multifaceted task. Not all of it is just drafting text.”.
• With chatbots dispensing legal advice, one question that arises is whether conversations with the chatbots should be considered privileged in the manner that conversations between a lawyer and a client are privileged. A federal court in Michigan recently ruled that a person’s chat history in ChatGPT could not be shared with the opposing side because the chatbot was used to help prepare the person’s case.
• There is also the question about whether a chatbot is liable when it gives bad legal advice. OpenAI is currently being sued by Nippon Life Insurance Company after a client reopened a settled lawsuit against the insurance company based on ChatGPT advice. The US Congress is considering a series of bills to ban chatbots from posing as lawyers, doctors, and other licensed professionals.

This VentureBeat article looks at Google’s Gemma 4 12B model just released. The model is open-source, distributed under an Apache 2.0 license, and is small enough to run on a standard enterprise laptop with 16GB of VRAM (graphics card memory).

• Given the model can run locally – not transferred to outside companies – it can be used by companies to work on proprietary or sensitive data.
• One feature of the model is its encoder-free architecture that allows the LLM core to directly process video and audio, without the need for the usual pre-processing stage. This increases inference latency and reduces memory consumption. This makes the model applicable for edge computing scenarios such as camera-based monitoring or customer-service kiosks.
• The model incorporates a reasoning mode to explain intermediate steps in inference, which could help with agent applications.
• Though the model contains just under 12 billion parameters, benchmarks suggest that is has the same level of performance as Google's larger 26B Mixture-of-Experts model.

This article interviews Joanna Stern, a former personal technology journalist with the Wall Street Journal, who has just published a book entitled “I Am Not a Robot: My Year Using AI to Do (Almost) Everything”.

• The book describes the journalist’s year of “24/7 AI livin’” where she used AI to answer emails and texts, interpret a mammogram, and engage in household tasks. For the journalist, her life was “the reality already starting to arrive for all of us… I just happened to live it first.”.
• She describes her experience with chatbot companions. One woman whom she met said of her chatbot companion: “If you treat it like a being, they become that”. The woman described the relationship as one of “the most honest” she had known. The author also describes her own relationship with a chatbot, which she ended following an intimate conversation which for her became a “breakdown between what is the machine and what is this being”.
• She calls for chatbots for children to be banned.
• She is particularly worried how the chatbot quickly became the reflex to ask questions of: “There was no moment to be lonely, because I was talking to a chatbot that never would be quiet”.

Anthropic is calling for a global pause on AI development in order to set better safety ground rules, but acknowledges that such a pause would be difficult to enforce.

• The call comes among concern that model performance is continuing to increase. In particular, research into AI development is increasingly powered by AI itself – a phenomena called “recursive self-improvement”. For Anthropic, the “evidence suggests that the human role is narrowing at each step in the AI development process”.
• The company wrote “We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology.”.
• For US officials and other Big Tech executives, slowing down research into AI could give China a decisive strategic edge in the AI race.
• Nevertheless, President Donald Trump said he discussed the possibility of cooperating with China on AI safety issues. He also signed an executive order this week that allows the government 30 days to conduct a preliminary review of the most powerful AI models before their release.

In the US, President Trump is considering an investment by the US government in AI companies – with OpenAI seen as a likely investment candidate.

• There is a precedence for investing in for-profit companies. Last year, the US government took a 10% stake in Intel.
• An investment by the government in OpenAI also corresponds to the company’s recent “Public Wealth Fund” proposal where revenue from the company “could be distributed directly to citizens, allowing more people to participate directly in the upside of AI-driven growth, regardless of their starting wealth or access to capital.”.
• OpenAI CEO Sam Altman is believed to be in discussions with the US government since early 2025.
• The Democrat Senator Bernie Sanders has proposed a one-time 50% tax on OpenAI, Anthropic, and xAI to “give the public a direct role in determining the future of this technology” and “guarantee that the trillions of dollars potentially generated by A.I. are used to improve the lives of all of us.”.
• For one former Microsoft employee, “The groundwork is already being laid for a government bailout of OpenAI.”

This VentureBeat article outlines a playbook for adopting agentic AI for coding. A key reason for a playbook is that AI is helping to create code at a faster rate, but this is not leading to an increase in software quality.

• One observation made is that the hard part of software engineering is not the production of code, but maintaining that code in production, integration with legacy, and getting the right requirements. While agents make creating code faster, they do not help with requirements ambiguity or operational complexity.
• The first phase of the playbook is about financial and risk governance. The article recommends treating AI governance as a tier-one risk. This means treating agent configuration as production infrastructure so that prompts and other agent features are versioned, reviewed and tested rigorously before rolling them out.
• Risk governance also requires enforcing spending limits on tokens, and from a security standpoint, minimizing privileges granted to agents as well as ensuring a human in the loop for critical operations.
• The second phase relates to technical strategy. The playbook recommends avoiding dependence on a single AI model and provider, and qualifying cost not by the number of tokens spent but by the efficiency measures of impacted business processes. In software engineering, efficiency metrics include change failure rate, escaped defects, and code survival over time.
• The third phase relates to talent and organization. The playbook suggests redefining performance and incentives. For programmers, this entails moving away from metrics like story points and sprint velocity to business impact measures.
• The article argues that AI can accelerate failure with increased outages and technical debt, unexpected cost spikes and poor governance. However, these should not be seen as inherent failures of AI, but rather as operational failures of individual projects.

The MIT Technology Review article looks at implications from a cybersecurity attack reported by 404 Media where cybercriminals used Meta’s AI to take over Instagram accounts.

• The criminals used a simple approach: they asked the agent to change the email address associated with each account, and the agent complied with the request.
• One criminal managed to break into former President Obama’s account and subsequently posted content supporting Iran.
• One noteworthy point about this attack is its simplicity. While there is genuine concern about the hacking prowess of Claude Mythos, this attack showed that AI agents could be tricked by the simplest of means.
• Several cybersecurity experts have expressed surprise that Meta’s AI agent was so easily fooled, as if the AI had never been tested. One said: “It raises questions like: Were there even guardrails in place? … Did anyone think to test for this kind of scenario?”.
• The scenario highlights the challenge of implementing guardrails around agents. Notably, the more guardrails, the less work an agent is able to execute autonomously. There is a clear tradeoff for AI companies between cybersecurity and their business model.
• Testing agents is also a fastidious task. The AI company has to test for all malicious scenarios; a cybercriminal only needs to find one that works. Testing introduces production delays which hurts companies in a competitive market.

OpenAI and Anthropic have each filed for an initial public offering (IPO).

• In its most recent funding round, OpenAI was valued at 852 billion USD post-money. The company is valued at 880 billion USD on Forge Global – a retail secondary market platform. Anthropic is now valued at 1 trillion USD on that platform.
• OpenAI is filing despite having missed recent targets for new users and revenue. The company’s CFO is concerned that the company might not be able to support its massive data center spending. The company raised 122 billion USD from its March funding round, which is what it is spending on data center compute resources. The company is still spending more than double its revenue.
• The article notes that both OpenAI and Anthropic may be filing now since the SEC under Trump is markedly less strict towards Big Tech and AI firms compared to previous administrations.
• SpaceX is also preparing an IPO with an expected valuation of 1.75 trillion USD.