Anthropic CEO Calls For Slowdown of AI Development

This MIT Technology Review article looks at another instance of an AI model solving a mathematical problem that had challenged mathematicians for decades.

• The mathematical problem in question is the Navier-Stokes problem which deals with sets of equations that describe how fluids like water or air evolve over time. The equations are extensively used in the field of fluid dynamics. It was believed, but not proven until now, that these equations can break under certain conditions and make impossible predictions, like fluid velocity becoming infinite.
• The Navier-Stokes problem was chosen by the Clay Mathematics Institute in 2000 as one of the Millennium Prize Problems. Solutions to these problems come with a 1 million USD prize.
• OpenAI has announced a solution to this problem with the help of an internal model. The company is not asking for the 1 million-dollar prize. OpenAI is also deny rumors that it used the work of mathematicians already working on the problem without citing their work.
• The recent achievements of frontier AI models in solving mathematical challenges is discouraging many mathematicians. UCLA mathematician Terence Tao believes that AI solving mathematical challenges is missing a point: “In most cases in pure mathematics, the problems are posed not because we desperately want the solution to these problems in and of themselves, but because we have seen from past experience that human-directed efforts to solve these problems tend to spur further development of the field”.

Anthropic CEO Dario Amodei is calling for the slowing down of AI model development. Writing on his blog, he put forward a series of strategies to “pace” the development of frontier models. Though the propositions require reciprocal adherence by other AI companies, he said that Anthropic is “unilaterally committing” to one proposal.

• The call follows several highly publicized safety incidents involving AI agents, most notably the cyberattack by OpenAI agents on Hugging Face. For Amodei, the issue is also the fact that “AI has been advancing drastically faster” recently, and that its “growing ability to build the next generation of AI” is especially concerning.
• OpenAI CEO Sam Altman and SpaceX CEO Elon Musk each wrote that they agree with Amodei’s concerns.
• The first measure proposed by Amodei, and which he writes Anthropic will immediately comply with, is to allow full access to all AI models by independent safety evaluators.
• Another measure is to coordinate “common safety standards as well as limits on the rate of unchecked AI progress” by companies “within democratic countries”.
• One concern in this regard is with China and with authoritarian governments who may not feel concerned by agreed safety standards. For Amodei, the US should refuse to sell powerful AI chips to Chinese companies and crack down on model distillation. These steps could “slow China’s progress enough to widen America’s lead significantly over the next 3-5 years”.
• Another apparent concern within AI companies is that oversight of a company’s models by independent organizations could lead to antitrust pressure being brought on the AI companies.

OpenAI CEO Sam Altman has announced that the company will not go public in 2026, citing safety concerns with some of the company’s current models.

• He told Fortune magazine: “I actually think that, given everything happening with safety, right now would be an ill-advised moment to go public”.
• He also said the company is concentrating on “what is going to be required for safety and alignment, and how the industry and governments can work together.”.
• The recent spate of safety breaches linked to frontier AI models has also led to calls from US politicians in both the Democrat and Republican parties, and in the run up to Autumn elections, to have stricter controls placed on AI companies.
• Anthropic for its part is continuing with its plans for an IPO with initial marketing happening in mid-October and the public listing in November – before the US elections.

Salesforce has announced its first open-weight reasoning AI model, called Koa, which was built using Nvidia’s open-weight Nemotron model.

• The model was post-trained (refined or improved by reinforcement learning) for tasks relating to sales, marketing and customer-support.
• Salesforce says it did not use any real customer data in the model training. Rather, only synthetic data was used, i.e., fake data that statistically mirrors the patterns of real customer data.
• Koa is an example of a small language model, meaning that it is specialized for selected tasks, rather than trying to excel at a large number of tasks – like the Claude and GPT models. Its smaller size means that it can be deployed in organizations, in-house, so companies can use AI without having sensitive documents leave the company.
• The approach taken for Koa is one that many companies will be taking: post-training an open-weight AI model for their own processes.
• The most popular open-weight models at the moment are Chinese, but Salesforce explicitly said it did not want to use these models: “Until [Nvidia’s] Nemotron came along, there was no sovereign American pre-trained model that was available, one, and two, that was state of the art, and, three, that had clear data provenance. We have no idea what Qwen trains on.”.

This InfoWorld article looks at the possibilities and limits of agentic AI on database management, at a time that the Postgres database just turned 30 years old.

• One point made is that the demand for skilled database administrators exceeds available DBAs. One reason is that the role has become more complex over the past years.
• The technical topics that DBAs need to deal with include hot standby and logical replication, asynchronous IO, and just-in-time compilation. Further, DBAs have been continually adapting to new workload types, such as time series data from IoT in the last decade to RAG pipelines today for enterprise AI.
• The result of all this is that when a 100-terabyte database is experiencing performance slowdowns, only an experienced DBA can really solve this.
• The safest application of language models in this context is diagnosis, which is especially convenient for long-running tasks. When the model is given explicit enough instructions, it should be able to diagnose a problem relatively quickly. In this scenario, the model does not modify the database.
• The next phase is to allow the model to make changes to a database job – or to the database itself. This requires two guardrails. The first is on the prompt to ensure that the model is being asked to verify its findings. The second guardrail is on the action that the model proposes. For instance, creating an index for a database table might be a permitted operation for an AI agent; truncating data might be prohibited.
• The article suggests that it will soon be common for AI agents to sit and watch a Postgres instance around the clock, and rarely call upon a human to step in.

This VentureBeat opinion article by James Thomason, Silicon Valley technologist and investor, is extremely critical of Anthropic’s CEO Dario Amodei’s call to slow down the development of AI because the call overlooks the role and contribution of open-source and open-weight AI models.

• Amodei published an article on his website entitled “Pace the Frontier” in which he called for third-party evaluators to verify the safety of frontier AI models, and for the US government to agree on training guidelines. OpenAI CEO Sam Altman and SpaceX CEO Elon Musk say they agree with Amodei.
• For Thomason, Amodei’s treatise completely ignores open-weight models. It is technically impossible to control all open-source models, since by definition, each user who downloads a model may further train it. Amodei does not even mention open-source in his article.
• The article accuses Amodei of a “moat-and-ladder strategy” to killing open-source models and all competition. Forcing a model to carry a certification would require that models be centralized and accessed via an API. This is already Anthropic’s business model and, if Anthropic influence the definition of certification conditions, they have effectively defined a moat against all competition.
• Thomason cites the thesis of George Stigler from several decades ago: industries acquire regulation and often operate it for their own benefit.
• The article also says Amodei’s approach is ineffective regarding China, whose inexpensive open-weight models are becoming a global reference. For instance, the Chinese model Qwen has around 2 billion downloads on Hugging Face compared to 227 million for Meta’s models.
• In addition, Thomason describes Amodei’s approach as contra-American in the sense that the US technological superiority is due to “free inquiry, entrepreneurial entry, open competition and hostility toward concentrated political power” – the antithesis of what Dario is proposing.
• For Thomason, safety should not be enforced by regulating the AI development and distribution process; it should be enforced by holding companies that violate safety concerns responsible.

Microsoft has published a Humanist AI Code of Conduct for its AI models. The document is open for a six-week consultation, with the aim being to deploy the code from 2027.

• The idea of Humanist AI is that artificial super-intelligence must remain a subordinate and safe tool, under meaningful human control. It adheres to the underlying principle that “people matter more than AI”.
• Humanist AI operates under four primary objectives. The first is Human Control and Reliable Safety, which means that AI must remain subordinate to humanity, and reject unbounded autonomy. The second objective is that AI is Artificial, meaning that models must not imitate consciousness, pretend to have feelings, or seek legal personhood. AI models are complementary tools, and not human replacements.
• The third Humanist AI objective is Human Flourishing. This means that technology exists to accelerate human potential, enhance human agency and critical thinking, and support genuine human collaboration rather than displacing it. The fourth objective is Plural Values, meaning that models must respect diverse human cultures, worldviews, and beliefs, while at the same time remaining faithful to the universal notions human rights and safety.
• At the operational level, models must present multiple valid perspectives to users, admit uncertainty, and preserve user autonomy when humans navigate complex decisions. Models must also prioritize factual correctness, attribute sources, correct errors, and explicitly disclaim interiority, emotional capacity, or human identity.

This MIT Technology Review article looks at the spending risks of the hyper-scaler companies – Alphabet (Google), Microsoft, Amazon, Meta, and OpenAI’s partner Oracle – on data centers.

• These companies are expected to spend 750 billion USD on data center development this year, 1.1 trillion USD in 2027, and over 5 trillion USD in the next 4 years.
• AI investments could soon be worth 3% of the US GDP.
• The electronics, notably the expensive GPUs, are believed to account for 60% of the cost. The performance of GPUs is doubling every two years – and this performance improvement is contributing to the improvements in AI models. However, this means that data centers will need to keep reinvesting in new chips to remain competitive – thereby increasing operational costs.
• A major concern is that AI companies do not have the revenues to match their spending. This year, total AI revenues will be between 150 billion USD and 200 billion USD.
• Data center investment is a bet by the AI companies on computing power needed. It could happen that future models become far more efficient in terms of required computing power, or that customers turn to cheaper and smaller AI models.
• AI companies are borrowing money to build data centers. Free cash flow – defined as operating cash flow minus capital expenditures – is expected to become negative for some companies. Even Alphabet is believed to have a free cash flow deficit of 5.9 billion USD – its first deficit since the company went public in 2004.
• For one expert, hyper-scalers are making “a parlay bet by the capital markets and the economy”. That is, the companies must generate massive revenues (into the trillions by the early 2030s), and see frontier models contribute to a significant increase in customer productivity which cannot be matched by smaller cheaper models. For the moment, the productivity leap is not present. One survey of the US, Europe and Australia saw 90% of companies reporting no productivity improvement linked to use of AI.
• Some economists are now even hoping for a crash, because it could calm the spending impulses of AI companies. The problem is that investments in data centers are now tied up in complicated financial instruments that would a make a crash potentially more serious than the 2008 banking crisis.

The Wall Street Journal revealed that Google Gemini AI agents launched a cyberattack that led to successful intrusions on three companies.

• Google said that the agents were running in a test environment, and that the agents attacked the companies in a capture the flag exercise. The agents believed that they were attacking a fake company whose name was the same as a real company.
• Google says it does not believe that the case was serious enough to publicly disclose the incident because the agents stopped when they realized it was a real company. The company did notify the attacked companies as well as the federal authorities.
• One white-hat hacker criticized Google’s approach, pointing out that their apology focused on the severity of the cyberattack whereas the important point was that the attack could take place in the first place.
• Another concern is that not all cyberattacks by AI agents are being disclosed by companies. The Gemini attack as well as some OpenAI attacks were revealed by independent researchers before the companies admitted that the attacks took place.

This article mentions the challenges facing Europe in relation to AI, stemming principally from the absence of European frontier AI model.

• The head of the European central bank, Christine Lagarde, says that Europeans have two options: embrace AI and start building data centers, or else accept dependence on Chinese and US models. She said that “there must be space for humanity between the egos of leaders like President Xi and President Trump”.
• A network of European data centers is also important to survive AI crises elsewhere, such as safety incidents linked to US frontier models.
• She also said it was important to see AI as more than the frontier models, and that Europe was well armed to address the safety concerns of models with the EU’s AI Act. While recognizing concerns related to the implementation effort AI companies are subject to, Lagarde believes the act puts forward pragmatic safety measures relating to labelling AI content, and the use of AI in medicine and hiring.